cancel
Showing results for 
Search instead for 
Did you mean: 

Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

Datalink
Up to speed

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.


The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...


Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...


Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...


That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.


Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.


Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 

 

 [MOD EDIT: Subject heading changed to assist community]

4,478 REPLIES 4,478


wrote:

What happens when the packet(s) are delayed by 200 to 300 ms?  In theory you have a string of packets arriving one after the other, with some packets potentially lost as they timed out in the modem itself. I suppose as you indicated "stuff will jump around".  I can't see anyway around that.  


In UDP it'll be specific to the application. 

Most game engines take your keystrokes and movements and apply them "at the time they happened" rather than "at the time they arrived" they do this by rolling back time on server. This typically means if you saw someone on your client and hit them then it will register as a hit on the server, i.e you don't have to lead players to account for network lag. It does however mean you get artifacts like you can look around a wall, step back and then die to a shot - this is because you were shot while you were looking but the server only resolves this after you hid.

Why does it do that when the client knows you hit them? Well because people are dishonest and the server can't trust the clients.

When you see kids in games saying "Whaa whaa Bill is hacking he's shooting people through walls" - this is their ignorance about lag compensation.

In terms of the client it interpolates movement to make it look like everyone is moving smoothly.

Firstly you have to realise, whatever your FPS, let's say it's 120fps. Most multiplayer games are not sending 120 updates a second. By default some only send around 20. So everything is choppy. If you saw what your typical multiplayer game actually looked like without client interpolation and lag compensation it would be a choppy mess.

The whole thing where people are running around smoothly (even if you have a low ping) is an illusion created by client interpolation.

And the default time for interpolation is set so that even if a snapshot is lost it doesn't matter. Obviously there are limits but ironically where it goes wrong is often when people who are ignorant about how it works decide to change their settings and they turn off a lot of the features that would workaround issues like the odd slow packet.

They have their config so it works great if and only if you have a close to perfect internet connection.

@plums1234

Do you play online at all? If yes have you noticed an improvement?

Ok time for longish post again least we have impression "SH3 is good, 100ms spikes are nothing, problem is overblown, VM should just ignore the whiners, etc".

Full disclaimer - I am a long time game programmer myself and no stranger to solving network quirks in good old days of 64kbit internet and 500ms roundtrips. I've also spent extensive time analysing whatever SH3 done to my connection after VM "upgraded" me (I wasn't aware of problem at the time) and I've noticed the odd choppiness (as a game programmer I have a discerning eye for frame rate and jitter), so had to do full analysis with packet analysers etc, since I went all way up to VM CEO office to prove I really am affected and need that downgrade.

First of all, let remind ourselves, what is SH3 problem? Problem is, it delays packets roughly each one minute by amount of time above 100ms dependant on amount of bound channels - in worst scenarios up to 150ms. Also it completely drops UDP packets sometimes. There is a beta firmware which by all intents appears to be Arris "V" firmware which solves delays with ICMP packets (making ping appear stable & good) and UDP dropouts (leading to more responsive websurfing experience), but leaves per-minute 100-150ms spikes on all other traffic (TCP & UDP) as they were. Arguably fixing ICMP (which mostly used for diagnosis) while leaving TCP&UDP with problem did nothing but muddle issue (I don't think it was deliberate attempt to hide it, just "fix whatever we can manage to").

It is true that most game engines interpolate and do pretty good job of hiding odd jitter. So most likely your objects on screen still move well and smooth. Does this mean everything is hunky-dory? Not necessary. Under hood, there are still per-minute irregular 100ms spikes in latency.

The server can't reverse causality, if it known that player pressed the trigger 150ms latter, that it. Yes, there are certain "rewind time" techniques, but its not magic - all it can be done is by delaying (buffering) everything which limits by how much you can actually "rewind time". Simplified, this means that to rewind this 120ms spike, you need at least 120 ms of buffer. This is too long for most modern games - most modern games don't expect users internet connections to be that bad! Creating 100ms of buffering will make very "laggy" experience for everyone - since there will be too perceptive delay for all people reaction.

Netflix, youtube, anything not-realtime streaming will be fine, no surprise. Since its not realtime, its usually have huge buffers up to several seconds - enough to absorb jitter 10x times worse.

Something like RDP won't bear as good though. Move a mouse and you will instantly spot difference in perceived "choppiness" comparing to SH2AC - specifically because many "server-side mouse" techniques explicitly depend on roundtrip staying as fast & consistent as possible.

Its true that network game engines working at ~20fps is not uncommon (though I must say many fast-paced games now increase it, since it may be simply too sluggish for twitch shooters). However 20fps is 50 ms per frame. Having 100ms lag spike still will drop at least one frame which will not be interpolated - again, because most games a) don't expect very "spiky" jitter and b) unable to completely absorb such large delay completely.

How this delay will affect you will vary by game and playstyle. I'd say it still possible to spot it (I did after all), but you really have to have discerning eye, and only in certain games with low buffering and sustained lag requirements <40ms.

More likely and regular effect which actually happens is "control jitter", which is mostly invisible. It only have "indirect effect", like, decreasing accuracy of your shots. Imagine trying to lead something fast moving, while something slightly bumping your aim every minute, but without you noticing, because game client & server try their best job to hide it. Still in many cases you will miss shots simply because you've happened to press trigger when lag spike happened - server is not prescient, unable to remove that spike, and you miss a shot while you shoul've hit - but normally it happens when everything is moving so fast anyway, so you kind of "bah, whatever, I did miss I guess". It would be fine if this "hidden shoulder bump" will happen only each hour or so, like on usual modern domestic internet, but with SH3 it happens every minute, like a clock.

So in a nutshell:

- does problem exist? yes

- does it affect games & applications? yes, because lag spikes are here

- does it affect them do detrimental extent? this entirely up to specific user expectation and needs

Most argument I see here stems from desire to make "one shirt that fits everybody", e.g. trying to claim that "SH3 is universally bad" or "SH3 problem is overblown and could be completely ignored". This is pointless IMO and better to stay on topic discussing the actual problem impact (measured impact preferrably) and possible solutions.

 

So wait... they tested a new release with a handful of people, didn't release that to the general public, but the released one that hadn't been tested (to my knowledge). 

I either hope it's the same release, but a different version number or that at least a very small number of people did test it. Seems completely bizzare that they would do this, although in the same breath... unsuprising. 

--------------------------------------------------------
Look behind you, a three-headed monkey


@RidingTheFlowwrote:

So in a nutshell:

- does problem exist? yes

- does it affect games & applications? yes, because lag spikes are here

- does it affect them do detrimental extent? this entirely up to specific user expectation and needs

 


You forgot to include a: "My BQM graph looks bad, I really need it fixed. What are VM playing at?" category.

My Hub 3.0 took a massive **** overnight it seems, it was inaccessible this morning (remotely) until I had VM support restart the router.

hub322dc1c2f4e5aa00c44c2db4cbc0630a06ebc76dc.png

Still on the trial firmware so no update yet for me.

It makes me sad to compare my graph to my mate who managed to get his Superhub 2ac reactivated (he had an outage yesterday during the day).

hub2ac0b9bac914590be737475dccbf36d160cb5b02b03-13-02-2018.png

Hi, yes I play FPS on XB1X, like PUBG, COD's, TF2 would be my most played online, I don't see the momentary freeze or dropped frames I used to see, I know that might not be correct terminology and there's a debate going on here about it and i'm not trying to get involved in either, but to me was like a micro stutter and people could teleport a few feet in game terms making them not where I was aiming, felt like the data lost for the split second, also the said games feel smoother and more responsive now, like it used to be on my Sky Fibre.....I used to win 90% of 1 on 1's now I'm losing 90% of them and dead before I even see them now lol might be because I'm an old man and the lag helped me lol


@RidingTheFlowwrote:

Ok time for longish post again least we have impression "SH3 is good, 100ms spikes are nothing, problem is overblown, VM should just ignore the whiners, etc".

So in a nutshell:

- does problem exist? yes

- does it affect games & applications? yes, because lag spikes are here

- does it affect them do detrimental extent? this entirely up to specific user expectation and needs

Most argument I see here stems from desire to make "one shirt that fits everybody", e.g. trying to claim that "SH3 is universally bad" or "SH3 problem is overblown and could be completely ignored". This is pointless IMO and better to stay on topic discussing the actual problem impact (measured impact preferrably) and possible solutions.

 


One thing I would like raise from your points is that the firmware released now to general VM users is not the same version as the beta release so there may have been further improvements since you last ran all your checks?


@boltedenergywrote:

@RidingTheFlowwrote:

Ok time for longish post again least we have impression "SH3 is good, 100ms spikes are nothing, problem is overblown, VM should just ignore the whiners, etc".

So in a nutshell:

- does problem exist? yes

- does it affect games & applications? yes, because lag spikes are here

- does it affect them do detrimental extent? this entirely up to specific user expectation and needs

Most argument I see here stems from desire to make "one shirt that fits everybody", e.g. trying to claim that "SH3 is universally bad" or "SH3 problem is overblown and could be completely ignored". This is pointless IMO and better to stay on topic discussing the actual problem impact (measured impact preferrably) and possible solutions.

 


One thing I would like raise from your points is that the firmware released now to general VM users is not the same version as the beta release so there may have been further improvements since you last ran all your checks?


See my comments above about the new release.

--------------------------------------------------------
Look behind you, a three-headed monkey

Looks like I got the new firmware rolled out to me, not home so all I can comment on right now is my BQM which while showing less ICMP spiking still shows a large amount of 2x spiking





Forever waiting for the Hub 4