Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

[Datalink]

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.

The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...

Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...

Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...

That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.

Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.

Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 [MOD EDIT: Subject heading changed to assist community]

[plums1234]

Hi

I have both Sky and VM as just signed up for the VIP bundle and within my cancellation period with sky so have both live at same time, I only changed due to my discount with Sky ending bringing it up to the same price as the VIP bundle for same TV package but no multi-room or phone calls included with Sky package.

My setup on both is quite similar on both, VM= SH3 in modem mode + Asus RT68U as router, Sky = Openreach HG612 modem + Asus RT68U with Merlin firmware as router. 

I've never attached these TBB graphs before but I've had a go at attaching link for both VM and Sky, sorry if not done correctly.

VM one 

https://www.thinkbroadband.com/broadband/monitoring/quality/share/d8dab92182dc9b46189990e60e1194332154d74d-10-02-2018]My Broadband Ping

Sky one 

https://www.thinkbroadband.com/broadband/monitoring/quality/share/80a25dcbcae44e919e867732b5738ff5a9912236-03-02-2018

I mainly play FPS online on the XB1X, games I play are COD:MWR, COD:Blops3: COD:WW2, PUBG, Titanfall 2, Forza 7, Destiny 1 and 2, that would be the main ones. 

Maybe I'm not good enough to notice but the only game I see enough lag to annoy me is on COD:MWR, but that could be the game as can see lag on that game no matter what connection I use and situations where I blame the lag for my death is hard to replicate and hard to belive they where better than me lol. I do get lag spikes on VM and is easy to prove by just using the command prompt or TBB BQM, even my wife was watching COD briefly and when someone teleported she asked is that a new thing as never seen that before 🙂 

I am by no means saying the SH3 is acceptable, as I'm quite disappointed at how poor it is and for how long this has been going on for, I've only had VM for 2 weeks and almost regret it as had years of super smooth service from Sky on normal broadband and VDSL, only thing with Sky Fibre is your restricted on what router you can use as not all work with it.

I do get full speed on my VM connection which is great for games and updates, sorry my post is not that informative as I'm undecided myself still and not sure if it was the right move, I never really stressed my 76mb Sky connection except for game downloads or updates so although the the VM bandwidth is nice I'm not sure its needed.

I'm hoping a new hub or some sort of firmware update improves it, probably in vain! If not its off back to Sky I go.

I suppose to sum it up, my Sky Fibre is smoother for me but the VM connection doesnt make it unplayable for me but I do understand some being put off by the SH3 ping spikes. I'm not a COD god on Sky connection, still get killed as much, for me its not an extra bullet.

Anyone any ideas on when he new firmware for SH3 arriving to public? Although feedback don't look good.

[wotusaw]

Thanks Plums1234. Very helpful that was.

I have the new update as was put on the guinea pig list. Don't think it's made any difference. At first I did but now think that was just me turning the HUB3 off for a couple of hours a day to give the poor thing a rest.

Think the above is the update you mean but there is another which I've heard may actually work...something I find hard to believe from info here and on DSL.

Was hoping for a HUB4 later in the year. Perhaps that's just a dream.

The difference for me with SKY would be £480 a year. My contract with Virgin has run out now. I've never had anyone else other than Virgin since the 90's. So just curious is all.

OMG!! Just looked at your graphs, . Actually my VM graph is SOLID YELLOW! I'd kill for a VM graph like yours, .

[Andrew-G]

plums1234: That's about the sum of it. 

The inherent latency problems (ping) of the Hub 3 aren't enough on their own to stop you playing, but they are enough to take the edge off for some players.  I can see the difference between gaming on the last VM hub, and the Hub 3. 

When gamers talk of lag, they normally mean a persistent high latency connection, or one where the latency is high for several seconds at a time.  What the Hub 3 does is different to that, its CPU pauses processing input and outputs for an internal system process every couple of seconds, and the latency of say 30 ms leaps to 150-200ms, and then drops back.  So that's like really momentary, the duration of the blink of an eye, but for a monitor refreshing at 60 Hz, ten of those frame refreshes are lost.  If you're really observant you can see this in texture tearing, or very small jerky jumps of other players.  They aren't actually jumping, but your system has not had the info from the game server, and the jump or texture tear is where your system updates with the latest game data, and there's a discontinuity between the two.  Practical implications are that you can be shot by a player who you may not see, or who appears to be facing a different way.  And you may miss what looked like a certain hit, because the target player has moved whilst your input was paused.  During those pauses, your character momentarily stops moving, but all the other action on the game server continues, so there's some downside, but no upside.  For me, gaming on the Hub 3 is better than not gaming at all, but like you I'm no gaming god, so I could do without the blasted thing working against me.

Lots of Hub 3 users don't even notice.  I do, and I find it infuriating because the problem was entirely avoidable, if Virginmedia had listened to customers who did testing, or if VM had decent procurement.  But VM deny there is a problem, have continued to dole out the horrible, cheap device even eighteen months after the problems of the Puma 6 became public knowledge, and even have the audacity to describe the Hub 3 as "best in class".  There's a litany of other faults with the device, but mostly not relevant to the gaming concerns.

I'd say only change if you're sure what you're seeing is the Hub 3, and that is a deal-breaker, otherwise stick with VM until your contract ends and they hike the price, and then re-evaluate your options.

[plums1234]

If your going to save nearly £500 a year then I think I know what I would do, I changed only due to VM being about same a year cheaper for me for similar TV package or I would still be with Sky. 

Could my graph be better due to using my own router? As I never tried it the SH3 in router mode, as soon as the guys left I put it in modem mode and setup a new RT68U

Not good news that your a trialist and still still not impressed, hopefully your correct about some better fix in the pipeline, fingers crossed. I've also read on DSLR, REDDIT and this very forum that software wont fix the hardware issue, which is worrying.

Like you i've been with same supplier for years and wanted a change, I will more than likely see out my discounted contract and then review it at the 10 month period and if still no progress then I will move on.

[Andrew-G]

"Could my graph be better due to using my own router? "

No, not at all.  If anything using your own router takes some of the load off the puny Hub 3 electronics, but the core latency problems are related to the cable modem side of things.  And you'll be getting far better wifi using a purpose made router.

"Not good news that your a trialist and still still not impressed,"

I've not been, but my comments are based on those who have been, you'll find their comments scattered throughout this thread.  Some people on versions of the trial firmware say it improves things, some say it doesn't (and at best it can only fix the software aspects, the hardware limitations are there for good).  Whether that's different version, or simply different expectations I can't say. 

I don't think VM really care - they've foisted about two and half million Hub 3s on their customers, and the moaners are a tiny minority, albeit an irritating one.  I do believe something will be rolled out on the firmware front, but from VM's point of view the Hub 3 looks like a modern hub, it's cheap, the casual or easily pleased users are happy, and it works better with their side of the network than previous hubs, even if it causing problems for a few.  Which means that if the firmware fix turns out to be inadequate, they aren't going to bother trying again.  There certainly won't be much effort to proactively replace them as (say at a programme cost of £30 per device) that would cost £75m.  And when they launch a Hub 4, it will be a much more expensive device because of the DOCSIS 3.1 technology, so limited to people on very fast 500 Mbps or higher connections.  Even then, when I say expensive, I mean relative to the Hub 3 - the Hub 4 will also have every corner cut to reduce costs.

VM are being caned by their parent company for cash and profit.  Unless the Hub 3 issues cost them more money than fixing the problem properly, they won't fix it.  And because the moaners are few in number, the costs are low, even if the unhappy cancel their contracts.  The bigger problem is the noise and reputational damage, because that puts off some prospective customers who mostly would not have noticed any problem, but VM probably judge this is still a small number. 

[plums1234]

Thanks for taking the time to type such a detailed reply.....as an advocate of using my own purpose built gear it would be good if they allowed us to use our own modem, even if VM provided an approved list we had to buy from? I certainly would purchase one if it cured the current SH3 latency issue. I have seen comments scattered around and as I read this thread while waiting on installation and it seems mixed feedback but I feel more biased towards it not being what we are looking for.

[Boygroucho]

"Has anyone here had SKY fibre unlimited 62mb-76mb and Virgins 350mg.

How do they compared in actual gaming?

Virgin is hugely faster but does that show up in gaming especially with the HUB3 latency/maintenance mode/whatever problems against SKY fibre 62mb-76mb?

Sky comes in at £50 and Virgin at £90 a month. As said Virgin is much faster. Need someone whose actually tried BOTH."

As a long time virgin media user and COD fps player I would add:

Pre SH3 i was would normally settle at around a 1.2 K/D ratio on COD.

Post SH3 I can't get above 0.8.

Typical of the problem...I saw a shadow of a guy about to come round a corner so I started shooting in advance...he came round the corner and shot me instead.   In a sudden face to face meet, I will most always lose.

If gaming is important to you then I would seriously consider your options.

Noting than your VM BQM graph is essentially identical to mine.

[wotusaw]

Boycroucho@

"As a long time virgin media user and COD fps player I would add:

Pre SH3 i was would normally settle at around a 1.2 K/D ratio on COD.

Post SH3 I can't get above 0.8.

Typical of the problem...I saw a shadow of a guy about to come round a corner so I started shooting in advance...he came round the corner and shot me instead.   In a sudden face to face meet, I will most always lose.

If gaming is important to you then I would seriously consider your options.

Noting than your VM BQM graph is essentially identical to mine."

Yes, agree. I play Titanfall 2 and my son plays Overwatch. Both suffer from the hub3 curse and it can be plainly seen on killcams.

Andruser is right on the button. They won't care and by that I mean the people in charge, not the poor sods who have to deal with our complaints.

I don't won't to go but it's been on my mind alot. It's a saving but it does mean alot of hassle. I've supposedly got virgin engineers coming round to sort my line out in March. I had to fight for that and the tester engineer who orders the job was not a happy bunny at all and made me know it. 

For my Virgin line I can get BT and SKY at almost the same cost. With me I need something to make me angry to get my **** into gear and sort it all out.

However, you know, it's just the curiousity about what the others are like(sky/bt) that's keeping this on my mind. Maybe I just gotta know.

[Murfi99]

Hi All

Are there any real updates here?

Anything at all happening? every time i check i am still on the same firmware : 9.1.116V

and there just seems to be banter on here and not any real updates of this situation?

Any information would be grateful thanks

[legacy1]

I think VM are just going to keep sending hub 3 out till the next hub being that firmware can't fix under powered chip in hub 3.