Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

[Datalink]

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.

The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...

Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...

Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...

That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.

Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.

Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 [MOD EDIT: Subject heading changed to assist community]

[RidingTheFlow]

---

@Guybrush85 wrote:

Hub 4: Again, it’s a waiting game and speculation is that it’s a puma 7 device, although hopefully some of this spotlight means a rethink on that, which could mean delays in the hub 4.

---

Hub 2/2ac is Puma 5, so "Puma" platform by itself is not inherently bad - just that they've managed to screw up version 6 and tried to play "nobody will notice/not a fault" game.

So Puma 7 may be not bad either. I haven't seen any proper tests of Puma 7 modems yet.

[Guybrush85]

The understanding I have is the puma 5 and 7 aren't as bad as the 6. Since the DS channel increase seemed to be thing that made it go to pot so to speak, it's possible that the 2acs limiting channels is it's saving grace in what should be its downfall. I guess if they could some way limit DS channels on a Hub 3 to 8 again, that would be as much of a work around.

Datalink posted some puma 7 ping plotters graphs a fair few pages back now, if you look in my posts I thanked him so can use that to find it. I was interested in the latency on docsis 3.1 vs 3.0 and didn't expect a response. For the record Xymox didn't care much for it, but I found it useful.

[RidingTheFlow]

---

@Guybrush85 wrote:
The understanding I have is the puma 5 and 7 aren't as bad as the 6. Since the DS channel increase seemed to be thing that made it go to pot so to speak, it's possible that the 2acs limiting channels is it's saving grace in what should be its downfall. I guess if they could some way limit DS channels on a Hub 3 to 8 again, that would be as much of a work around.

---

I doubt channels will fix everything, since SH3 (bottom graph) exhibits ~80ms spikes even on Ethernet local interface.

These spikes add to ones produced by WAN - therefore in modem mode (when your own router is pinged by BQM) max latency is higher than in router mode (when BQM pings SH3 itself).

[Guybrush85]

---

@RidingTheFlow wrote:

---

@Guybrush85 wrote:
The understanding I have is the puma 5 and 7 aren't as bad as the 6. Since the DS channel increase seemed to be thing that made it go to pot so to speak, it's possible that the 2acs limiting channels is it's saving grace in what should be its downfall. I guess if they could some way limit DS channels on a Hub 3 to 8 again, that would be as much of a work around.

---

I doubt channels will fix everything, since SH3 (bottom graph) exhibits ~80ms spikes even on Ethernet local interface.

These spikes add to ones produced by WAN - therefore in modem mode (when your own router is pinged by BQM) max latency is higher than in router mode (when BQM pings SH3 itself).

---

The way I'm seeing this is the increase in DS channels causes extra load on the CPU, so essentially whether or LAN or WAN side, the increase in latency will be exhibited. Unforunately I can't go back now and test on 8 channels again to prove it though.

Pinging my router in modem mode neither added or removed latency from my BQM or otherwise. 

[Adduxi]

---

@Guybrush85 wrote:

Now I can’t seem to match ‘the every 2 seconds spike’, but there’s certainly some odd spikes in there and enough to cause unwanted issues when gaming. I also wanted to prove this wasn’t just a fluke, so performed the same tests on a friends BT Infinity setup… then misplaced it.

---

Here's my BT Infinity graph, over wireless, UDP top and TCP bottom. It may help .....

UDP_TCP_Capture

[Guybrush85]

---

@Adduxi wrote:

---

@Guybrush85 wrote:

Now I can’t seem to match ‘the every 2 seconds spike’, but there’s certainly some odd spikes in there and enough to cause unwanted issues when gaming. I also wanted to prove this wasn’t just a fluke, so performed the same tests on a friends BT Infinity setup… then misplaced it.

---

Here's my BT Infinity graph, UDP top and TCP bottom. It may help .....

UDP_TCP_Capture

---

Thanks for that. Think best thing I can do is open a new thread and I'll post those details again and add those as well. This thread moves so fast it's hard to find that specific bit. Will do it later 🙂

[shanematthews]

---

@TheChairman wrote:

Virgin got shafted. No they didn't the issue was reported during testing but they decided to go ahead anyway. The customer was the one shafted.

Why they went ahead we can only speculate.

They are other cable modems out there running different chipsets ideally they should have trailed two and I'll happily be corrected if they did but that would have been the best business approach.

---

And i've already explained the likely reason for why they went ahead with the deployment

[shanematthews]

---

@itinfocus wrote:

---

@shanematthews wrote:

---

@Drewley wrote:

@shanematthews

If people other than yourself want to go to put in effort to try and resolve a problem they PERSONALLY feel is affecting THEMSELVES, why not let them be?

I really don't see your motivation behind putting in so much effort to discourage people from making noise about this issue.

---

I'm not telling them not to put in the effort, i'm telling them to put in the effort by complaining about the CORRECT company, in this instance intel who make the modem, intel is the one who is unable to fix it which as a result means arris can't fix it which in turn means VM can't fix it, VM have zero power over the completion of the firmware update to fix the issue, and this is assuming that the issue CAN be fixed via software at all, if its a hardware flaw then no amount of firmware updates will fix it, so yes i encourage you to complain but to the right people, whining to VM doesn't do anything

---

Seriously Shane, I just don't get where you are coming from on this.

As has been stated on numerous documented occasions. The very first trial group reported latency issues to Virgin with the Hub 3.0. How are Virgin not responsible for continuing with the purchase and mass deployment of this kit after the feedback?

For a poster so attached to logic and real world expectations your attitude is bizarre. If we all follow your advice of not putting pressure on OUR supplier I guarantee you the companies assessment will be that whilst documented our customer base is not unduly concerned as we have had no negative feedback.

To say that most customers are not experiencing issues is utter nonsense. All customers are impacted by reduced web browsing performance. Just because most customers do not understand that the constant delays and stutters are due to the Puma 6 and not how it should be, is not an acceptable defense.

Your arguments are logically opposed:

1) No problem as most non technical people don't understand the issue so just accept poor service and do not complain.
2) Don't complain to Virgin. It's not their fault. They are victims.

Rubbish, we are the victims. We are the customers, continuing to be charged at full price for a service that the supplier knows is not working as it should be. If Virgin decide to supply the service for free or a substantially reduced cost I would then accept their status as a victim. I'd even applaud them publicly and laud them as a paragon of proactive customer service.

Instead we have a company that is aware of the issues but continues to actively push a known defective product onto a ever larger percentage of the customer base.

---

As i've already said, they will have gone ahead because they would have been told that a fix was possible, and intel may have actually believed this was the case, with the benefit of hindsight it turns out that it wasn't possible to fix, so yes, VM got shafted, they arne't going to just take what the trialists say as the only truth, yes they were made aware that there was an issue, they would have communicated this back up the chain and as i've stated would have very likely got a response stating they were aware of the issue and it would be fixed, why would anyone doubt when a company like intel says they will deply a fix? they took a leap of faith based on intels good name and the previous modems they had made, unfortunately that was the wrong call, but they wouldn't have known that back then

[shanematthews]

---

@chand93 wrote:

By Shane's own 'logic' his constant defending of VM makes him either a undercover employee or someone with an association to the company.

---

lol by my logic?

Well you're free to believe what you want i guess, all i'm doing is looking at the wider picture with a pinch of common sense and some experience working in the tech industry, and no, not at VM before you ask

[TROTER]

As a gamer I just want any option to rectify this problem now until this mess is sorted out. Given that I pursued a SHUB2 months ago with complaints to head office with no success it further upsets me to find that it is indeed possible to reactivate a SHUB2 or AC. I have again requested a HUB2 from customer complaints and wait for a reply.

When it comes to informing this gaming community to the puma problem I think that this post is badly named and should be called "SHUB3 bad for Gaming because of the Puma chipset". I dont think "
Re: Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause" Draws in new readers especially with 130 pages to read. So after reading this topic for months now I will add myself to the list of disappointed and urge all others to do the same.
Please virgin give us gamers some options here?