Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

[Datalink]

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.

The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...

Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...

Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...

That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.

Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.

Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 [MOD EDIT: Subject heading changed to assist community]

[Scouts]

So, my third engineer visit today. Lovely helpful chap. Was well aware of the latency issue, and understood my issues with it, but I've been the first customer on his patch to actually complain about it. He said he wasn't aware of any timescales for a firmware update.

Fortunately he had a SH2AC on the van from another customer. He had to do some sweet talking on the phone to a few different people but he managed to get it re-assigned to him, the mac address changed on it  and I now have a shiny second hand SH2AC working on my desk next to my computer. I did hear him say on the phone when asked about it, "The customer has a very good reason for wanting this to be installed".

Unfortunately I think it may be luck of the draw, but I'm really happy.

So, it IS absolutely possible to get this changed by a visiting engineer, just don't give up.

Here's today's BQM - the latency difference between the SH3 and SH2AC is painfully clear.

[mobculture]

Great news. So happy for you!

[RidingTheFlow]

Scouts wrote:

Fortunately he had a SH2AC on the van from another customer. He had to do some sweet talking on the phone to a few different people but he managed to get it re-assigned to him, the mac address changed on it  and I now have a shiny second hand SH2AC working on my desk next to my computer. I did hear him say on the phone when asked about it, "The customer has a very good reason for wanting this to be installed".

---

... and I think that pretty much concludes the argument against "they can't activate/assign used hubs to other people". They are perfectly able to (as long as people in activation's stop being boneheaded about it).

Yeah, and that they are also screwing some clients over (by "upgrading" them and taking their hubs) to fix others...

[Scouts]

"Yeah, and that they are also screwing some clients over (by "upgrading" them and taking their hubs) to fix others..."

Not sure where you get that from - they are not taking SH2's to fix others, they (ie VM) are trying to take them out of circulation altogether. Today's engineer had it in the van as he'd done an upgrade and the previous customer had asked him to take the SH2 with him. Whilst I was assured on the phone to VM that there was a note on the system to bring a SH2AC today, then engineer who attended said there was no such note. I was lucky, I just had a decent engineer who listened and was able and willing to sort it.

[RidingTheFlow]

---

@Scouts wrote:

"Yeah, and that they are also screwing some clients over (by "upgrading" them and taking their hubs) to fix others..."

Not sure where you get that from - they are not taking SH2's to fix others, they (ie VM) are trying to take them out of circulation altogether. 

---

Not sure where you get this from either since it clearly was different from your experience. I don't buy these stories of engineers "just having SH2 around in the van by pure chance". They are perfectly aware SH2 is in demand. 

And do you really think they have informed customer they've taken SH2 from that its actually better? Bet not, most likely just sang how much "better" SH3 is, like they did to dozens of people in this thread.

And if they intend to just throw them away its no less silly - since they are tricking people to give away working hardware which is actually *better* just so they can "dispose" of it. 

I appreciate them trying to solve the problem (and I am ok with people having *informed* decisions to upgrade to SH3), but disinformation is not the way to go, all it does is creating a huge mess.

[steelysteph]

Let's not argue amongst ourselves folks, most of us are in the same boat on this thread and I think the frustration, lack of communication, and downright lies from VM are getting to us now.

[Scouts]

How do you quote a post on here?

"Not sure where you get this from either since it clearly was different from your experience. I don't buy these stories of engineers "just having SH2 around in the van by pure chance". They are perfectly aware SH2 is in demand. 

And do you really think they have informed customer they've taken SH2 from that its actually better? Bet not, most likely just sang how much "better" SH3 is, like they did to dozens of people in this thread.

And if they intend to just throw them away its no less silly - since they are tricking people to give away working hardware which is actually *better* just so they can "dispose" of it. 

I appreciate them trying to solve the problem (and I am ok with people having *informed* decisions to upgrade to SH3), but disinformation is not the way to go, all it does is creating a huge mess."

I'm not arguing, it's just I don't believe for a second that there are that many people complaining about this. If there were, then something would be done. This forum isn't a true reflection of every VM customer out there unfortunately. I have no reason to doubt the engineer today, he had no clue what he was walking into, or what the problem was. After I explained it to him and showed him the BQM graphs he understood perfectly and went to check his van. He didn't walk in with a SH2 under his arm and he had to make a lot of phone calls to actually get it swapped over. VM have adopted a policy, rightly or wrongly and their staff are generally following it through. If the majority of their customers were tech-savvy gamers I very much doubt we would be having this problem. VM are out of order introducing deficient equipment, but I don't think for a second there's this massive conspiracy you seem to be suggesting. It's just staff following orders.

[RidingTheFlow]

Lets not devolve into absurd conspiracy claims, etc. I never suggested anything like that.

But *somebody* had to adopt the policy to:

- Ignore the problem with SH3. It was know since it had *trials* and they have still decided to go ahead with it for goodness sake.

- Don't inform staff that taking perfectly working SH2 and replacing it with SH3 may lead to complaint and unhappy customer. Instead, misinform own staff that they should tell customers that SH3 is *better* and push people to upgrade.

- Misinform support that activation of used hubs is impossible (and even for cases when person kept their old SH2). Disallow activation of old used SH2s, even when nothing technically prevents it.

- Deal with complaints on absolutely arbitrary and ad-hoc fashion, resulting of "squeaky wheel gets the grease" policy instead of clear procedure. 

It does not come from low-level staff, but *somebody in management deliberately had to make all these decisions*.

And don't fall into VMs trap of trying to convince you that "its vocal minority". Problem is widespread, they are just trying to downplay it by any means possible, this thread didn't become 100 pages long just from dozen of unique tech-savvy people. When there are so much complaint on forum, you can only imagine that in the field there are even more. Also, since its such a pain to get it fixed, some people just prefer to change ISP and be done with it.

[Xymox]

Well its a good question how we ended up with sucj **bleep** modems and no one caught this. Arris, Linksys, Hitron, etc... They ALL did exactly the same thing VM did. Then there are the ISPs. All over the world.. Here in the US all the major ISPs knew in january there were very serious and confirmed issues with all Puma 6 based devices. Yet ALL of them world wide have continued roll of of these deices and, far worse, they are rolling out technologies that can only be implemented using a Puma 6 device. CableLabs failed their mission miserably. As the world wide industry standards body and testing center, they were the single worst single point of failure that has us where we are. Latency and jitter have no specs and testing criteria. This is a horrendous failure of a standards body. A chip maker who had NEVER made a cable modem chip in its history was suddenly allowed to be the exclusiove supplier of devices for entire countries with thier first chip that had no prior testing or a in field proven record. **bleep**. Intel has a SERIOUS history of anti-trust. They literally paid PC vendors to not use AMD. Worldwide. . It seems to me, thats what could have happened here. Maybe somehow Intel paid the right people to take over the cable modem market ? Intel is desperate for new markets as the PC market is dwindling and they have no footprint in mobile. Intel has a past history of paying people under the table to take over a market. Maybe thats exactly what got all these vendors and MSOs to adopt their product ? So... This is a MUCH bigger issue then VM. VM acted just modem vendors and other world wide ISP. The STUNNING issue is they are still shipping these devices. All these vendors and ISPs appear to believe Intel that they can fix it with a firmware update. If they could fix it, they would have by now.

[RidingTheFlow]

---

@Scouts wrote:

Whilst I was assured on the phone to VM that there was a note on the system to bring a SH2AC today, then engineer who attended said there was no such note.

---

Btw, this what also happened to me. Was assured by our own forum staffer Nat_J that they will clearly mark on work order to bring SH2.

Then engineer came and claimed that there were no such note for my order. Instead he shown me couple of work notes for some previous orders on his PDA which said "if customer has SH2, replace with SH3".

So either somebody *ahem* *exaggerated the truth*, but mind boggling, why?

Or system automatically directs notes about SH2 downgrade requests to the black hole - now that I'd call conspiracy theory 😉