Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

[Datalink]

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.

The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...

Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...

Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...

That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.

Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.

Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 [MOD EDIT: Subject heading changed to assist community]

[Guybrush85]

---

@shanematthews wrote:

---

@RidingTheFlow wrote:

---

@shanematthews wrote:

---

But as i've said by all means complain to ofcom that the device actually provides you the service detailed in your contract, yes i appreciate you want a place to vent your annoyance at, vent it at the people who will care and be able to do something about it, intel

Umm, according to Intel issue was fixed for months. Its Virgin who slow with rolling out the fix. Or maybe Arris, or Compal - couldn't care less really. I am buying services from Virgin and will take issue with them.

They(Virgin) said "we about to start updated firmware trials shortly" back in March! (http://www.ispreview.co.uk/index.php/2017/03/virgin-media-prep-firmware-fix-superhub-3-latency-packe...). Now they just've repeated same thing on this thread - so for all intents and purposes there is no real change. Intel claims its fixed already, Virgin claims it will be fixed "soon" (for 3.5 months already). And its still not fixed.

If, for example "gaming package" ads clearly lead you to believe its superior for gaming - as a customer you don't even have to make yourself into network and latency expert - you should just expect VM to know these things and make good on their promise of good gaming experience. If instead their package gives you random lag which makes you miss random shots - as a consumer its not your fault that you didn't make full investigation before finding this small print, its a fault of a party that hidden this deceptive small print in first place.

---

No, intel had a partial fix for some of the other stuff months back, the full fix is only a recent thing, the partial fixes wouldn't have helped and they didn't want to bother with double certifying and testing so waited for the actual fix

The gaming package is incorrectly named but the package itself is fine its just the hub that sucks, i have the package on a hub2 and it provides all it claims it offers with no upstream traffic management, it should be renamed the streamer package as its more fitting for what it actually provides, the package itself however works

Also, trials can fail, just because they tested it doesn't mean it worked properly

---

At this moment in time there is no full fix. The patch which it's most likely what will be coming to the hub 3 is some variant of the 93V patch, which is not a complete fix.

As for the trials, I'm aware of no trials going on prior as of yet, both internally and otherwise. 

[Capt_Sarcastic]

There's a lot of talk that the orginal "fix" simply fixed the issue for DNS lookups and UDP.  This would have made the problem disappear on tests, but would have left the issue silently happening for actual data traffic.  It seems to have been an idea to simply hide the latency issue, rather than fix it.  This would go a long way to explaining the delays.  Along with this is the path for the fix is for Intel to make it, then it goes to Arris to be integrated into their firmware for various models, then to Virgin for a test group, before final roll-out.  At each stage I'd hope there's a testing period.  All of this is taking far too long to happen, but that's why.

[Guybrush85]

Feel like it may be worth a bit of a... where are we at update. Albeit, this is all based on my research and knowledge and by no means official.

So the puma 6 issue has been known about for some time, but only really when Xymox pushed it into the public light did we get any kind of movement on it. Since then across the pond there have been 2 firmware patches. The first one fixed ICMP spikes, the second (93V for that particular device) the only thing I can say for certain fixed DNS (UDP). Now those with that patch, including the Hitron used by Virgin Media Business have reported better results using the puma 6 test on the DSLReports website and other tests, so what exactly is fixed is open for debate. The patch notes released by Netdog don't help. Regardless of what exactly it does, it's a step forward from where we on the hub 3 are now.

Not long after this Mackey (I believe) discovered a fault that meant that the Hub 3 could suffer a DOS from a very low powered attack (as low as <5Mbit). My opinion is whilst this is certainly a bad thing, for me, I'm not suffering with it and most likely you're not either. I think the majority of attacks are done from high bandwidth/distributed methods, so unless someone was targeting a subnet, the scale of the issue means you're more susceptible sure, but most likely it wouldn't make a difference anyway. That said, there were a small number of reports of people suffering with the DOS, or at least a degradation of services simply down to having networked cameras with the large amount of UDP connections.

Since then, most likely due to the on going legal battle in the states, the user Netdog - an Arris employee who has been the only person giving us anything has gone silent. Another user, who we believe is confirmed to be from Intel came onto the forums and posted to say a full and complete fix (we asked, but got no response on the exact details of that). Since then though, we've not heard anything, or seen any new patches. This again is probably down to the on-going legal battle, but some believe that said patch didn't pass testing/fix anything, so that's why we've heard nothing.

So where are us Hub 3 users. Currently, no futher on, but as James_W stated a trial of a new firmware is due soon. Again, it is my belief that the firmware version will just be the same as that of the 93V patch.

I personally have done tests with my current setup using the usual tools (BQM, Ping, MTR and Traceroutes) and some more advanced stuff like Ping Plotter, so can compare before and after the patch. Setup is 24 DS, 2 QAM 64 US on a c-BR8 in modem mode.

As for the whole Ofcam debate. Look, I think that the handling of the issue from VM's part has been bad. I think an acknowledgement with some sort of statement to say, ok, we're aware of this issue and we're working our manufactures (Arris, maybe Intel directly I'm not sure) and we will deliver what we can when we can. I can see how doing this would open the door for compensation etc, so I think the management decision was... stay quiet until we can deliver. The forum staff are simply told what to do, so frustrations sent that way are injust. Back to the Ofcom part, there's possibly a case based on the handling of it, but as for the 'who's to really blame', that is in indeed Intel, or at least intel and manufactures who's testing seems to have missed the trick.

So in summary, let's hope the trials start soon and the outcome is good. Whatever the result, it's a step forward and hopefully, being outside the US we're not stuck under the umbrella of this legal battle and maybe that works in our favour and we get something new (I'm sceptical to say the least).

[fat4l]

thats funny. Over 1000 pages and still nothing 🙂 cool VM. Ya r da best

[Xymox]

Great job Butler85 in summing up.. Maybe you could come copy paste that on the badmodems forum ?

You have brought up a point.. I need a timeline of events on the main web site..

Everything you said above is in order and accurate.

The only thing I can contribute to that is one of the top guys at Technicolor in france sent me a email telling me that had received a firmware patch from Intel that was supposed to fix the DoS issue and other things. They had not done testing or any work on it when he sent that. I assume that went out to all modem vendors. We dont know what is in that. That firmware has never seen the light of day. We dont know what happened to that. That was May 31st.

_______

Hi Chris,

Just to keep you informed : Intel has announced some firmware patch for this flaw. We'll have to implement and validate it in our products, as soon as it is available.

_________

Intel Product security last update was on may 9th. I had heard from a little birdie that there was a announcement made and i was asking them for a update.

_________

Chris,

 We have assigned a CVE # to this issue.  It will be CVE-2017-5693.  It will be submitted to NIST/NVD in accordance w/ CVE submission policy.

Unfortunately, PSIRT is not at liberty to divulge information of Intel’s communications w/ customers.

 Thanks & regards,

Intel® Product Security Incident Response Team (PSIRT)

www.intel.com/security

secure@intel.com

 ______________________________________

So.. There is no patch for anything as of today that anyone knows of. The above is the latest info I have on this.

[Xymox]

The CVE has never been filled in and Intel said that before they fill it in the CVE needs to be validated by ALL vendors who have EVER used the chip... IT it wont ever happen..

The CVE might say that a hardware flaw is present. This would doom every Puma based product ever made and trigger all sorts of backlash like lawsuits and recalls. So I really want to see the CVE... However Intel has every reason to hide it forever..

The class action lawsuit might force it to pop out..

[fatalist88]

Any update to this abortion of a service?

[alexsim2004]

I can't quite believe what I've read here - over 6 months of knowing about a critical Denial of Service vulnerability and still absolutely nothing done about it despite a fix made available upstream! - This is worse than disgraceful Virgin (and shame on Intel too).

Please include myself in the upcoming firmware trials, I would be very interested to try it out and see if it's really fixed or just swept under the rug.

[cje85]

There is a patch which improves the latency problems to some degree but the DoS issue has yet to be patched by Intel. Virgin can't do anything until if/when that happens.

[planetf1]

I just thought I'd check my SH3 network status (I'm using modem mode with An Asus AC3200) and the page at 192.168.100.1 has gone all blue and green. Looks completely different to the virgin red. (it's actually rather nice)

Wondering if I've been pushed updated firmware........... but I have no idea where to find the version, without switching out of modem mode?

It's also possible it's a bug of some kind and it's reverted to arris defaults or similar.

I've not touched the hub... and it's working fine.

UPDATE - checked y f8lure logs. No change in latency, and hubs been up for 15 days. Just weird. False alarm sadly.