Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

[Datalink]

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.

The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...

Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...

Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...

That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.

Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.

Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 [MOD EDIT: Subject heading changed to assist community]

[RidingTheFlow]

---

@TROTER wrote:
Given that I pursued a SHUB2 months ago with complaints to head office with no success it further upsets me to find that it is indeed possible to reactivate a SHUB2 or AC. I have again requested a HUB2 from customer complaints and wait for a reply.

---

As long as you followed proper formal complaint procedure and it wasn't resolved to your satisfaction, you should've pursued further with third-party arbitration (CICAS).

Unless you've agreed with Virgin that they've closed the complaint - in that case you will need to open a new one.

[Datalink]

@Guybrush85, great post. Your 16 to 24 channel BQM is a very good example of what happens when you increase the processing load on the CPU. We run 32 channels on the Rogers network, so the high pings reach the upper 200 ms range. Much worse than what you see on VM's network. That was my starting point at the end of May 2016, new Puma 6 - 32 channel modem, new CASA Systems CMTS, and outrageous ping times to the CMTS. It didn't make any sense at the time. That was Xymox's reaction as well 6 months later. To recreate the plots that I and Xymox can plot, you need to run Pingplotter with interval times less than 0.5s. 0.1s should do the trick. You can experiment with the interval times to see what works for you, given the combination of your pc processing capability, modem and CMTS. The key is low interval times and low plot timespans. The problem with low interval times/high repetition rates is the fact that Pingplotter averages the plot times on the graph. That averaging depends on the number of data points in a given horizontal pixel range. The higher the number of data points, the fewer number of high pings you actually see due to the averaging. That is very noticeable when you run 100ms or less for interval times. The only way to get around that is to restrict the plot time that you see to 60 seconds or 5 minutes at the most. If you have the Max time shown in the upper text data area, look at that Max figure and compare it to the max time shown in the plot area. If they're the same, then there is no averaging taking place. If you can see that the max plot time is less than the Max data text figure, then you know that for the time scale you have selected, Pingplotter is averaging the data and therefore you lose the capability of seeing the high times on the plot area. Only thing you can do is step down in plot times displayed until the Max text data matches the maximum plot time. If you don't have the Max time shown in the upper text area, right click on the upper data title bar to bring up the selection menu and select MAX. Drag that to the right so that it sits near or next to the MIN time so that you can compare the two figures very easily.

The data averaging is very problematic at this point. One way around this is to run Wireshark and use the STATISTICS .... IO Graph to plot the data. Wireshark does not average the data, so, over the course of 24 hours, this becomes a totally different graph. Wireshark is a free application that takes a little experimentation to display the data. If you want to try this please let me know and I'll post instructions to do just that.

Another point of interest is the problem with running a UDP test. Pingplotter runs a UDP ping to a target. Unless you have a co-operative target, the normal result is an ICMP Destination Unreachable return from the target itself, or an ICMP Timeout from an enroute server. It's not UDP, out UDP in. So, the pingplotter result in this case is not accurate or truthful, but, there is no indication from Pingplotter itself. The real UDP test to use is a UDP query to your ISP Domain Name Server, as normally occurs when you enter a web address into a browser address bar, but do this on a continuing basis. Easier said than done, but, I've stumbled across Dns Monitor, which can be used to transmit continuous UDP requests to the DNS server, just like Pingplotter transmits continuous pings to a target.

http://www.tallsoft.com/dnsmonitor.htm

Its only IPV4 capable from what I've seen so far and has a minimum of 1 second intervals, but, the result is UDP requests out and UDP responses back from the DNS. Captured with Wireshark, you can then export the data from Wireshark into Excel and plot the result. Wireshark can plot the data, but appears to have a bug in the latest version where some of the data points are plotted at 0ms instead of the real data point. The data it correct, but the plot isn't totally correct. Excel will plot the data that its given. So, that might be food for thought if you're looking to see what the real result looks like when it comes to running UDP thru your Hub 3. I'm still looking for a Windows IPV6 equivalent to Dns Monitor. Too bad I don't run a linux box.

@RidingTheFlow, one point to keep in mind is that there are two issues afoot here: LAN latency, and LAN to WAN throughput latency. If you ping a Puma 6 modem which has not been loaded with the necessary update, you will see random high ping times, just to the modem. That is not the same issue as the throughput latency although the two might be related in some technical details and processing. Rogers fixed the LAN latency at the end of May 2015. Too bad I didn't realize at the time that there was much more that we didn't understand at the time. Running 16 and 24 channels modems at the time, the latency wasn't apparent or I wasn't looking hard enough to see it. There may be an additional processing issue with Telephony modems, whereas the Rogers modems are straight Gateway modems.

@Adduxi: nice comparison with the BT graphs. Keep in mind as I indicated above, with Pingplotter, the upper UDP plot is actually UDP out, ICMP destination unreachable reply or ICMP Timeout response back to Pingplotter. You would need to monitor this with Wireshark to filter out the enroute Timeouts and plot just the ICMP Destination Unreachable which actually comes from your target.

@TROTER: The thread title, Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause.

At the time that I posted the first post, I was more concerned about every modem that ran a Puma 6 chipset, not just the Hub 3. A Hub 3 title of some type is more appropriate to the gaming sub-forum, but, would exclude the other modems. That would require a second thread in another sub-forum, so, no single answer meets all requirements. For now, this seems to work. Your point is well taken however.

[nickking]

I've had my phone call back from the CEO's office, to summarise.

• They no longer support or supply the superhub 2ac, customers aren't asked to return these and are asked to recycle them, actively pushing the Hub 3.0 despite the issues.
• Knocked £15 a month off my bill (this isn't compensation of any sort just a better deal for our services).
• Firmware update 4th quarter of 2017.
• Despite earlier posts stating otherwise the powerline adaptors provided by Virgin Media have helped, the devices that were wireless and suffering the most now stay connected and online gaming has improved.  The issues are still there, I'm fully aware of that, but where I was getting a constant problem on Xbox live disrupting gameplay this is now to a minimum.

Hopefully they are true to their word and a firmware update is just around the corner.

[grindax]

Apart from poor latency for gaming, it appears DNS name resolution via VPN connections is another area where performance really suffers with the Hub 3.0.

A few days ago a Virgin Media engineer replaced our SuperHub 2 with the Hub 3.0 after we kept getting frequent disconnects. The engineer said the Hub 3.0 would likely offer a more stable connection. (The infuriating thing is that just prior to him replacing the SuperHub 2 I said to him "I'm sure I'd read in the past about performance and latency problems with the Hub 3.0 -- is that all resolved now, and will this actually be an upgrade?", and his answer was "Yes, there aren't any issues with the Hub 3.0".)

Now I'm reading all the ongoing horror stories about the latency and packet loss associated with the Hub 3.0. I can easily see the packet loss issue by simply continuously pinging any internet host (such as VM's own DNS servers) and a certain percentage of those pings (ICMP packets) get lost. Not good.

But the worst thing I've noticed is that if I connect to a VPN service (such as Hotspot Shield), then the performance of DNS name resolution via the VPN connection is horrendous. Every time I try to visit a new URL, I see the message 'Resolving host...' in Chrome's status bar for 5-10 seconds before anything happens. This never happened with the old SuperHub 2.

I wonder if the upcoming firmware patch will do anything specifically to address VPN performance.

[AlteranAncient]

I've been told time and time again that this patch is "just around the corner". It was in the press as far back as March that a "fix was in the works", and I was then told back in June that a fix was expected "around July or August.". Now we're getting something as vague as "Q4 of 2017", so "some time between October and December". One year from "we've discovered an issue" to "we've fixed the issue" is pretty poor for an ISP with millions of end users potentially being affected. What's worse is that the problem was known since December last year, yet they continued to roll out the devices to new or upgraded customers knowing they were faulty. That sounds like grounds for a class action to me.

My connection is usable for day-to-day stuff, but I start to notice problems when gaming or streaming 4K content. Given that in my case, I was given the Hub 3 when switching over to the "Gamer" package, I've been sold a product that is literally not fit for purpose.

The sparsity of information or feedback from Virgin Media only gives me the impression that they don't actually care about this issue, and the empty promises of a fix are just to tide the complainers along until their contracts expire.

[wotusaw]

AlteranAncient wrote:

"

I've been told time and time again that this patch is "just around the corner". It was in the press as far back as March that a "fix was in the works", and I was then told back in June that a fix was expected "around July or August.". Now we're getting something as vague as "Q4 of 2017", so "some time between October and December". One year from "we've discovered an issue" to "we've fixed the issue" is pretty poor for an ISP with millions of end users potentially being affected. What's worse is that the problem was known since December last year, yet they continued to roll out the devices to new or upgraded customers knowing they were faulty. That sounds like grounds for a class action to me.

My connection is usable for day-to-day stuff, but I start to notice problems when gaming or streaming 4K content. Given that in my case, I was given the Hub 3 when switching over to the "Gamer" package, I've been sold a product that is literally not fit for purpose.

The sparsity of information or feedback from Virgin Media only gives me the impression that they don't actually care about this issue, and the empty promises of a fix are just to tide the complainers along until their contracts expire."

This is EXACTLY HOW I FEEL!!!!

I take 2/3 seconds to load pages from my clansite now. That's 1 second....2 second...3 seconds. This happened immediatlely I changed over the hubs from a HUB 2 to a HUB 3 way way back in February. I still play Titanfall 2 but have to camp alot. However I've lost heart in it now as it's just so difficult to play with the spiking latency problem.

To be honest I don't think it can be fixed. The great and powerful wizard 'XYMOX1' has said so and I believe him.

Sure, they'll do something to try and shut us all up. Recon 5/10% improvement at best.

Anyway we'll see, won't we in quarter 4, whatever that's supposed to mean.

I'm so angry right now!!!!

[rio_w]

I don’t think it can be fixed if the hardware is flawed, I’m hoping that having spoken to a management I’ll be sent a SH2AC this time round. If not, I’m leaving since there’s the upcoming price increase.

[Spitfire16]

so we have gone from

March 2017 - Testing to "begin shortly"...

August 2017 - Firmware Expected Q4 2017?

Meanwhile we all suffer

Nice 

[RidingTheFlow]

---

@nickking wrote:

• They no longer support or supply the superhub 2ac, customers aren't asked to return these and are asked to recycle them, actively pushing the Hub 3.0 despite the issues. 

---

While its maybe true they may not have ones at the moment to supply, "no longer support" is a plain outright lie - since even Superhub 1 is still perfectly supported and still used by thousands of clients. And if they truly ask people to throw their SH2 away, this is just plain idiotic - on purpose getting rid of hardware which is actually *works better*, just for sake of being pig-headed to not admit the problem with SH3.

@nickking wrote:

• Knocked £15 a month off my bill (this isn't compensation of any sort just a better deal for our services).

Well, at least that's not a bad discount at all. I definitely would've settled for it.

@nickking wrote:

• Firmware update 4th quarter of 2017.

Ahh, so that's "shortly" in Virgin's time - any time before Christmas 🙂

@nickking wrote:

Hopefully they are true to their word and a firmware update is just around the corner.

Wouldn't count on it. "True to their word" seems to go extremely against their whole nature.

[Guybrush85]

Just remember, there's only been 1 official (on this forum) mention of any firmware patch and that was about 4 weeks ago when James_W said a trial was due 'soon'. 

If the CEO's office says Q4 for patch, I would take from that, that they mean for public release, so the trial still needs to happen prior to this.