Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

[Datalink]

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.

The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...

Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...

Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...

That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.

Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.

Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 [MOD EDIT: Subject heading changed to assist community]

[Andrew-G]

Skunkfu: "there are talks going in their office about SH4 in the near future"

I'm sure there will be and already are, but who knows how long before customers every see it?

The lead time on procuring up to six million modems with a combined price tag around £180m is going to be pretty long, and there will be extensive negotiations over the spec and the price.  Currently a DOCSIS 3.1 modem router would retail for about $120; buying in bulk as a trader would halve that, but its still double the sort of amount (maybe more) than the Hub 3, so to get the costs down Virginmedia will be cutting every corner, to the extent that I expect the Hub 4 to be completely spherical.

[RidingTheFlow]

---

@Andrew-Gwrote:

so to get the costs down Virginmedia will be cutting every corner, to the extent that I expect the Hub 4 to be completely spherical.

---

Actually VM hardly have any say in the matter anymore. As it was with SH3, they will use whatever LG tells them to. 

SH2AC was the last Superhub which was procured by VM itself (from Netgear). And that was pretty decent.

[philjohn]

Quick request from people with the updated and old firmware - wonder if you could run a few tests using iperf3[1]?

Specifically, iperf3 allows you to test send and receive jitter over UDP, once downloaded and extracted could you please open a console/terminal and run:

iperf3 -c ping.online.net -p 5202 -u -R -V -t 120 -i 10

(this tests downstream UDP for 120 seconds)

iperf3 -c ping.online.net -p 5202 -u -V -t 120 -i 10

(this tests upstream UDP for 120 seconds)

If you could then paste the results (make sure to censor your own IP address) that would be great and would give a better idea of jitter and packet loss.

As an example, here's mine on a BT FTTC 80/20 connection downstream - this is from my router, which is Linux, but there are Windows and MacOS builds on the homepage of iperf:

root@router01:~# iperf3 -c ping.online.net -p 5202 -R -u -V -t 120 -i 10
iperf 3.1.4
Linux router01 4.4.110 #0 SMP Tue Jan 16 08:02:39 2018 armv7l
Time: Fri, 16 Feb 2018 21:38:01 GMT
Connecting to host ping.online.net, port 5202
Reverse mode, remote host ping.online.net is sending
      Cookie: router01.1518817081.140282.6a8336404
[  4] local xxx.xxx.xxx.xxx port 50852 connected to 62.210.18.40 port 5202
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 120 second test
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  1.26 MBytes  1.06 Mbits/sec  0.782 ms  0/161 (0%)
[  4]  10.00-20.00  sec  1.25 MBytes  1.05 Mbits/sec  0.724 ms  0/160 (0%)
[  4]  20.00-30.00  sec  1.25 MBytes  1.05 Mbits/sec  0.687 ms  0/160 (0%)
[  4]  30.00-40.00  sec  1.25 MBytes  1.05 Mbits/sec  0.654 ms  0/160 (0%)
[  4]  40.00-50.00  sec  1.25 MBytes  1.05 Mbits/sec  0.646 ms  0/160 (0%)
[  4]  50.00-60.00  sec  1.25 MBytes  1.05 Mbits/sec  0.651 ms  0/160 (0%)
[  4]  60.00-70.00  sec  1.25 MBytes  1.05 Mbits/sec  0.635 ms  0/160 (0%)
[  4]  70.00-80.00  sec  1.25 MBytes  1.05 Mbits/sec  0.621 ms  0/160 (0%)
[  4]  80.00-90.00  sec  1.25 MBytes  1.05 Mbits/sec  0.638 ms  0/160 (0%)
[  4]  90.00-100.00 sec  1.25 MBytes  1.05 Mbits/sec  0.710 ms  0/160 (0%)
[  4] 100.00-110.00 sec  1.25 MBytes  1.05 Mbits/sec  0.648 ms  0/160 (0%)
[  4] 110.00-120.00 sec  1.25 MBytes  1.05 Mbits/sec  0.618 ms  0/160 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-120.00 sec  15.0 MBytes  1.05 Mbits/sec  0.618 ms  0/1921 (0%)
[  4] Sent 1921 datagrams
CPU Utilization: local/receiver 0.4% (0.0%u/0.4%s), remote/sender 0.1% (0.0%u/0.1%s)

[1] https://iperf.fr/iperf-download.php

[philjohn]

Just posted a useful guide to properly test UDP jitter and packet loss - looks like the mods have removed it.

If anyone with the old and new firmwares would be willing to PM me I can send you the instructions to try yourself, it's using a commandline program called iperf3 that's freely available for Linux, Windows and MacOS.

[shanematthews]

---

@Sephirothwrote:

---

@shanematthewswrote:

---

@nickkingwrote:
Good morning all,

I'd like to share with you an excerpt from a letter I received from the Virgin Media complaints department after contacting the CEO's office regarding the Hub 3.0.

"I am sincerely sorry for the overall experience you have described on the account. I appreciate the frustration this has caused you. I would like to confirm that Virgin Media will not be changing the super hub 3 based on the issues you have raised. The super hub 3 has been passed as fit for purpose to which it provides broadband connection on a residential contract. The equipment has been passed fit and satisfactory for purpose by our regulators"

They also go on to remind me our contract ends in August, a sort of if you aren't happy then leave statement if ever there was one.

Obviously I've replied on how disgusted I am with their response and lack of understanding of the issue.

---

Oh they "understand" the issue, but technically speaking he is correct, the hub 3 DOES meet the requirements laid down in the contract you agreed to, latency is not guaranteed and the hub 3 is more than able to provide a connection at the advertised speeds so it is "fit for purpose", sure it doens't provide the lowest latency but until thats actually part of a regulation or advertised as being within a certain range, neither of which is ever going to happen, then they aren't going to spend a fortune replacing the device with something else

---

I dispute what Shane says - though only a legal case would show who's right.

The contract is not allowed to trump the advertising you that induced the purchase of VM's service.  http://www.virginmedia.com/shop/broadband/vivid-gamer.html comes to mind.

If they knowlingly put out a flawed device that defeats their claim " It’s broadband for gamers . For the hardcore. For the best. ", then they are liable at law, imo.   It needs testing - a bluff might do it like getting a solicitor to write to them. 

I'll bet there hasn't been a word from VM as to what the new firmware actually cures.  Because it cures nothing - it has merely shuffled the deckchairs.

---

The "broadband" is perfectly suitable for gaming on, the hub3 isn't, you also need to define what is or isn't required as games come in many shapes and sizes, the hub3 will have zero to minimal effect on anything not latency intensive, the likes of grand strategy or casual games, most MMO's also aren't going to be impacted by the tiny microstutters, hell i know an MMO that will still work perfectly fine over an actual dialup connection, so i would be expecting the outcome of that to not go in your favour, especially considering most judges likely aren't gamers so won't understand the issue, the advertising doesn't make any mention of latency either so again ,thats not false advertising

[shanematthews]

---

@Luke_113wrote:
@hi @shanematthews
I'm sorry but thats a defeatist way of looking at things, and complaining and raising things to the right people is how change occurs, sure I'm certain something like this isn't going to be at the top of these MPs lists, but look at it like this, its an issue that affects potentialy millions of people across the UK, if it gets some REAL publicity such as Watchdog etc... VM wouldn't need to be forced to change it they'd do it willingly, how many parents do you think just buy 300mb because their children tell them get this one get this one its good for gaming, and their kids may understand about ping etc... if people realise that they've been conned when they've signed up for the 'gamer' package and that a half the price BT line would give their child a better gaming experience what do you think they'd do? Companies like Virgin, BT etc... it matters greatly because people do just leave companies from being disgusted by the way they do business it reflects badly on their credability, I mean you only have to go as far as the BT forums to see the level of outrage that phone line and broadband customers are funding the stupid football rights (partly why I left and actually spend less with Virgin Media Business)

Its about publicity, sure my mother doesn't understand the technical side of it, but when I said imagine buying a Ferrari but it takes 20 times as long to change gear, wheres the fun in driving that? Or you've paid for a access to a 4 lane motorway but actually you've only been given access to 1 lane of that motorway and told thats what you get or you can go? (in relation to Intel Puma 6 issues, and VM's generally over saturated network, I know they're not perfect examples but close enough) She absolutely understood those and agreed that it was shocking that they've known for 2 years and done very little to correct it - despite it being passed by regulators. I mean you only have to look through and find the really sad story of the guy on here who's been going 4 years on the same fault to see how much VM really don't want to spend money if they can't recoupe it in a small amount of time, until a few years ago Virgin didn't believe enough people would sign up on the estate that I live on and so refused to even do surverys etc... then realised that at least half of the estate get sub 40mb from DSL services and all of a sudden realised money could be made and within weeks we had fliers coming through the door.

Virgin won't learn till someone in a position of power publically talks about this, embaresses Virgin and makes them appear to be idiots on a national level, WHEN (not if) that happens they'll pull their finger out, because as is they have a bunch of customers over a barrel and have no incentive or motive to fix things that do a job just about, just like in a monopoly, and the government loves to hate on a monopoly (see Openreach)

---

Its already had publicity, the regulators have already looked at it, they agreed that as nothing was wrong with the network and that you were getting what you were contractually sold there was nothing they could do, at this point nobody can really do anything because your embarrassment would just be countered with "You're getting what you were sold" you could go contact some news outlet and i'm willing to bet they wouldn't even bother with it, the technical details required to explain it will go over most peoples heads and then, like i said, contract doesn't guarantee latency

Your examples are also horridly flawed, you're still getting all 4 lanes that you paid for (bandwidth) and for the vast majority of traffic a little extra latency isn't going to affect how fast anything loads, latency only affects responsiveness in applications that require low latency, browsing isn't one of them

As for a monopoly, do you honestly think they care about VM at this point, they have been the only major cable provider for decades, the reason openreach gets hit is because ADSL providers have to use openreach owned lines, VM isn't leasing their lines to anyone else they are used by only them, its not a monopoly if you're the only person using the infrastructure that you installed and paid for 😛

[shanematthews]

---

@Luke_113wrote:
@and this is what its going to take before Virgin do anything thanks @Skunkfu for giving us that, they are alienating gamers but at the same time trying (IMO) to defraud them with a 'gamer' aimed package!

---

Heres the thing, the "package" works perfectly for gamers, its what i use, just with a hub2, the hub3 is garbage but claiming the 200 gamer is bad is just flat out wrong

[Datalink]

@philjohn don't know if you're aware of this, but, you have to be running iperf 3.17 or higher for accurate UDP testing with Windows operating systems.  3.16 and below will indicate higher than actual UDP losses.  

The iperf site itself only has V3.13 for Windows, which won't do any good for anyone in this regard.  Newer Windows compatible builds can be found at:

https://www.neowin.net/forum/topic/1234695-iperf-34-windows-build/#comment-596631033

The latest build on the site is V3.4 for 32 and 64 bit Windows operating systems.  Anyone else using CentOS Linux, FreeBSD, and macOS and other Linux systems should refer to the Githib site for iperf:

https://github.com/esnet/iperf

Fwiw, running iperf on a gigabit service, the best that I've ever done without any losses is ~600 Mb/s on a Puma 7 modem.  Above 600 Mb/s, you start to see losses on the download side, which will grow as you push up the data rate.  I've also found that running iperf at high data rates requires one to push up the receive window, or, the result will be UDP download losses at low data rates.  On a 300 Mb/s service, perhaps that problem isn't noticeable?  Don't know.  Here's the commands that should work with the Netherlands speedtest.serverius.net site for UDP testing:

Receive test:

iperf3 -c speedtest.serverius.net -p 5002 -u -l 1k -b 300m -w 510M -t 31 -i 1 -f k -R -V --get-server-output

Transmit test:

iperf3 -c speedtest.serverius.net -p 5002 -u -l 1k -b 30m -w 510M -t 31 -O 1 -i 1 -f m -V --get-server-output

The test bandwidths are 300 Mb/s on the receive test and 30 Mb/s on the transmit test.  Those can be adjusted up and down to determine the effect that changes in the data rates will have on the UDP datagram losses. 

[Piers]

---

@Badvokwrote:

---

@ShadowOfDeth69wrote:
@RidingTheFlow How is your minimum latency so low? My building is only feet from the cabinet but my minimum latency is in the high 20s.

---

Minimum latency for TBB BQM depends on how far away you are from London, where the pings come from, not your cabinet.

---

TBB uses four-five datacentres in three locations:

- London Docklands

- Manchester

- Surrey

[Sephiroth]

---

@shanematthewswrote:

---

@Sephirothwrote:

---

@shanematthewswrote:

---

@nickkingwrote:
Good morning all,

I'd like to share with you an excerpt from a letter I received from the Virgin Media complaints department after contacting the CEO's office regarding the Hub 3.0.

"I am sincerely sorry for the overall experience you have described on the account. I appreciate the frustration this has caused you. I would like to confirm that Virgin Media will not be changing the super hub 3 based on the issues you have raised. The super hub 3 has been passed as fit for purpose to which it provides broadband connection on a residential contract. The equipment has been passed fit and satisfactory for purpose by our regulators"

They also go on to remind me our contract ends in August, a sort of if you aren't happy then leave statement if ever there was one.

Obviously I've replied on how disgusted I am with their response and lack of understanding of the issue.

---

Oh they "understand" the issue, but technically speaking he is correct, the hub 3 DOES meet the requirements laid down in the contract you agreed to, latency is not guaranteed and the hub 3 is more than able to provide a connection at the advertised speeds so it is "fit for purpose", sure it doens't provide the lowest latency but until thats actually part of a regulation or advertised as being within a certain range, neither of which is ever going to happen, then they aren't going to spend a fortune replacing the device with something else

---

I dispute what Shane says - though only a legal case would show who's right.

The contract is not allowed to trump the advertising you that induced the purchase of VM's service.  http://www.virginmedia.com/shop/broadband/vivid-gamer.html comes to mind.

If they knowlingly put out a flawed device that defeats their claim " It’s broadband for gamers . For the hardcore. For the best. ", then they are liable at law, imo.   It needs testing - a bluff might do it like getting a solicitor to write to them. 

I'll bet there hasn't been a word from VM as to what the new firmware actually cures.  Because it cures nothing - it has merely shuffled the deckchairs.

---

The "broadband" is perfectly suitable for gaming on, the hub3 isn't, you also need to define what is or isn't required as games come in many shapes and sizes, the hub3 will have zero to minimal effect on anything not latency intensive, the likes of grand strategy or casual games, most MMO's also aren't going to be impacted by the tiny microstutters, hell i know an MMO that will still work perfectly fine over an actual dialup connection, so i would be expecting the outcome of that to not go in your favour, especially considering most judges likely aren't gamers so won't understand the issue, the advertising doesn't make any mention of latency either so again ,thats not false advertising

---

I have no idea what planet you spend your days on.  The adverts are very clear in their wording about how perfect the Hub 3 is for hard core gamers - and with that comes the obvious implication of meeting latency requirements.  VM know in advance of the latency flaws in the Hub 3 and they are thus knowingly making false claims.  Poor latency due to congestion, etc, is what the contract covers in the same way as speed.