cancel
Showing results for 
Search instead for 
Did you mean: 

Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

Datalink
Up to speed

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.


The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...


Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...


Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...


That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.


Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.


Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 

 

 [MOD EDIT: Subject heading changed to assist community]

4,478 REPLIES 4,478


@ShadowOfDeth69wrote:

There's obviously an issue with the Hub 3 making it impossible for a good portion of people to enjoy online gaming and Virgin Media are painfully aware with the problems being well documented.


While it is certainly an issue, and it obviously affects a few people it certainly doesn't affect a 'good portion of people'. Even if you counted every individual post on this thread as a customer experiencing the issue it would still only be a very tiny percentage of all Hub3 users. Realistically you are talking very very tiny percentages of customers.


@richtowrote:
The UDP packet loss is also particularly unavoidable to notice as it causes your internet browsing to often seem to hang or pause. And also frequently interrupts streaming.

 Very few people will actually notice this, it happens even without the Puma 6 bug especially on mobile/cell connections, they just hit refresh and try again. The UDP packet loss doesn't affect streaming, no more than a momentary glitch which often goes unnoticed, that's the whole reason UDP is used for streaming..

@Badvok Yeah I chose those words carefully as to not upset anyone having problems. A few here seem to take it personally when you say you're not overly affected by the issue.

richto
On our wavelength

@Badvokwrote:

@richtowrote:
The UDP packet loss is also particularly unavoidable to notice as it causes your internet browsing to often seem to hang or pause. And also frequently interrupts streaming.

 Very few people will actually notice this, it happens even without the Puma 6 bug especially on mobile/cell connections, they just hit refresh and try again. The UDP packet loss doesn't affect streaming, no more than a momentary glitch which often goes unnoticed, that's the whole reason UDP is used for streaming..


Its very noticeable - especially with UDP. If it was TCP then it would timeout and then resend, but not with UDP. As you say you have to try again. Thats about as noticeable as it gets! And a "momentary glitch" is also very noticeable when watching a stream.

What you actually mean is that most people dont realise that it is the SH3 to blame.

Yes, I had UDP drops (DNS failures) when I had SH3.

Certain browsers like Chrome try to hide it, by auto-retrying/refreshing if page load fails. In most cases it just shows as a page opening with a delay instead of instantly like it should with pre-fetch on.

But this UDP drop issue should be fixed in new firmware (unlike latency spikes).

 

Oh, I know there are people suffering the problem, I'm just saying using iperf (which is a more reliable way of testing for UDP packet loss, as well as TCP retries) I'm seeing very few, typically between 0% (most of the time during tests) and 0.3% (in the worst case)


@Badvokwrote:

@richtowrote:
The UDP packet loss is also particularly unavoidable to notice as it causes your internet browsing to often seem to hang or pause. And also frequently interrupts streaming.

 Very few people will actually notice this, it happens even without the Puma 6 bug especially on mobile/cell connections, they just hit refresh and try again. The UDP packet loss doesn't affect streaming, no more than a momentary glitch which often goes unnoticed, that's the whole reason UDP is used for streaming..


This forum post is for people who did notice it so your entire point is moot.... Actually can anyone who isn't having issues with latency just stay off this thread unless you have some actual knowledge to share?

It's really not useful at all to post on this topic if this issue doesn't effect you.

Its kind of like you bought a toaster, it works fine and your bread slices comes out all nice and crispy, but you keep going to a toasters anonymous meeting and telling everyone their burn toast isn't a problem. First off, that doesn't help people with burnt toast. Second, you shouldn't be hanging around like some gloating willy wonka wierdo telling everyone about how evenly toasted your bread slices are when the people are around you ended up with burnt toast!! 

Thank you for your cooperation in this matter.

Fwiw, for anyone running windows operating system and wanting to run a simple IPV4 UDP test using DNS queries to your DNS, have a look at dnsmonitor from tallsoft.com.  If you didn't want to bother plotting the data, the application will show the min/max/avg response times, number of sent queries and number of received responses.  Looks like you have to crunch the numbers yourself to determine the number of missed responses (essentially lost UDP packets).  I would suggest using a DNS query such as www.google.co.uk for example, which should be cached by the VM DNS, so, what you would end up with is the transit time to and from the VM DNS, without any extra lookup time caused by some unique query.  You can use any DNS that you prefer, but, for test purposes, I'd suggest running at least one query test run to the VM DNS to determine the loss rate within the VM network.  This is similar in idea to the GRC test, but, you can leave this running forever if you so choose.  It will run a maximum of three queries of your choice every second.  This runs in demo mode, allowing three separate queries per second.  Registered users can apparently run thousands of queries if they choose.  The application isn't designed for the purpose of modem testing, but, the ability to run this for days on end means that it can be used as a simple test application to determine the UDP losses over that period of time.

Site page and direct downloads are as follows:

http://www.tallsoft.com/dnsmonitor.htm

http://www.tallsoft.com/dnsmonitor_setup.exe

http://www.tallsoft.com/dnsmonitor_setup.zip

If anyone wanted to plot the result with Wireshark, the instructions are located here:

http://www.dslreports.com/forum/r31737637-

Fwiw, the total number of queries sent and received don't match when you look at the DnsMonitor and Wireshark numbers, but, the total number of missed responses is the same.  For example, running on Puma 7 modem the following were  observed for a 24 hour test run to my DNS:

DnsMonitor:  Sent queries = 86136  Received queries = 86135   Missed responses = 1

Wireshark:    Sent queries = 85934   Received queries = 85933  Missed responses = 1

% loss = 0.0012%

The number of lost responses from OpenDNS was 2, the number of lost responses from Quad 9 was 19.  That loss for Quad 9 is most likely the result of a much longer transit to one of the Quad 9 servers.  Those higher loss numbers are the reason why anyone who is testing for UDP losses should do so within their own network first.  When you look at the results for any UDP loss as indicated in gaming, the first question that has to be answered is what is my loss thru the modem and within my ISP network.  Its pretty easy for a game company to blame the user ISP, but, its possible now for users to actually determine their typical UDP loss within the ISPs network.  With more testing, its probable that VM users might determine that even with the Puma 6 running, the losses that you see thru the modem and to the VM DNS are considerably lower than what you see in a game.  Next question that follows is, what's the problem with the routing, from VM to the gaming network, and inside the game companies network.  If a game company were to set up a DNS server with a singular DNS response, it would be possible to determine the usual UDP loss to the game servers and back.  Personal opinion, pinging the company servers with an ICMP ping isn't a very useful gauge of a game's UDP latencies and losses, given how UDP traffic is handled throughout the transit. 

@boltedenergy.

Hey guy, this is NOT your post.

 

You have ZERO right telling ANYONE not to post their experiences. The post is applies to EVERYONE in some way or other with the Hub 3 ans if we're not affected as much as you are so bloody what? We're all in this together and we all have a vested interest in the discussion here. I mean technically the whole OP doesn't apply solely to gaming. It affects all aspects of internet use.

 

You and a the VERY specific vocal minority need to stop trying to stifle the discussion. Again, we're all in this together one way or another, man.

 

[MOD EDIT: Inappropriate Language removed, please review the Forum Guidelines]

 


@Sephiroth

wrote:
The 200 doesn't need a Hub 3. But they'll send you one and you can use your current hub instead.


 

So if I was to upgrade to 200mb BB package is there any reason why VM would not allow me to use the SH1 with my own router?

What about if I used the Superhub1 as a router with 200 broadband would it work,  or is just solely in modem mode that its capable to carry higher tier broadband speeds.

 

Thanks