cancel
Showing results for 
Search instead for 
Did you mean: 

Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

Datalink
Up to speed

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.


The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...


Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...


Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...


That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.


Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.


Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 

 

 [MOD EDIT: Subject heading changed to assist community]

4,478 REPLIES 4,478

these are all my log entries dated yesterday the 13th

2018-03-13 22:29:30.00 critical No Ranging Response received - T3 time-out;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:af;CM-QOS=1.1;CM-VER=3.0;
2018-03-13 13:15:27.00 notice SW download Successful - Via Config file
2018-03-13 13:07:34.00 notice SW Download INIT - Via Config file

horseman
Alessandro Volta

@Pyr0wrote:

these are all my log entries dated yesterday the 13th

Yes that's normal region TFTP rollout (Beta/Trials will contain "NMS" in the SW Init/Download stanza),  T3 often occurs during ranging but it often helpful to keep a benchmark of DownStream/Upstream stats including (Upstream Tick Timers) along with any TBB/BQM in case you get any local SNR/FEC issues.

Otherwise you're "good to go" for another 7 years (since you last posted in 2011?)  😛 Hopefully with 350 Tier you should get prioritised for SH4 D3.1 Hybrid rollout but that's still a tad premature to speculate.....  

Regards Tony
"Life is a Binary Inspired Turing Computed Hologram"(don't PM or @Mention me - in case ignoring you offends)
DEFROCKED

risc19
Well-informed

I just phoned 150 and asked to have my firmware updated, they had no clue what I was talking about, she said she would transfer me to a manager.

The manager then told me it was "impossible" to manually upgrade the firmware.

I guess all of you are lying...

Quality Indian call center!

I phoned again and went through to the 'im thinking of leaving' option.

Indian guy told me very bluntly that if thats what the technical team said then there is nothing he can do. 

 

I give up, I really give up.

Why can't I ever get someone on the phone thats not just reading a script in front of them.

wotusaw
Superfast

risc19@

'Out of the people here that have the new firmware (9.1.116.603) are you on 350mb?'

 

Listen carefully, I will say this only once.

I had the NEW OLD firmware. 

I hard reset my modem by pushing a pin/whatever into the hole at back/lower of HUB3. Held down for 30 seconds.

When it rebooted....after some time...had updated to the NEW NEW firmware.

I'm on 300MB. If I could get the 350MB I'd have it.Robot wink

I think if you were on the NEW OLD firmware it updates you when a hard reset is done to the NEW NEW firmware....maybe?

 

Incidentally just in case any of you are gamers. I found my lag was being caused in the most part by my graphics card which was a R9 280X...not the HUB 3. Although before the NEW NEW update obviously it caused some lag etc.

For some reason which I won't go into now, that card and all similiar ones have a problem. Anyway when I got an RX 580 all lag/crashes just became history. Now I annihilate stuff with ease.

Anyway with the new RX 580 and latest HUB 3 firmware...and a re-pull to replace the RG6 with RG11 cable. Possibly Virgin are back on my Christmas card list....I just don't know for sure yet.Heart

Maybe too much info there but hope that helps you 'risc19'.Smiley Wink

PS: I am not a Virgin spy and belong to the World famous and feared 'BN Nitwitts' gaming clan. My real fantasy gaming name is Necessitor.Smiley Very Happy

 

 

Yes, I've done a factory reset before, I just did it again and i'm still on 9.1.116V.

Also my nvidia 1070 ti with 8700k causes no lag thanks.

 

 

wotusaw
Superfast

risc19@

Am stumped.....I was on the trial and got the new update on hard reset.

Why you haven't got it doing the same thing is a mystery.

Maybe it's a triple whammey...meaning possibly it's the 'area code your in' and 'on trial' and 'hard reset'?

The NVIDIA cards are not effected so your ok. Only R9 280X and previous, I think....and only with certain games.

 

The NEW NEW firmware works for me. Anyways all the widgety-woo gadgets/whatever show a marked improvement and in game it's alot betterer.Smiley Wink

Persevere with Virgin. Tell them your a personal friend of Mr Putin......erm, maybe not.Smiley Frustrated


@risc19wrote:

I just phoned 150 and asked to have my firmware updated, they had no clue what I was talking about, she said she would transfer me to a manager.  The manager then told me it was "impossible" to manually upgrade the firmware.

I guess all of you are lying... Quality Indian call center!

I phoned again and went through to the 'im thinking of leaving' option.  Indian guy told me very bluntly that if that's what the technical team said then there is nothing he can do. 

I give up, I really give up.  Why can't I ever get someone on the phone that's not just reading a script in front of them.


I thought going via the I'm thinking of leaving option would give you a UK call centre but it seems not from your experience.   Smiley Surprised

Unless you try later in the evening when the Indian call centre is closed (not sure if that's a thing or not).

 

supernovi
On our wavelength

Here's a crazy idea, how about asking to speak to someone in the UK.

During one of my calls I was getting nowhere and asked to be transferred to someone in the UK, they did it. Good job I did because the UK guy pointed out my hub was not the 2 but actually the 1 and they shouldn't have re-activated it as it will not work at all... DOH!

I got nowhere in terms of resolving the spikes but might help you.


@horseman

The Puma6 Tool is questionable because many do not have the time/inclination/apitude/skill to ensure their own end device is booted in SAFE MODE, and solely cabled to SH3 in Modem Mode without intervening Router/Switch and also offering/posting link to associated Protocol (ala Wireshark) dumps.  😞 


If the Puma6 tool is a genuine test then I have seen a massive improvement from one red packet every 2s at 140ms+ to the odd one per run at around 90ms, and to be honest as I wasn't testing with a single computer that's likely other traffic, sometimes there are no red packets.

I am more than happy to run separate Wireshark captures with both a PC and Mac hardwired in modem mode then post those and the associated Puma results if that is of interest/help, be good to compare with old firmware which I no longer can.

If anyone has any other tools they can suggest, and that they can run on old firmware, happy to set that up as well.

For what it's worth my current result from Safari on Mac running via two switches & a firewall to Hub in modem mode running 9.1.116.603 @350 looks like this. Sadly I can't find a grab of my old tests, but there was a lot of red.

puma.png

I am also no long seeing ping spikes, note maximum ping below, before I'd see max in the 140ms+ range, however that's not a surprise given how much cleaner my BQM is.

[user@host:~] $ sudo ping -f 8.8.8.8
Password:
PING 8.8.8.8 (8.8.8.8): 56 data bytes
--- 8.8.8.8 ping statistics ---
14674 packets transmitted, 14635 packets received, 0.3% packet loss
round-trip min/avg/max/stddev = 10.069/14.010/26.153/1.720 ms
[user@host:~] $

I am now wishing I'd kept better records of when the problem was still visible to compare.

On a slightly amusing note, whenever I do run tests in modem mode I get an email and printed letter a few days later about mDNS packets leaking from my network, but that's what you get from putting a computer naked on the internet!

Edit: Spelling


@horseman

The Puma6 Tool is questionable because many do not have the time/inclination/apitude/skill to ensure their own end device is booted in SAFE MODE, and solely cabled to SH3 in Modem Mode without intervening Router/Switch and also offering/posting link to associated Protocol (ala Wireshark) dumps.  😞 


If the Puma6 tool is a genuine test then I have seen a massive improvement from one red packet every 2s at 140ms+ to the odd one per run at around 90ms, and to be honest as I wasn't testing with a single computer that's likely other traffic, sometimes there are no red packets.


I am more than happy to run separate Wireshark captures with both a PC and Mac hardwired in modem mode then post those and the associated Puma results if that is of interest/help, be good to compare with old firmware which I no longer can.

If anyone has any other tools they can suggest, and that they can run on old firmware, happy to set that up as well.
For what it's worth my current result from Safari on Mac running via two switches & a firewall to Hub in modem mode running 9.1.116.603 @350 looks like this. Sadly I can't find a grab of my old tests, but there was a lot of red.

puma.png

I am also no long seeing ping spikes, note maximum ping below, before I'd see max in the 140ms+ range, however that's not a surprise given how much cleaner my BQM is.
 
[user@host:~] $ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=58 time=15.126 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=14.242 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=17.879 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=11.748 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=58 time=12.438 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=58 time=12.506 ms
64 bytes from 8.8.8.8: icmp_seq=6 ttl=58 time=13.508 ms
64 bytes from 8.8.8.8: icmp_seq=7 ttl=58 time=12.222 ms
64 bytes from 8.8.8.8: icmp_seq=8 ttl=58 time=12.598 ms
64 bytes from 8.8.8.8: icmp_seq=9 ttl=58 time=11.492 ms
64 bytes from 8.8.8.8: icmp_seq=10 ttl=58 time=13.656 ms
64 bytes from 8.8.8.8: icmp_seq=11 ttl=58 time=16.324 ms
64 bytes from 8.8.8.8: icmp_seq=12 ttl=58 time=12.925 ms
64 bytes from 8.8.8.8: icmp_seq=13 ttl=58 time=13.597 ms
64 bytes from 8.8.8.8: icmp_seq=14 ttl=58 time=17.057 ms
64 bytes from 8.8.8.8: icmp_seq=15 ttl=58 time=14.325 ms
64 bytes from 8.8.8.8: icmp_seq=16 ttl=58 time=11.250 ms
64 bytes from 8.8.8.8: icmp_seq=17 ttl=58 time=14.765 ms
64 bytes from 8.8.8.8: icmp_seq=18 ttl=58 time=11.455 ms
64 bytes from 8.8.8.8: icmp_seq=19 ttl=58 time=14.977 ms
64 bytes from 8.8.8.8: icmp_seq=20 ttl=58 time=11.611 ms
64 bytes from 8.8.8.8: icmp_seq=21 ttl=58 time=15.944 ms
64 bytes from 8.8.8.8: icmp_seq=22 ttl=58 time=21.766 ms
64 bytes from 8.8.8.8: icmp_seq=23 ttl=58 time=10.999 ms
64 bytes from 8.8.8.8: icmp_seq=24 ttl=58 time=20.405 ms
64 bytes from 8.8.8.8: icmp_seq=25 ttl=58 time=11.840 ms
64 bytes from 8.8.8.8: icmp_seq=26 ttl=58 time=12.679 ms
64 bytes from 8.8.8.8: icmp_seq=27 ttl=58 time=11.873 ms
64 bytes from 8.8.8.8: icmp_seq=28 ttl=58 time=17.621 ms
64 bytes from 8.8.8.8: icmp_seq=29 ttl=58 time=11.294 ms
64 bytes from 8.8.8.8: icmp_seq=30 ttl=58 time=11.529 ms
64 bytes from 8.8.8.8: icmp_seq=31 ttl=58 time=11.769 ms
64 bytes from 8.8.8.8: icmp_seq=32 ttl=58 time=11.406 ms
64 bytes from 8.8.8.8: icmp_seq=33 ttl=58 time=11.266 ms
64 bytes from 8.8.8.8: icmp_seq=34 ttl=58 time=12.274 ms
64 bytes from 8.8.8.8: icmp_seq=35 ttl=58 time=14.153 ms
64 bytes from 8.8.8.8: icmp_seq=36 ttl=58 time=12.736 ms
64 bytes from 8.8.8.8: icmp_seq=37 ttl=58 time=12.686 ms
64 bytes from 8.8.8.8: icmp_seq=38 ttl=58 time=19.209 ms
64 bytes from 8.8.8.8: icmp_seq=39 ttl=58 time=12.241 ms
64 bytes from 8.8.8.8: icmp_seq=40 ttl=58 time=12.668 ms
64 bytes from 8.8.8.8: icmp_seq=41 ttl=58 time=11.488 ms
64 bytes from 8.8.8.8: icmp_seq=42 ttl=58 time=12.411 ms
64 bytes from 8.8.8.8: icmp_seq=43 ttl=58 time=11.000 ms
64 bytes from 8.8.8.8: icmp_seq=44 ttl=58 time=13.545 ms
64 bytes from 8.8.8.8: icmp_seq=45 ttl=58 time=28.942 ms
64 bytes from 8.8.8.8: icmp_seq=46 ttl=58 time=14.137 ms
64 bytes from 8.8.8.8: icmp_seq=47 ttl=58 time=14.924 ms
64 bytes from 8.8.8.8: icmp_seq=48 ttl=58 time=13.665 ms
64 bytes from 8.8.8.8: icmp_seq=49 ttl=58 time=11.660 ms
64 bytes from 8.8.8.8: icmp_seq=50 ttl=58 time=11.096 ms
64 bytes from 8.8.8.8: icmp_seq=51 ttl=58 time=12.205 ms
64 bytes from 8.8.8.8: icmp_seq=52 ttl=58 time=13.223 ms
64 bytes from 8.8.8.8: icmp_seq=53 ttl=58 time=12.242 ms
64 bytes from 8.8.8.8: icmp_seq=54 ttl=58 time=12.216 ms
64 bytes from 8.8.8.8: icmp_seq=55 ttl=58 time=11.574 ms
^C
--- 8.8.8.8 ping statistics ---
56 packets transmitted, 56 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 10.999/13.685/28.942/3.137 ms
I am now wishing I'd kept better records of when the problem was still visible to compare.
 
On a slightly amusing note, whenever I do run tests in modem mode I get an email and printed letter a few days later about mDNS packets leaking from my network, but that's what you get from putting a computer naked on the internet!
 
Edit: Spelling and repost as original pulled. I think I know why having checked AUP, hopefully this is acceptable.