Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

[Datalink]

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.

The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...

Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...

Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...

That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.

Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.

Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 [MOD EDIT: Subject heading changed to assist community]

[Guybrush85]

---

@astrohominis wrote:

---

@Butler85 wrote:
Secondly, find a point where I have claimed it is fixed. Improved, yes, fixed no.

---

Maybe you can explain how advocating for a firmware update that doesn't fix the issue, whilst throwing shade on alternate solutions is a good way to go?

There's more than one way to bake a cake. xoxo

---

Because it's progressive. Would you not take any improvements you can get? I'll quote below the DSL Reports test done on both the hub 3 and Hitron done by Mud. Whilst not a definitive fix for all things, it's a good step forward. Remember, we all want the same thing here, I'm just not going to lie to myself that the option you and others think is going to happen ever will. 

If over utilisation goes on for years without fix and even after appearing on watch dog still goes on will continue to go on with large timescales for fixes (reading the comments of when people reported this to Watch Dog, it took months, years for them to do something with it too). Why do you think that you sending a strongly worded email to Virgin can get them to suddenly U-turn on their policies? Especially with the rumours of Docsis 3.1 around the corner, of which a new router would be required.

I think if you want to keep banging this drum then, then create a new thread and debate till the cows come home (excuse me if I don't participate, think you've had 10p's worth out of me now )

---

@MUD_Wizard wrote:

Comparison latency test

Hub 3 - modem mode - 20 channels

Vs.

Hitron CGNv4 - modem mode - 20 channels

---

[astrohominis]

---

@Butler85 wrote:
Would you not take any improvements you can get?

Of course one would if such an option existed.

I'll quote below the DSL Reports test done on both the hub 3 and Hitron done by Mud. Whilst not a definitive fix for all things, it's a good step forward.

Actually it's not, you are in fact misinformed.

Remember, we all want the same thing here,

Well, I'd like to think that! However it seems your very specific goal would appear to be "get firmware fix SH3". What I'm proposing is "sort out latency problems". See the difference? And another thing, why are you so bent out of shape that another option could exist? I just don't get it.

In my very first post in this thread I started out saying something along these lines: 'hey I don't think this will work but I'm going to try get VM to connect a SH2 -- will let y'all know how it goes!'

A reasonable response might be nothing at all if you not interested, or "woohoo go for it - the more options the merrier :)"  instead you start discouraging that, and trying dictate that this thread be about a firmware fix and nothing else and peeing on alternatives.

Who made you king? Did I miss something?

I'm just not going to lie to myself that the option you and others think is going to happen ever will. 

How very strange - no one is asking you to lie.  

---

@MUD_Wizard wrote:

Comparison latency test

Hub 3 - modem mode - 20 channels

---

xymox1 is the original discoverer of the PUMA6 bug and certainly more clued up on the technical details than me or you. He had this to say about MUD_Wizard's post:

"No you dont have a TCP fix. It does not exist in the market and without doubt did not exist in april. I dont know where your charts came from. Sorry thats wrong."

 https://www.dslreports.com/forum/r31453545-SB6190-Puma6-TCP-UDP-Network-Latency-Issue-Discussion~rev...

Many more in the DSL reports are expressing doubt that a firmware fix will ever come out. It's really not an unusual position to be skeptical and that doesn't mean we don't want it. A firmware fix would be amazing. However one doesn't have to fixate and pin their hopes on something that may or may not happen. 

(excuse me if I don't participate, think you've had 10p's worth out of me now )

 You're excused 🙂

[DarkBahamut]

Ah, the old chestnut that xymox1 'discovered' the 'bug'. Lets just ignore all the posts the testers made about it that predate their posts by more than a year 😉

Xymox1 has done a lot of great work, but I just have to facepalm every single time they are credited as the 'discoverer' and therefore the only source of information. This issue has been known about since the SH3 first landed in anyone's hands, which was an awfully long time before they made any posts on the matter. You'll have to ask VM why they didn't investigate the reports seriously or take it any higher - I suspect because the latency wasn't too bad at the channel counts in play at that time. Still, the issues were reported many many moons prior to those DSL reports threads coming up.

Or, we could keep pretending that Xymox1 was the first person to ever check the latency on their connection with a Puma6 hub 😉

/endrant

[astrohominis]

Fair point 😛 Not the best choice of words on my part. Maybe 'popularizer' would be more apt.

[mobculture]

Just so people are aware, those that have a Superhub 3 are pretty much stuck with it, and can't downgrade to a 2ac if they've only ever had a 3 (you can still use your old 2/2ac if you had one previously).

I say this from bitter experience. I've had my Virgin package for about 2 months now since moving flat and have been experiencing regular (every 30 secs or so) ping spikes of around 150-200ms while idle. This was especially disappointing as I used VM for 3 years prior to this move, and had no issues whatsoever. That was on a Superhub 2...

I've spoken to VM countless times now, first trying to identify the cause, then trying to downgrade to a Superhub 2ac. On several occasions, I have requested a Superhub 2ac, and have been told that one has been ordered and will be sent within 3-5 working days. After each request I have indeed received a Superhub...but it has been a Superhub 3.0! I now have a collection of FIVE Superhub 3's. FIVE!

My last contact with VM was their customer retention team as, I am sure you can imagine, I am at the end of my proverbial tether. I have been told categorically that while staff have been trying to order Superhub 2ac's, VM don't manufacturer them any more, so I cannot get one, and I am stuck with the Superhub 3.

I even tried to buy one from eBay, and then promptly found out that VM would refuse to activate it. Most disappointing, seeing as it would likely solve my issue, but I was at least able to return it.

I have asked to cancel my contract (reluctantly I might add...I really want the service I have paid for, but it is simply not fit for purpose with these constant ping spikes) but have been told I'll need to pay an early redemption cost of £240...!

I now have an engineer coming round today, for which I am having to take time off work, which I fully expect to be a waste of time, but has to be done before VM will consider writing off the early redemption fee. So they say anyway, I am not going to hold my breath.

Part of me hopes that the engineer will have an old Superhub 2/2ac in his van that I can use. Part of me knows it is wishful thinking though!

Anyway, the early redemption fee is far too high for me to afford, so I will likely be stuck with this pitiful service until a firmware fix or new Hub comes out. Either way I am a most disappointed customer, and I feel absolutely powerless to resolve this issue. Ofcom and Watchdog next, I imagine.

[Guybrush85]

---

@astrohominis wrote:

---

---

*sigh* Nothing but postulated replies based on opinions.

As you're using Xymox for the basis of your argument (although as stated, back during the trials of the hub 3, we noticed odd latency issues no sooner than the first Arris E6K's start provisioning the modems on 16 DS channels, and whilst noted as a 'to be fixed', didn't go anywhere), I've actually been in contact through PM's with him (and other members both here and on DSLR), for quite a while and while some things I agree with him on, others 'due to the variance and confusion', is still open to speculation, hence me saying the only things I can confirm are fixed are ICMP and UDP DNS. This because not only is that the opinion of many people on the forum both in testing and in patch notes, but also in patch notes to other vendors, such as one in Poland " 9.1.103AZ - DNS latency Issue - Ping latency intervals Too High".

Now assuming nothing else is fixed, DNS is still a rather large part of your every day internet usage, so saying "if one existed" is utter nonsense, not only that, but the driving person as your basis to argue also agrees that it's a worthwhile step and a push to release the firmware available is a worthwhile cause. How much would you be willing to bet that given the option to wait for a fix/new router or take a patch that only fixed those 2 functions in interim that just about everybody would take that. Your end goal is a latency fix, but there are more things on the list and everyone has their own priorities on that.

This is where I have a problem with it though. The patch notes released by Arris in April before the silence indicated changes to TCP/UDP latency. Not only that, but following this the puma test on DSLReports returns better results and nobody had explained to me technically why this is. The most we have been offered is comments such as "The DSLR Puma test is not a "serious" test. Its just a little quick tool that is not comprehensive". It doesn't explain that using Muds screenshots for an example, the test shows a consistently better result (both wired I might add) On response to this confusion, the answer I got was:

xymox1 

That is confuzing. I agree. On mine it looks the same.

As for your "sort out latency problems" speech. It's simple, if you don't want to hear the facts about Virgins policies on used hardware, then don't. I didn't respond to 'get on over on you'. Heck, I'm doing you a favour by telling you the facts.

As for why you should put it in a new thread. Whilst you wanting this is a byproduct of the puma 6 issue, it is not clear by the very nature of a forum that, that is the discussion point of the thread. I'm simply suggesting if you want to discuss that matter and get responses based on that subject (maybe even a forum team member reply), then a new thread directly in relation to it 'might' be a good idea.

Lastly, I'm hopeful but sceptical just like yourself that a proper fix will ever come out, but unfortunately I feel like we are stuck somewhere between a rock and a hard place right now. My hope is that the rumours around docsis 3.1 around the corner are true, but this of course will be subjective to the situation with your area's CMTS and testing etc.

[wotusaw]

There's are buckets of this boffinated techtalk which frankly leaves me lost and bewildered most of the time. (IQ possibly that of a monkey)

Don't get me wrong..it's very much appreciated and I do my best to understand.

However, can someone just tell me which bit is important to 'FPS' gaming? DNS, UDP, PMS?

I signed up for the 'GAMER PACKAGE' thinking that's actually what it meant.

Obviously English is another thing that must leave me lost and bewildered most of the time since 'GAMER PACKAGE' must mean something completely different because it sure ain't any good for 'Gaming'.

I'm so angry right now!!

[Guybrush85]

---

@wotusaw wrote:

There's are buckets of this boffinated techtalk which frankly leaves me lost and bewildered most of the time. (IQ possibly that of a monkey)

Don't get me wrong..it's very much appreciated and I do my best to understand.

However, can someone just tell me which bit is important to 'FPS' gaming? DNS, UDP, PMS?

I signed up for the 'GAMER PACKAGE' thinking that's actually what it meant.

Obviously English is another thing that must leave me lost and bewildered most of the time since 'GAMER PACKAGE' must mean something completely different because it sure ain't any good for 'Gaming'.

I'm so angry right now!!

---

No problems.

Essentially for the majority of the time gaming is going to be UDP and TCP. A simple way to think of it is TCP to networking is like recorded/signed for delivery to royal mail. Each packet provides and acknowledgement, if one isn't received another is sent. UDP is just firing packets in the direction.

PM's - just a private message.

DNS has the job of translating a name to an IP address, so if you browse to website.com your DNS provider would translate that into an IP and you would then would route to that page.

The gamer package is misleading to say the least. What it provides you with a higher upload speed. Gaming really isn't going to be making any use of that increase in upload either! It doesn't provide you with any change in hardware with capabilities such as QOS (sorry, another acronym), nor does it use any other technologies that would lower latency or any routing preferences or peering changes (think being able to use a private road to get somewhere in a shorter time as an example) so marketed as a gaming profile seems more of a cheeky sales trick. I think it's useful if you do something like stream while gaming as that can utilise the increase and the lack of traffic management.

QOS btw, since I made things worse by dropping another in there stands for quality of service. There are various ways in which this can work, but essentially the goal is to prioritise traffic types, such as gaming so anything else on a network (the Mrs watching Netflix etc), wouldn't impact your service.

Hope that helps.

[wotusaw]

Absolutely brilliant, thanks for the expla!!

[Spitfire16]

Hello,

My BQM is looking very colourful today 

Do you think we will see a new firmware being released from Virgin Media or a new hub being released?