cancel
Showing results for 
Search instead for 
Did you mean: 

Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

Datalink
Up to speed

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.


The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...


Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...


Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...


That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.


Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.


Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 

 

 [MOD EDIT: Subject heading changed to assist community]

4,478 REPLIES 4,478

@shanematthews Nobody here is "whining" about anything, they are raising their concerns in the correct setting to do so.
+ I find it odd you are weighing in on this issue at all whilst stating you do not own a SH3, and are therefore are not subjected to the poor performance.

Anyhow, I agree with @deviousiphone, we should keep this thread on topic.

 

My contract is with VM not intel and VM most certainly can do something about it. They can allow the relatively small minority of vocal, informed people to switch to the SH2.
I know there are no new SH2 being created - but I'm sure that VM have enough returned units to satisfy this group.

This is like a broken record. Intels fault, but VM ignored the errors in trials and pushed it out. Now let's stop making up numbers of who visits the forums, it's not relevant.

I think we can say with a degree of logic that those that know they are affected is the lesser number, but it's still an issue regardless. Moving on I'll do my timeline update later on
--------------------------------------------------------
Look behind you, a three-headed monkey


@RidingTheFlow wrote:

He also authoritatively tells us here who decided on what and who trusted who and what transpired - while he actually has no idea and just making stuff up from his "industry knowledge" (that he read about on the internet somewhere). I won't mind being proven wrong, but by hard facts please, and not just some hollow opinion.

Also, guess what, my "whining" actually got me my SH2 back and my latency fixed. While according to shane's extensive "knowledge", nobody actually cares because law is on VM side, yay!


And you and i both no that this isn't a fix, there is no new stock of hub 2's and the refurb units will only remain stocked for so long, so you're one of the lucky ones, or do you think they are going to magically give everyone who complains a hub2?

And no its not stuff i've read on the internet, its called logic, i don't need to actually work at the company to read between the lines and understand whats going on in the background, VM got shafted by arris who provided a device not entirely able to provide what it should, yes they were made aware of an issue and they likely spoke to arris/intel regarding it, the standard canned response of "we will look in to it" was likely sent back to VM and/or a fix was promised without a specific timeframe, as such VM would have pushed ahead with the hub3 mainly because they assumed the issue owuld be fixed before it affected too many people and because they needed it to be able to push out the VM wifi hotspots as the hub2's are unable to provide this service

Nnow we get closer to today and the issue that was supposed to be fixed never was, its now affecting more customers, VM is now stuck in the middle between angry customers and a a company unable to fix the devices they purchased from them, they are now under pressure to fix a problem (the puma6 latency issue) that is entirely outside of their control, they don't have a direct contract with intel so they have to pressure arris who then has to pressure intel, from what i understand there isn't a fix in the works and it would seem the puma7 also suffers from a similar design flaw

VM only have 2 devices they can send to customers, and both of them are puma6 based, the arris hub3 or the hitron business router, both of which suffer from the same latency issues for the most part, they will have a handful (by comparison) of hub2/ac's that were returned, remember that at one point VM wouldn't have even asked for the device to be returned as it was considered obsolete like a number of the older TV boxes they don't want back either, their options are limited and most of them are not even on the list to be considered

At this point VM will be doing two things, the first thing is waiting and seeing if there will actually be a firmware fix for the puma6 devices, as it stands even VM at this point will be assuming that either no fix is coming or that its taken too long, the second thing they will be doing is looking at getting a hub4 in to circulation, this process involves multiple steps which includes but is not limited to companies bidding for contracts, agreeing terms, getting full specs and requirements, drawing up and designing both the internals and externals of the device including but not limited to disabling any firmware features not required and making sure to lock down the device, there will also be a private testing phase followed by a public testing phase followed by a general release

Now there is one spanner that has been thrown in to this, there are rumours of a puma7 based hub4, as this chip has also been shown to have the same issues its older sibling also had they may have been forced to scrap that idea and start from scratch which will delay the process even more, unless the company that won the contract is able to provide an alternative device using a different chip

No i don't have any hard evidence or proof of the above but at this point i'm not really bothered if you want to trust or believe what i've said, i'm not defending VM i'm just explaining that despite all the whining and moaning there isn't anything VM can really do to speed the process up, because when you rush things you get badly designed products that end up being worse than the ones we already have 😛

shanematthews I'm not sure what you're actually contributing to this thread apart from taking it off topic.

We're looking for a fix, what's your beef with that?  If all you have to offer is "fan boy" rhetoric then it's wasted in here. 

Since when do posts start to get moderated?

Please note this is purely from my own research and by no means official. I am concentrating this on the hub 3 in particular, although due to the global nature of this, I will be referencing thing on other modems and across the pond.

So where did this start?

Back in late 2015 when the Hub 3 was on trial, from a pure latency point of view, the difference between 8 DS Channels on a 2ac to my new Hub 3 on mine and others BQM’s looked almost identical. As the trial continued and the new CMTS were being installed (either an Arris E6k or Cisco c-BR8) it meant that 8 channels suddenly became 16 and at this point the BQM looked a little less than like for like and the max latency was a lot higher. Now, today on 24 DS channels, the graphs look worse than they ever have and the increase is not just on the maximum latency, but the average as well.

Unfortunately, I cannot get the image from the day my graph changed from 8 – 16 DS Channels, but this is when it changed from 16 to 24, so you can see the difference it made.

BQM 16-24.png

At the time it was thought the issue was either a false positive i.e. something to do with TBB BQM, or something related to ICMP traffic in particular. Which is less of a worry than TCP/UDP. As latency isn’t as noticeable as other variables within networking aside from gaming, thinks ticked on for a while.

Fast forward to about Q4 2016 and across the pond on the website DSLReports a user Xymox created a thread essentially highlighting the same issues that we’d done. Though since we were essentially using a BQM (so ICMP), which he did initially, he then went on to show that TCP/UDP, ARP etc were also subject to the same latency issue. Put 2 and 2 together and the common factor of these devices is of course the puma 6 chipset. The puma 6 chipset is made by Intel. Intel bought out the puma chipset and rights from Texas Instruments and then went on to produce the puma 5, 6 and 7. Whilst the intricacies of what cause the problem exactly I cannot say, the laymans version is a high priority maintenance task runs every couple of seconds, essentially bottlenecking the CPU and therefore normal tasks take a hit.

So now we have a problem, what next?

So thanks to Xmyox’s posts and his seemingly unwillingness to let this go a user Netdog, an Arris employee created a thread to talk about the issue and any patches, test results. Not only that, but Xymox reached out to tech reports i.e El Reg and since all that there have been 2 patches across the pond! To date, no variant of these patches have come to the hub 3.

So what did they fix?

The first patch fixed ICMP latency spikes, so essentially your BQM will start to look pretty and that’s about for general day to day use.

The second (.93V for that particular device) the only thing we can say for certain was it fixed is DNS (UDP). Now those with that patch, including the Hitron used by Virgin Media Business have reported better results using the puma 6 test on the DSLReports website and other tests, so what exactly is fixed is open for debate. The patch notes released by Netdog don't help as it reads as TCP/UDP latency fixes. Regardless of what exactly it does, it's a step forward from where we on the hub 3 are now.

Well at least it can’t get any worse…

So not long after this a user Mackey on the DSLReports website discovered a fault that meant that a puma 6 device could suffer a DOS afrom a very low powered attack (as low as <5Mbit) and further on from this found a certain attack would cause the modem to reboot. The likelihood is you’ve not fallen victim to this and unfortunately, I think until an event like that does occur, this will remain mostly ignored.

So what now for the Puma 6?

Well things took a turn at some point and a class action lawsuit was filed against Arris. The good thing is eventually we will probably get lots of juicy information from it, the bad news is until then it seems things have gone quiet and Netdog hasn’t been since. Another user, who is believed to be confirmed as an Intel spokesperson came onto the DSLReports forums and posted to say a full and complete fix was being pushed out to the relevant manufacturers. Since then though, we've not heard anything, or seen any new patches. This again is probably down to the on-going legal case, but some believe that said patch didn't pass testing/fix anything, so that's why we've heard nothing.

So enough about them, what about us VM users?

So where are us Hub 3 users?. Currently, no futher on, but as James_W stated a trial of a new firmware is due soon… 4 weeks ago. Based on the information I’ve provided, it is my belief that the firmware version will just be the same as that of the 93V patch.

So is the hub 3 as bad as over the pond?

Well… not quite, or atleast I can't match it. Like every good test, you need some information, so my setup is as follows. Tests were all performed on a laptop, with nothing significant running and connected via Ethernet. I am using the hub in modem mode wired into a Netgear R7800 on 24x DS, 2x US (64 QAM) and my CMTS is a Cisco c-BR8. All my power levels, SnR etc are well within spec and my area suffers from no utilisation. For arguments sake, I’m calling this a best case scenario with little to no external factors that could taint the results.

Here is both an ICMP (not fixed for us) and TCP Ping Plotter graph.

TCP:

TCP-google.PNG

ICMP:

ICMP-Google.PNG

Now I can’t seem to match ‘the every 2 seconds spike’, but there’s certainly some odd spikes in there and enough to cause unwanted issues when gaming. I also wanted to prove this wasn’t just a fluke, so performed the same tests on a friends BT Infinity setup… then misplaced it. I will endeavour to get another (unless someone can provide one quicker), but the results were no big spikes. It’s worth noting that latency within docsis 3.0 will never match that of BT (roll on 3.1 and AQM) but the spikes are not the norm.

If you have any additional pingplotter/BQM’s or anything else you would like me to add to this, please PM me and include the details of the test and your setup (I don’t need war and peace, just something similar to what I’ve given about my setup).

Due to the not being able to edit posts over a certain age, I will speak to the forum staff to see if I can have them make edits to this post with any updates.

So what options do we have?

I’ll break it into 3 categories.

Hub 3: We wait on a fix, see if the trial brings good results (you never know, it could be the answer we’ve all been looking for).

Hub 4: Again, it’s a waiting game and speculation is that it’s a puma 7 device, although hopefully some of this spotlight means a rethink on that, which could mean delays in the hub 4.

Hub 2/2ac: Ok, so I nearly didn’t add this because it’s cause it’s such a grey area and it’s not a fix as such, but here goes. Some members have been able to get hold of an older hub and get it provisioned to swap out the hub 3. It seems very much luck of the draw, but the main advice seems to be if you really want this option, right in with an official complaint and follow it through this route. The caveat is if you’re on Vivid 350, 300 (and possibly the older gamer profile these days), you may have to step down, but if latency is main objective, it might be worth a shot.

So hopefully if you’ve just joined us for the ride, this gives you a bit of insight.

TL;DR Damn you intel... should have left the puma to Ford.

--------------------------------------------------------
Look behind you, a three-headed monkey

Virgin got shafted. No they didn't the issue was reported during testing but they decided to go ahead anyway. The customer was the one shafted.

Why they went ahead we can only speculate.

They are other cable modems out there running different chipsets ideally they should have trailed two and I'll happily be corrected if they did but that would have been the best business approach.

 


@shanematthews wrote:

@Drewley wrote:

@shanematthews

If people other than yourself want to go to put in effort to try and resolve a problem they PERSONALLY feel is affecting THEMSELVES, why not let them be?

I really don't see your motivation behind putting in so much effort to discourage people from making noise about this issue.


I'm not telling them not to put in the effort, i'm telling them to put in the effort by complaining about the CORRECT company, in this instance intel who make the modem, intel is the one who is unable to fix it which as a result means arris can't fix it which in turn means VM can't fix it, VM have zero power over the completion of the firmware update to fix the issue, and this is assuming that the issue CAN be fixed via software at all, if its a hardware flaw then no amount of firmware updates will fix it, so yes i encourage you to complain but to the right people, whining to VM doesn't do anything


Seriously Shane, I just don't get where you are coming from on this.

As has been stated on numerous documented occasions. The very first trial group reported latency issues to Virgin with the Hub 3.0. How are Virgin not responsible for continuing with the purchase and mass deployment of this kit after the feedback?

For a poster so attached to logic and real world expectations your attitude is bizarre. If we all follow your advice of not putting pressure on OUR supplier I guarantee you the companies assessment will be that whilst documented our customer base is not unduly concerned as we have had no negative feedback.

To say that most customers are not experiencing issues is utter nonsense. All customers are impacted by reduced web browsing performance. Just because most customers do not understand that the constant delays and stutters are due to the Puma 6 and not how it should be, is not an acceptable defense.

Your arguments are logically opposed:

1) No problem as most non technical people don't understand the issue so just accept poor service and do not complain.
2) Don't complain to Virgin. It's not their fault. They are victims.

Rubbish, we are the victims. We are the customers, continuing to be charged at full price for a service that the supplier knows is not working as it should be. If Virgin decide to supply the service for free or a substantially reduced cost I would then accept their status as a victim. I'd even applaud them publicly and laud them as a paragon of proactive customer service.

Instead we have a company that is aware of the issues but continues to actively push a known defective product onto a ever larger percentage of the customer base.

chand93
On our wavelength

By Shane's own 'logic' his constant defending of VM makes him either a undercover employee or someone with an association to the company.