cancel
Showing results for 
Search instead for 
Did you mean: 

Hub 3 / Compal CH7465-LG (TG2492LG) and CGNV4 Latency Cause

Datalink
Up to speed

Good Day Ladies and Gentlemen,

Greetings from the other side of the pond, so to speak.  Over the last few weeks I've been perusing various user forums across North America and Europe for issues related to Intel Puma 6 modem latency.  Of those forums, your Hub 3 stands out as yet another Puma 6 based modem where users see continuous latency no matter what site is used or what online game is played. Considering all of the problems that are on the go, the following information should be of interest to all Hub 3, Compal CH7465-LG and Hitron CGNV4 modem users.  There is much more to post regarding this, so this is a start, to alert VM users as to the real cause of the latency and hopefully engage the VM engineering staff, via the forum staff, with Arris.  I am surprised to see that there has been no mention on this board of users from other ISPs who are suffering the exact same issues with their modems, so, this may come as a surprise to some, and possibly old news to others.

So, the short story ........

The Hub 3 / Compal CH7465-LG (TG2492LG) & Hiton CGNV4 modems are Intel Puma 6 / 6 Media Gateway (MG) based modems.  These modems exhibit high latency to the modem and high latency thru the modem.  The latency affects all IPV4 and IPV6 protocols, so it will be seen on every internet application and game.  The basic cause is the processing of the data packets thru a CPU software based process instead of thru the hardware processor / accelerator.  It appears that a higher priority task runs periodically, causing the packet processing to halt, and then resume.  This is observed as latency in applications and in ping tests to the modem and beyond.  For the last several weeks, Hitron, along with Intel and Rogers Communications in Canada have been addressing the latency issue within the Hitron CGNxxx series modems.  To date, only the IPV4 ICMP latency has been resolved.  Although this is only one protocol, it does show that a Puma 6MG modem is capable of using the hardware processor / accelerator with good results.  Currently Rogers is waiting for further firmware updates from Hitron which should include an expanded list of resolved protocol latency issues.  For Arris modems, "Netdog" an Arris engineer indicated last week that Arris was onboard to address the issue for the Arris SB6190 modem.  That should be considered as good news for any Arris modem (read Hub 3) user as Arris should be able to port those changes over to other Puma 6/6MG modems fairly quickly.  This is not a trivial exercise and will probably take several weeks to accomplish.  Note that there is no guarantee at this point that it is possible to shift all packet processing to the hardware processor / accelerator without suffering from any packet loss side effects.  Time will tell if all of the technical issues can be resolved with the current hardware included in the Puma 6/6MG chipset.  Last night, Netdog loaded beta firmware on selected test modems on the Comcast Communications network.  As this was only done last night, it's too soon to tell what this version resolves and if it was successful or not.  Netdog has contacts with staff at Comcast, Rogers, Charter and Cox Communications to fan out beta versions and modifications for testing.  I'd say its time to add Virgin Media and/or Liberty Global to that group as well.

Recent activity:

Approx three weeks ago a DSLReports user, xymox1 started a thread where he reported high latency to an Arris SB6190 and illustrated that with numerous MultiPing plots.  This is the same latency that I and other users with Rogers communications have been dealing with for months so it came as no surprise.  As well as reporting via that thread, xymox1 took it upon himself to email several staff members at Arris, Intel, Cablelabs and others.  The result of that campaign was Netdog's announcement, last week, that Arris was fully engaged at resolving the issue.  That has led to last nights release of beta firmware, although as I indicated its too early to determine what the beta firmware resolves, if anything.


The original thread that xymox1 started is here:

https://www.dslreports.com/forum/r31079834-ALL-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-mis...


Yesterday, DSLReports issued a news story covering the thread:

https://www.dslreports.com/shownews/The-Arris-SB6190-Modem-Puma-6-Chipset-Have-Some-Major-Issues-138...


Today, Arris responded:

https://www.dslreports.com/shownews/Arris-Tells-us-Its-Working-With-Intel-on-SB6190-Puma6-Problems-1...


That response was also picked by Multichannel.com

http://www.multichannel.com/news/distribution/intel-arris-working-firmware-fix-sb6190-modem/409379

This is more news likely to appear in the next few days as additional tech and news staff pick up on this issue.


Hub 3 observations:

Like many others using a Puma 6/6MG modem, Hub 3 users are experiencing latency when they ping the modem, or ping a target outside of the home, game online or use low latency applications.  The common misconception is that this is Buffer Bloat. It's not. Its most likely a case of the packet processing stopping while the CPU processes a higher priority task.  The packet processing is done via the CPU no matter what mode the modem is operating in, modem mode or router mode and no matter what IPV4 or IPV6 protocol is used.  Normally, the latency is just that, latency.  The exception are UDP packets. In this case there is latency and packet loss.  The result of that is delayed and failed DNS lookups, and poor game performance for games that use UDP for player/server comms or player/player comms.


Can this be fixed?

So far, it appears that the answer is yes.  Rogers Communications issued beta firmware to a small group of test modems in October.  This version shifted the IPV4 ICMP processing from the CPU to the hardware processor / accelerator, resulting in greatly improved performance in ping latency.  At the present time we are waiting for the next version firmware which should shift other protocols over to the hardware processor / accelerator.  That can be seen in the following post:

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/369...

The details and results of last nights beta release to the Comcast group have yet to be seen.

At this point there is enough reading to keep most staff and users busy.  My intention is to post some of the history leading up to this point and instructions on how to detect the latency and packet loss.  This is not thru the use of a BQM.  I had hoped to post this all at once but events are moving much faster than I had thought they would.  For now this should suffice to get the ball rolling.

Below is a link to a post with a couple of HrPing plots from my 32 channel modem to the connected CMTS.  This shows the latency that is observed and reflects what others have posted in this forum using Pingplotter and HrPing.

https://www.dslreports.com/forum/r31106550-

HrPing is one of the freebie applications that can be used to monitor the latency to and thru the modem. 

Pingplots with Pingplotter which show the latency from my modem to the CMTS can be found in the first two to three rows of my online image library at Rogers Communications, located below.  They are essentially what the BQM would look like if you were able to zoom into the plot to the point where you could see the individual ping spikes.  Those ping spikes are common to Puma 6 and Puma 6MG modems.

http://communityforums.rogers.com/t5/media/gallerypage/user-id/829158

 

 

 [MOD EDIT: Subject heading changed to assist community]

4,478 REPLIES 4,478

PhilMull, this is half the problem, that they do that, so everyday deliberately inflict this problem onto new customers, it could be argued that they can't help the Puma 6 fault, but it becomes mute when they hand it over to people deliberately!

It looks like management is afraid to admit that SH3 has a problem and they think that hiding the issue is better somehow. Therefore they don't inform even own employees - which creates this mess.

 

JJC1138
On our wavelength

@Gameruk wrote:
Can i leave my contract now because of the prise rise in november or do i have to wait for a email confirmation

I've been wondering that too. The precise wording of the contract clause is "The notice of termination must be given within 30 days of the increase in charges or changes to this agreement being notified to you" so if they follow that exactly then we have to wait. It's possible that they might let you cancel early, but I can imagine a scenario where the person you speak to says that's fine and puts through the cancellation, but then the system sees that they haven't sent you the notification yet and charges the early termination fee. So I'm not gunna risk it and will just keep eagerly checking the post every morning like I'm waiting for my flippin' Hogwarts letter.

We have had some pretty good discussion about the technical aspects of the issue over at DSLR. Mackey, who discovered the DoS and got root on the Arris SB6190 and has looked really carefully at whats going on has provided some some depth on whats causing the issue. This really looks untenable.

 

Intel has said officially they have started to release some patch to cable modem companies. We dont know who or what models. After the modem vendors work on it a bit it will go to ISPs and ISPs will work on it a bit and then it will come to us as users. We are weeks to months away.

No one in our discussion sees a way to fix the DoS and keep performance. The spike is just a symptom of the main issue. So it might not be possible to fix the spike without reducing the modem speed to like 60mbps.

Really the options for Intel seem to be:

1. Recall/replacement - staggering cost/lose market
2. Fix DoS fully, but at a loss of speed - lose law suit and market, still face recall because of not meeting speed tiers.
3. Rewrite everything using a new RTOS - huge development time, huge costs, stay in market & maybe have best chip.
4. Address DoS somehow keeping speeds but performance still abysmal because of OS. - Lose lawsuit, online reputation continues to ebb away at sales, slowly lose market.
5. New silicon + new RTOS. Best solution but may take years along with big investment.
6. MUCH faster silicon with bigger table, Same OS - Most issues resolved, still not as good as Broadcom but closer. Would need to sell cheaper VS Broadcom.

7. Just keep going, take lawsuit up the butt, stagger forward keeping sales going because of monopolies. **** US-CERT they have no legal powers. Keep paying people under the table to use Pumas. Ignore online BS as irrelevant.

A lot of pages moved over the weekend, but unfortunately nothing in terms of any news from VM. 

As it's been said and done I think a lot in this thread, the idea of VM allowing other kit is a pipe dream. It's ok saying "they could", you could go all day with what businesses could do, but there is a reality here and that is that until a new piece of a hardware comes out, the hub 3 is what we've got. If you're fortunate enough to have a 2/ac, then good for you, but buying one off eBay or shouting at the staff won't make them suddenly start producing stock they simply don't have, or provisioning old equipment that shouldn't have been sold in the first place.

Xymox, you were almost correct in the it's Intel/Arris thing, but VM do have a hand in this. Whilst they didn't cause the issue and it's not within their remit to come up with the solution, they were aware of issues with this hub during the trial. At first when getting the hub, we were all provisioned for 8x DS channels and in terms of the latency issue, there appeared to be no difference between the older hubs and the hub 3. When the CMTS in areas were upgraded to either an Arris E6k or c-BR8 and hubs we started getting 16, 20 or 24 channels did an obvious issue happen and all of the BQM graphs we have start spiking all over. This was ignored, or at least thought to be a false positive and despite that (amongst other things), the hub was rolled out.

We talk about better testing, but even when that happens, it doesn't mean companies will act on it unfortunately. Hopefully the testing is pushed further up the chain and these kinds of issues aren't highlighted by UAT users like me and you.

The information from Datalink about Docsis 3.1 is intersting. Based on how certain variables reflect in the severity of the issue (such as DS channels from my example above), I was interesting to know how a puma 7 device acted 3.0 vs 3.1. Xymox, I don't think anyone's trying to say it's not an issue, it's just technically the differences are interested, so I think your passion for this maybe seems to be a bit missplaced/heated in that response.

Anyway, I will try and do an update later when I get some time and add in new/additional info.

--------------------------------------------------------
Look behind you, a three-headed monkey


@Guybrush85 wrote:

A lot of pages moved over the weekend, but unfortunately nothing in terms of any news from VM. 

As it's been said and done I think a lot in this thread, the idea of VM allowing other kit is a pipe dream. It's ok saying "they could", you could go all day with what businesses could do, but there is a reality here and that is that until a new piece of a hardware comes out, the hub 3 is what we've got. If you're fortunate enough to have a 2/ac, then good for you, but buying one off eBay or shouting at the staff won't make them suddenly start producing stock they simply don't have, or provisioning old equipment that shouldn't have been sold in the first place.

Xymox, you were almost correct in the it's Intel/Arris thing, but VM do have a hand in this. Whilst they didn't cause the issue and it's not within their remit to come up with the solution, they were aware of issues with this hub during the trial. At first when getting the hub, we were all provisioned for 8x DS channels and in terms of the latency issue, there appeared to be no difference between the older hubs and the hub 3. When the CMTS in areas were upgraded to either an Arris E6k or c-BR8 and hubs we started getting 16, 20 or 24 channels did an obvious issue happen and all of the BQM graphs we have start spiking all over. This was ignored, or at least thought to be a false positive and despite that (amongst other things), the hub was rolled out.

We talk about better testing, but even when that happens, it doesn't mean companies will act on it unfortunately. Hopefully the testing is pushed further up the chain and these kinds of issues aren't highlighted by UAT users like me and you.

The information from Datalink about Docsis 3.1 is intersting. Based on how certain variables reflect in the severity of the issue (such as DS channels from my example above), I was interesting to know how a puma 7 device acted 3.0 vs 3.1. Xymox, I don't think anyone's trying to say it's not an issue, it's just technically the differences are interested, so I think your passion for this maybe seems to be a bit missplaced/heated in that response.

Anyway, I will try and do an update later when I get some time and add in new/additional info.


I think you should read up on the technical issues discussed by Mackey recently so you have a better understanding of the issues in the links I provided a page back. The DoS issue wont be effected by the number of channels or docsis 3 VS 3.1 as its simple the PPS and a too small table. This issue is present in Puma 5, 6 and 7. Its pretty clear this cannot be fixed with a firmware update without serious side effects. This underlying issue of too small a table size effects a great deal of different network flows. On top of this the single core design running Linux rather then a real time OS creates issues that really cannot be overcome without more cores. The silicon seems to be fundamentally flawed in design.

Discussing number of DS channels and 3.0 VS 3.1 and how this effects things is almost irrelevant as the chip is clearly flawed and firmware does not appear to have much hope of addressing it.

If you feel otherwise I would encourage you to come have a discussion over at the thread on DSLR as we have a number of people there who have a good understanding technically of the issues involved along with Mackey who has root on a Puma 6 modem.

What concerns me is that everyone needs to stay focused and continue to apply pressure. Distractions like effects from the number of DS channels and 3.0 and 3.1 cant help resolve the issue. We know what the problem is now. Problem isolation is over. Its the chip and it appears to be baked into the silicon. VM needs to stop pushing these into the market as they appear to be flawed. Any work toward a Hub 4 based on the Puma 7 need to halt. Broadcom needs to be brought into the picture. This is going to be a difficult road. However you guys across the pond need to start pushing this.

So my views on this are that discussion about problem isolation are over. Its time to discuss how to fix the problem, how to move VM over to Broadcom.

I think you're missing where I am going.

Ok, so perfect world for us on VM is we either get a Hub 4 that has a broadcom chipset and we get a firmware patch. We aren't getting 2ac's getting put back in stock and we aren't getting 2nd hand modems being provisioned.

So... if we postulate that the hub 4 is a puma 7 and the rumours are true and that it comes alongside 3.1, then as we have no access to said device, it is of interest to know the severity of it like for like with a puma 6 (3.0) and the when using Docsis 3.1. Why? Well:

  1. It could mean that VM take the stance of, ok, it's not great, but it's better and go with it anyway.
  2. If it's the same or worse, it might open the door to a broadcom device, which would be good (however doubtful)

So yes, it might not be 'helping the case', but we're talking about the hub 3 now, which was pushed out regardless of the issues at the time. 

--------------------------------------------------------
Look behind you, a three-headed monkey

God I wish we could use a cable modem of our own choosing, they could even carry on supplying their flawed SH3 as a freebie with the "it works fine for most people".

Of course, it's not going to happen, so we'll just have to wait until the SH4 and hope that it isn't based on a flawed chipset.


@fizzyade wrote:

God I wish we could use a cable modem of our own choosing, they could even carry on supplying their flawed SH3 as a freebie with the "it works fine for most people".

Of course, it's not going to happen, so we'll just have to wait until the SH4 and hope that it isn't based on a flawed chipset.


So just thinking through this, the premise is that you normally buy any cable modem and it should work theoretically with most cable service providers?

Then the reason for VM not allowing this at all is more a business decision then a technical one. What for shane to come out and tell us how extremely difficult it would be for VM to do this. sod that though! If they really wanted to, they could make the required changes to their network for anyone greatly affected by the issue with a nice big disclaimer that they wont support you if you end up having any modem issues.

But that wont happen as we know as VM decided to lock their network down and only provide one type of modem, certainly not in the interest of helping their customers have more choice...

 


@boltedenergy wrote:

Then the reason for VM not allowing this at all is more a business decision then a technical one. What for shane to come out and tell us how extremely difficult it would be for VM to do this. sod that though! If they really wanted to, they could make the required changes to their network for anyone greatly affected by the issue with a nice big disclaimer that they wont support you if you end up having any modem issues.

 


 Oh no, its not just for support reasons! Virgin Media is scared crazy of allowing third-party devices on the network - because several years ago they had massive troubles with hacked/cloned modems. Since then they tried everything to tighten security and now it lead to situation when they over-tightened it, and still scared (even when DOCSIS 3 is much more secure than DOSCIS 1 was), unable ever to re-provision own devices they have on hand 😉

Here is and example from these times back - https://www.theregister.co.uk/2009/03/19/virgin_media_cloned_modems/