IPv6 support on Virgin media

Long time since an update, but thought I'd post some additional information.

A while ago I was in contact with a member of the Liberty Global network team (can't reveal specific details at their request), who had seen the posts around the IPv6 tunnels issues and wanted more information (the power of LinkedIn!). By now most are aware what has been discovered is despite all previous doubts and finger pointing the 6in4 speed issue is a hardware one, not artificially created by Virgin Media i.e. deliberately slowing the traffic/routing etc. From their tests, they could not find any issues with the speed of 6in4 within their network only when behind a CPE, so the CPE was being pointed at as per the comments from their spokesperson in the ISPReview articles. This has in fact now been found to be the issue, more and more people have been able to confirm that the Hub4 does not have the same issues with 6in4, because of it's hardware being faster but this is probably by chance than specifically fixing this problem. The CPU processing power in the older modems is the bottleneck it would seem. The articles from ISPReview hint at a firmware update, I'm honestly sceptical about anything happening here. The lead time for firmware updates is long, it affects all previous modems other than the Hub4 so multiple vendors and ultimately 6in4 is niche. At this point the logical option would be to simply acknowledge the 6in4 speed issues (which they have done publicly) and just put all resources on a native IPv6 rollout, resolving two problems, the 6in4 issue becomes redundant.

Most will also be aware the Hub4 is only available to those who are in DOCSIS 3.1 area, this has been steadily increasing over last few months and will continue into 2021. In the beginning Virgin Media would not let you get the Hub4 without being on the Gig1 package, this however is no longer true, despite some of their support staff telling you otherwise. Virgin Media have been offering the Hub4 as a free upgrade to some, although the motivation for this, is simply to help the DOCSIS 3.0 network with congestion. I don't believe you can just request the Hub4 even if your area is DOCSIS 3.1 enabled (unless you take the Gig1 package which the Hub4 is mandatory by default). Virgin Media still seem at this time to be approaching customers directly for this. Despite not being in a DOCSIS 3.1 area, at one point I was going to be offered the Hub4 to test the theory through some internal channels. While the Hub4 can technically work on DOCSIS 3.0, ultimately I decided to not take this, because I would be basically waiving all standard support channels if anything went wrong given it is not a standard deployment and given work from home, this seemed too risky for something like 6in4. I have since moved on from 6in4 and instead use the L2TP service from AAISP. I also have another WAN now (Vodafone FTTC) which I can point a 6in4 tunnel on and get full speed, given it isn't behind a Virgin Media CPE, but ultimately 6in4 shouldn't be the permanent solution either anyway.

Regarding an IPv6 deployment from Virgin Media, there was very little information about this from the contact I spoke to. I sense the concern is they don't want to state anything because they can't officially speak on behalf of Virgin Media, they were however OK with mentioning the Hub4 as a "solution", although it isn't really, a native IPv6 deployment is. I got the impression that the 6in4 scenario getting some traction did start some internal discussions within VM on IPv6 and perhaps conversation were had, but nothing seems to have materialised since. Virgin Media Ireland is still going strong with DS-Lite IPv6, I've seen various posts from VM Ireland customers confused as to why port forwarding and modem mode options aren't present, until you explain that VM Ireland deploy DS-Lite by default and you have to request to be switched back to IPv4 only for those options to be enabled in the firmware. Many seem to think DS-Lite is dead for Virgin Media in the UK, given the IPv6 trials were mid 2018 and absolutely nothing public has happened since, suggesting they may have decided to abandon it. They weren't exactly in a rush to deploy IPv6 anyway but it possible they've decided it's not a good idea. The evidence of DS-Lite was strong, given parent company Liberty Global absolutely loves it and it's deployed in a lots of a other countries where they are but rumour says they might not. That however is entirely opinion and there's no official comment on DS-Lite being abandoned.

---

jamesmacwhite wrote:

Long time since an update, but thought I'd post some additional information.

The CPU processing power in the older modems is the bottleneck it would seem. 

there is nothing to process for the VM hardware, it just an IPv4 packet, the processing is done on the router after the HH which is in dumb router mode. Even the old HG612 has no issues "processing" 6in4. Also, we saw several reports users had good speed at times, with the same hardware. I strongly doubt it is related to hardware, especially bottlenecks in CPU on Hubs in router mode, I would say it highlights the incompetence of VM even more, they are not able to find the throttle configuration in their own network, not a surprise they are not able to roll out IPv6. 

ksim wrote:

---

jamesmacwhite wrote:

Long time since an update, but thought I'd post some additional information.

The CPU processing power in the older modems is the bottleneck it would seem. 

there is nothing to process for the VM hardware, it just an IPv4 packet, the processing is done on the router after the HH which is in dumb router mode. Even the old HG612 has no issues "processing" 6in4. Also, we saw several reports users had good speed at times, with the same hardware. I strongly doubt it is related to hardware, especially bottlenecks in CPU on Hubs in router mode, I would say it highlights the incompetence of VM even more, they are not able to find the throttle configuration in their own network, not a surprise they are not able to roll out IPv6. 

---

Well, there was this at https://www.ispreview.co.uk/index.php/2018/08/intel-coughs-to-puma-cpu-flaw-that-hit-virgin-media-hub-3-router.html

"At the root of all this is the fact that the somewhat weak CPU (processor) inside the modem component of Virgin’s router (Puma 6) was taking on too much work while processing network packets, which caused the chipset to run a high-priority maintenance task every few seconds. Sadly this extra workload ended up causing momentary latency spikes (increases of 200 milliseconds+), plus a little packet loss."

Aye. It's been confirmed through actual testing as an SH3 only problem. Doesn't surprise at all. 6in4 uses a non-standard IP protocol so likely punted to a slow path in the device. Given the travails they had getting decent performance it's not surprising that Arris, Intel, et Al ignored/deprioritised optimisation and testing of anything other than ICMP, TCP and UDP. For this reason I can't see a FW release ever being forthcoming.

they can tell any **bleep** they want, it doesn't make it true,
first, there were no spikes or inconstancy in 6in4 packages, you can see a clear cap at 20Mbit/s for 6in4, which doesn't affect any other traffic anyhow.
second processing 6in4 or TCP or UDP  is no different for SH in modem mode, it just sens all traffic as it is.
third, SH is not something to blame based on tests users on this forum provided.

Was it confirmed?
https://community.virginmedia.com/t5/QuickStart-set-up-and/IPv6-support-on-Virgin-media/m-p/4357314/highlight/true#M149714
https://community.virginmedia.com/t5/QuickStart-set-up-and/IPv6-support-on-Virgin-media/m-p/4357343/highlight/true#M149718
https://community.virginmedia.com/t5/QuickStart-set-up-and/IPv6-support-on-Virgin-media/m-p/4358450/highlight/true#M149784

6in4 is a standard protocol described in RFC 4213.

There is no optimization needed in modem mode, you just do not process packages and send them as "it is". there is no routing/prioritization/filtering involved.

SH3 and SH4 were tested back to back on the same line. SH4 achieves near line rate performance with 6in4, SH3 throttles to 20Mbps (as we all see and report). SH2 has been reported to work with similar performance.

---

ksim wrote:

There is no optimization needed in modem mode, you just do not process packages and send them as "it is". there is no routing/prioritization/filtering involved.

---

LOL. That's true in theory but unfortunately modern hardware and software are a lot more complex. The Intel Puma architecture has lots of clever hardware offloading and QoS queues independent of the application processor that runs Linux and does the routing (or modem mode). Modem mode is likely just a high level configuration change that flips from NAT masquerade to a bridging configuration on the Linux side of the fence. The underlying network interface and DOCSIS engine will be unaltered. From my understanding of the problems with the Puma system it was this area that had the performance problems. Intel were being too clever for their own good.

---

Anonymous wrote:

SH3 and SH4 were tested back to back on the same line. SH4 achieves near line rate performance with 6in4, SH3 throttles to 20Mbps (as we all see and report). SH2 has been reported to work with similar performance.

tested by whom? who confirmed that? published result? Can you explain Andy's results? and others who reported similar things.

LOL. That's true in theory but unfortunately modern hardware and software are a lot more complex.

is it? an example of a modem/router with the same 6in4 issues?

The Intel Puma architecture

---

has nothing to do with the issue.

Hi,

Just for the hell of it, I've just done another speed test (272 MBits/sec IPv4, 259 MBits/sec IPv6):

Still on SuperHub 3. I think we can say that almost certainly rules out any hardware issue. Don't necessarily expect it to stay like this for too long though...

Andy

I can't speak directly to the SH hardware VM are using, but I work with high-end Cisco switches and routers at work and we see similar problems to this regularly. The ASICs on linecards are highly optimised to process certain types of traffic very, very quickly to be able to maintain line speed. Any type of traffic that isn't specifically optimised gets handed over to the CPU instead. But if you see a large amount of this unexpected traffic (nb. I'm not saying this NON-STANDARD, it's all in the RFCs, it's just traffic that the hardware hasn't been OPTIMISED for) it can overwhelm the CPU in a badly designed system, or just result in poor performance for that traffic type. As I say, I have no knowledge of what's going on inside this hardware, but it seems very possible that this could be the issue here. It may not be on the CPE, either, but somewhere further upstream in the VM/LG network.

---

WiteWulf wrote:

I can't speak directly to the SH hardware VM are using, but I work with high-end Cisco switches and routers at work and we see similar problems to this regularly. The ASICs on linecards are highly optimised to process certain types of traffic very, very quickly to be able to maintain line speed.

We are on the same page here, yes high-end routers do that, but it is configurable behavior, you can configure it based on any protocol number. 6in4 is really simple if you not trying to extract encapsulated information from it, which makes no sense to do.

Any type of traffic that isn't specifically optimised gets handed over to the CPU instead. But if you see a large amount of this unexpected traffic (nb. I'm not saying this NON-STANDARD, it's all in the RFCs, it's just traffic that the hardware hasn't been OPTIMISED for) it can overwhelm the CPU in a badly designed system, or just result in poor performance for that traffic type.

but this "overwhelm" doesn't result in a constant 12mbit cap.

It may not be on the CPE, either, but somewhere further upstream in the VM/LG network.

---

This is what we are talking about, the issue is in the VM network, but the hardware used in the network is capable to cope with 6in4 with no problem unless configured to limit traffic.

I'm not wanting to be argumentative, nor completely agree/disagree with each of the assertions, however there are accepted flaws in the Intel Puma 6 CPU. Just look at the US class action lawsuit on the matter (e.g. Schubert Jonckheer & Kolbe Puma 6 Chipset Defect). When there is such an issue that affects a hardware platform, and in this case I would call it random, for the processing of packet data then all bets are off for other data types that are not well knows and processed through a hardware path. All other packets regardless of router or modem mode will traverse through the CPU. In Linux, when the in bridge mode the packets still use the kernel and thus the Puma CPU. This bit is speculation, but given the GUI is still accessible, the Hub 3 is running other applications which will take resource from the CPU at some level - and in an already constrained system could cause issues regardless of mode. I'm not a HFC expert, but as I understand from reading each consumer segment (CPE) also receives each of the packets (e.g. for neighbours) and filters them, not to mention the variation of local network traffic requiring WAN access that can affect any individual setup.

I also suspect there will be varying E2E network "issues" (e.g. network QoS or DPI or switch implementations) as well as those established and there may even be better binned versions of the Puma 6 (pure luck) that perform better. 

The only real way to prove the Hub 3 is the root cause, rather than just another part of the problem is via a standalone test configuration in a lab environment (and that's unlikely to happen), running with a varying degree of local network scenarios.

Disclaimer: I'm not running 6in4, just interested in the progress (or lack of) IPv6 on Virgin Media's network.

It's doable on the Hub 3, just how and when VM decide to implement it. 

---

jonathanm wrote:

accepted flaws in the Intel Puma 6 CPU.

has nothing to do with the VM 6in4 issue, not even similar or related.

This bit is speculation, but given the GUI is still accessible, the Hub 3 is running other applications which will take resource from the CPU at some level - and in an already constrained system could cause issues regardless of mode. I'm not a HFC expert, but as I understand from reading each consumer segment (CPE) also receives each of the packets (e.g. for neighbours) and filters them, not to mention the variation of local network traffic requiring WAN access that can affect any individual setup.

can't see how any of this will affect 6in4, even in full software mode it is a very light and trivial task, there is no difference in processing 6in4 or UDP package in this matter for the router mode. additionally, examples on the forum ruled out your theory several times, thanks Andy who confirmed that recently again.

Disclaimer: I'm not running 6in4, just interested in the progress (or lack of) IPv6 on Virgin Media's network.

---

then try and see for yourself, you can check latency, speeds, and CPU utilization, other traffic, etc, there are no signs of high CPU usage for 6in4 traffic on Hub 3, several users confirmed having full speed at times on Hub 3. I do not understand why you suddenly decided to believe this is Hub 3 issue? because VM said so? we know they are not competent enough to even setup the tunnel.

I think his point is more of the fact that without a ton of testing, where the environment can be fixed at all times, we will never discover where the fault really is.

There's several equipment places it could be, and in those places it could be hardware or firm/software based.  Virgin are not going to invest the time and effort to find this problem.  Perhaps a few staff to look at it in passing and that's about it.

If the problem goes away (or is not initially seen) when testing on newer hardware then they will just stick to the idea that it will go away.  Plus they are probably of the view that once we move to IPv6 it wont matter.

@VMCopperUser, indeed, my point exactly. 

The rough evidence/anecdotes as I've seen it so far (as reported by users or through news articles, and I may have missed some):

• There have been no reported issues with 6in4 from Hub 4 users (I think?)
• Hub 3 users have reported sporadic successes achieving their connection speed (or near enough), with a range of 12Mbps, 20Mbps and full speed (sometimes). 
• Other network/switch issues have not been ruled out. 
• Having been a software/hardware tester something slightly nonstandard (or should I say non-typical) is likely not the most throughly tested and cannot be assumed to be trivial for the components handling it (whatever/where ever in the path they are)
• The Hub 3 has had problems and VM are, reportedly, offering users Hub 4's for free in some areas even if not on the Gigabit package. 

In short, we don't really know. 

Personally, there's enough going on in the world, work & personal projects, that testing 6in4 is not something I currently care enough about to add (plus I can only control my local network variables and can only test CPU utilisation by seeing whether it gets hotter than it already does :-D) ... Testing with a "proper" v6 implementation is something different I'd be happy to do however, for which I hope VM engineers are working on making it so 🙂 

Just to reiterate James comment. I too have been in contact with the engineering team at VM. They have run back to back performance tests of SH3 and SH4 in their lab with Hurricane Electric 6in4 tunnels. The results they saw were identical to the other 6in4 tunnel users in this thread with poor performance of the SH3.

This doesn't explain adhawkins result where the SH3 performs well. I wonder if this is because they were connected to a different flavour of UBR from the majority of the VM estate. This could interact with the SH3 in a different way. DOCSIS 3.x is a complicated beast.

Contrary to ksim's assertions (😂) the VM engineering team appear to be very competent. I totally understand why they have appeared to put this issue on the back burner. It's a pretty minority interest, sadly, even compared to the wider deployment of IPv6.

I've just had an email from virgin telling me that "we’re constantly investing in our network and to help us continue this work, the price of your package will go up by £3.50 a month".

hmm.. they're not investing it in ipv6 from the available evidence...