Hub 4 modem mode: regular issues with SSL traffic

[heliosfa]

Hello,

For the past decade or so I have been running a Virgin connection with the hub in modem mode connected to a pfsense firewall. For the most part, my setup has was working fine on the 200 Mb/s service with a Hub 2ac for a fair few years. I upgraded on the 18th of January to the 500 Mb/s service with a Hub 3 and this was also working fine. The problems started on the 26th of January when I was upgraded to Gig1 with a Hub 4 as a Volt benefit.

Since then, I am having a regular issues with SSL traffic to *some* destinations breaking when the Hub 4 is in modem mode. In Firefox, this manifests as the browser hanging on "Performing TLS Handshake" until it times out. In Chrome it hangs on "Establishing Secure Connection".

Some of the problem sites include:

• https://www.bbc.co.uk/news
• https://www.hobbycraft.co.uk
• https://community.virginmedia.com
• https://www.netflix.com

When the problem occurs, I also have issues tunneling traffic (e.g. remote desktop) over SSH. Other websites (e.g. https://old.reddit.com and https://www.speedtest.net ) and anything routed over my Hurricane Electric IPv6 tunnel continue to work absolutely fine. Online game traffic (CS:Go, Tarkov, etc.) also seems to be unaffected. Speed tests continue to report the full expected speed.

Pings to the problem sites continue to work. Even the redirect from news.bbc.co.uk to www.bbc.co.uk/news works, the browser just hangs after the redirect.

Sometimes the problem is transient and resolves itself after a few minutes. Other times it takes a reboot of the Hub 4 to resolve. Rebooting the firewall does not resolve the issue and the issue occurs with other devices (including an alternative router and a straight Windows desktop) in place of the firewall.

Packet captures show that the traffic is correctly leaving the firewall's WAN interface.

Also, the admin interface for the Hub 4 at 192.168.100.1 stops responding within an hour or two of rebooting the hub. Ping to the hub also become sporadic with only one in every ten or 20 pings eliciting a response. It takes a reboot of the Hub 4 to regain access to the admin interface.

The issues (including the admin interface one) do not occur if I introduce a double-NAT scenario with the Hub 4 in "normal" mode with the firewall in the DMZ. Obviously this is not an ideal long-term solution.

Has anyone else come across this? is it a "feature" of the Hub 4 or have I just got a bad one?

I am dreading trying to explain this to support over the phone as it is an intermittent issue and involves modem mode.

[heliosfa]

OK, time for a very belated update.

Nathan contacted me by PM but we seem to be going around in circles with him trying to blame it on my setup.

There is some progress in the debugging - it looks like the Hub 4 does not like having multiple IPSEC VPN tunnels passing over it in modem mode. If I only have one tunnel up, everything is mostly fine. If I have more than one up, the problems get progressively worse. This is true irrespective of what is directly behind the Hub 4 or initialising the IPSEC connections.

In other words, the Hub 4 seems to be sensitive to the traffic passing over it when it is in modem mode. This is clearly not what should happen and must be down to a firmware bug.

After some back and forth, Nathan arranged for another engineer to visit and they replaced the Hub 4 again. This did not resolve the issue and the Engineer left notes stating that I would require either a downgrade to a Hub 3 (with the consequent service downgrade) or putting into the Hub 5 trial to see if either would resolve the issue. Unfortunately Nathan and VM seem to have other ideas and are refusing to implement either of these potential resolutions that their own engineer has stated are required...

One step forward and five steps back - it is now three months since I have had a properly working connection. Upgrades should not be regressive...

[Anonymous]

A timely reminder that there is no modem mode, there is only a bridge mode. There's no NAT or TTL decrementing so nothing appears in a traceroute but the hub is absolutely not outputting over the coax what arrives on the LAN.

Multiple IPSEC VPNs traversing bridges that are layer 4 aware is often a pain point. 

[Anonymous]

---

@legacy1 wrote:

With the hub in router mode if the MTU flips as the hub says whats my MTU what my MTU when the MTU drops the hub being a router sees a full size packets and sends a ICMP for the remote end to send smaller packets. with the hub in modem mode the hub should not need to check the MTU... but it likely acking like a switch with a faulty Maximum Frame Size in storage flapping about that when a full size packets do happen it is just dropped no ICMP to tell the remote end to send smaller packets.

---

Equipment pulls interface MTUs from configuration at start up and separately when interfaces previously disabled come up. Change an interface MTU the interface bounces to take effect. Once this is done they don't change and are held in RAM.

The VM Hubs do not act like switches regardless of mode. They do extensive layer 4 processing whatever on anything going through the WAN. 

No idea where the 18 bytes came from, though. 

Sorry I didn't see this earlier. I've seen more TLS faults due to MTU than anyone could wish to and that was my immediate thought here. Unencrypted exchanges may be fragmented, TLS handshakes may not due to the need to cryptographic sign as you go. Can't verify an HMAC when you don't know what the other side is doing the HMAC on. 

[legacy1]

---

@heliosfa wrote:

There is some progress in the debugging - it looks like the Hub 4 does not like having multiple IPSEC VPN tunnels passing over it in modem mode. If I only have one tunnel up, everything is mostly fine.

---

Are any of the tunnels using protocol 50 or UDP when the MTU limiting bug happens? if all tunnels use UDP do you have a problem?

[heliosfa]

Sorry for the very late reply on this - end of academic year fun with marking, exams, etc.

So, to answer the setup questions: The VPN is a standard setup in pfsense with NAT Traversal set to Auto. I'll give trying to force it not to use ESP a go. I'll also try it on custom ports.

Interestingly, if I disable the two IPv4 IPSEC tunnels and just leave the two IPv6 IPSEC tunnels that run over the Hurricane Electric tunnel (or disable everything and leave a single IPv4 IPSEC tunnel), everything seems to be significantly better. On the flip side, if I just have one IPv4 IPSEC tunnel and leave the IPv6 IPSEC tunnels enabled, the problem remains.

Also, the web interface stopping responding in modem mode seems to be unrelated to the IPSEC induced issues.

[bmhughes]

In case someone stumbles across this thread, I've had exactly the same issue after standing up an additional IPsec tunnel that was using ESP encapsulation. After forcing NAT-T encapsulation the issue has resolved, yet another wonderful superhub bug.

After all of the issues I had with the SH3 and non-TCP traffic I can say that all of the Intel Puma based platforms are exceptionally poor products.

Here's hoping the SH5 with a Broadcom SoC again is an improvement.

[heliosfa]

So despite this supposedly being logged as a security issue, nothing seems to have been done to resolve it.

NAT-T is a work around, but fair warning: unsolicited, invalid inbound IPSEC traffic from more than one external source will still trigger the issue.
Yes, you read that right, you don't even have to have an established session for this problem to rear it's head and any Hub 4 in modem mode is vulnerable to this.

After a lot of delay and waiting, I was finally able to get a Hub 5 out of Virgin a couple of months ago.
I have not had the chance to check whether it suffers from the same issues yet as I have been busy and the Virgin Gig1 connection has been relegated to backup duty as I am now enjoying symmetric gigabit FTTP (with native IPv6...) from a local ISP.
Checking the Hub 5 is on my todo list for some point in January when the students are busy with exams...

What I can say about the Hub 5 for now though is that it does not test very well on iPerf 3 - rather than over a gigabit steady as I got with the Hub 4, the Hub 5 seems to fluctuate about 600 Mb/s. Normal speed testing and usage does push gigabit, so it just seems to be something with iPerf 3.

[bmhughes]

You can DoS a SH3 in a similar method with a not-unreasable amount of anything that isn't TCP. At least thats more obvious to troubleshoot as the latency rockets skyward before it falls over.

Frustrating when the SH1 and 2 were pretty much bulletproof by comparison.