Hub 4 modem mode: regular issues with SSL traffic

[heliosfa]

Hello,

For the past decade or so I have been running a Virgin connection with the hub in modem mode connected to a pfsense firewall. For the most part, my setup has was working fine on the 200 Mb/s service with a Hub 2ac for a fair few years. I upgraded on the 18th of January to the 500 Mb/s service with a Hub 3 and this was also working fine. The problems started on the 26th of January when I was upgraded to Gig1 with a Hub 4 as a Volt benefit.

Since then, I am having a regular issues with SSL traffic to *some* destinations breaking when the Hub 4 is in modem mode. In Firefox, this manifests as the browser hanging on "Performing TLS Handshake" until it times out. In Chrome it hangs on "Establishing Secure Connection".

Some of the problem sites include:

• https://www.bbc.co.uk/news
• https://www.hobbycraft.co.uk
• https://community.virginmedia.com
• https://www.netflix.com

When the problem occurs, I also have issues tunneling traffic (e.g. remote desktop) over SSH. Other websites (e.g. https://old.reddit.com and https://www.speedtest.net ) and anything routed over my Hurricane Electric IPv6 tunnel continue to work absolutely fine. Online game traffic (CS:Go, Tarkov, etc.) also seems to be unaffected. Speed tests continue to report the full expected speed.

Pings to the problem sites continue to work. Even the redirect from news.bbc.co.uk to www.bbc.co.uk/news works, the browser just hangs after the redirect.

Sometimes the problem is transient and resolves itself after a few minutes. Other times it takes a reboot of the Hub 4 to resolve. Rebooting the firewall does not resolve the issue and the issue occurs with other devices (including an alternative router and a straight Windows desktop) in place of the firewall.

Packet captures show that the traffic is correctly leaving the firewall's WAN interface.

Also, the admin interface for the Hub 4 at 192.168.100.1 stops responding within an hour or two of rebooting the hub. Ping to the hub also become sporadic with only one in every ten or 20 pings eliciting a response. It takes a reboot of the Hub 4 to regain access to the admin interface.

The issues (including the admin interface one) do not occur if I introduce a double-NAT scenario with the Hub 4 in "normal" mode with the firewall in the DMZ. Obviously this is not an ideal long-term solution.

Has anyone else come across this? is it a "feature" of the Hub 4 or have I just got a bad one?

I am dreading trying to explain this to support over the phone as it is an intermittent issue and involves modem mode.

[jem101]

---

@heliosfa wrote:

---

@jem101 wrote:

It’s the Hub isn’t it?

I guess one good thing now is that I have a lot of diagnostic ammo to throw at them.

---

Ah yes, good luck with that one, as everything discussed in this thread is going to be so far over the heads of the customer support staff, as to be a very faint ‘whoosh’ in the distance. To be fair to them, the vast majority of users’ issues can be sorted by a quick ‘turn it off and on again’ - which apparently works for you, the speed is OK, so from their perspective, there’s nothing wrong!

So what can we try here? The first thing would be to get another Hub 4, on the grounds that it might just be a bad one-off, you’ll need to convince customer services that it has failed and the only way I can think of is to switch it off, call up and say you have no internet access. They’ll run some tests, ask you to turn it off and on again, check what lights are showing - you’ll need to be creative and make something up. Hopefully, they’ll conclude that the Hub is dead and organise a replacement?

Now if a new Hub 4 doesn’t work either, or they won’t play ball and replace it, then the next step is to call up and follow the ‘thinking of leaving us’ route - best done about 8am when they open. Explain that you have ‘requirements that the current hub won’t work with and that the previous Hub 3 did’, say that you have no option but to leave on the grounds that the equipment they supplied isn’t fit for purpose (they’ll probably argue that one, but be firm), but you would stay if they were to replace the Hub 4 with a 3 and you understand that the speed would need to be reduced to 600 Mb/s (which is the highest the 3 will sustain).

It’s going to be a bit of a mission, so good luck!

John

[legacy1]

I can't see it being the hub 4 in modem mode why would it just drop full size packets and would others with a hub 4 have this problems when in modem mode?

[Timwilky]

I run hub 4 gig1 with pfsense 2.5.2

HA proxy, pfBlockerNG, OpenVPN Server, IPSec VPNs, HE IPv6 tunnel, Hub4 is connected via a 2 port balanced LAGG.

I have acme certs, with wildcard and redirection of incomming http connections to https.

I have never had an issue with failed outgoing connections, so you see outgoing traffic on your WAN but no return packets.

My first move would be to put a switch between the hub and pfSense with port mirroring and sniff the mirror. The objective being to ensure your pfsense is not blocking anything incomming.

next, save your rule sets and revert to a basic system, no ipv6, outgoing allow all. nothing preventing incoming etc and then what happens? 

[heliosfa]

> My first move would be to put a switch between the hub and pfSense with port mirroring and sniff the mirror. The objective being to ensure your pfsense is not blocking anything incomming.

Did that early on (with an actual gigabit network tap rather than a swich with mirror port) and pfsense was emitting packets it claimed it was and nothing was coming back that pfsense did not see.

> next, save your rule sets and revert to a basic system, no ipv6, outgoing allow all. nothing preventing incoming etc and then what happens?

I have already tried with a different router, a straight up Windows 10 system and with a Windows 10 laptop with the pfsense's MAC cloned to it. All of these exhibit the same problem intermittently.

Early on when I was diagnosing, I disabled all of the inbound deny rules, IPv6 and VPNs. The same thing happened.

I am 99% sure it is not the pfsense box.

[jem101]

---

@legacy1 wrote:
I can't see it being the hub 4 in modem mode why would it just drop full size packets and would others with a hub 4 have this problems when in modem mode?

---

Unless, like I said, this is a particular issue with THIS hub or a subset of them.

Otherwise you do need to wonder what a remarkable coincidence it is that the CMTS suddenly decided to randomly change MTU sizes or the peering (strictly speaking it would be the routing not the peering), went weird, exactly at the same time as the OP changed to Hub 4.

Either the OP is the unluckiest person in the world, or it’s deliberate and VM really have it in for him, or it’s an issue with the Hub.

Occam’s Razor anyone?

[BaldrickBravo]

---

@heliosfa wrote:

If you want something else that will bake your noodle, try this:

I have access to other Virgin Media connections with pfsense behind a hub in modem mode in other parts of the UK.

Earlier, if I sent 1472-byte pings (1500-byte packets) from the pfsense box on the problem connection to one of the others, the other end saw the packet and replied. The reply just never got back to me. Pings from one of the others were never seen by the pfsense box on the problem connection.

I then went out to drop off a package. When I returned about 20 minutes later, I re-ran the same tests just to check and the situation was reversed. The pfsense on the problem connection could receive 1472-byte pings from the other boxes but the replies were never seen.

This was all verified with packet captures on the WAN interfaces of all the pfsense boxes involved.

The other pfsense boxes were able to send 1472-byte pings (and get a response) from my CMTS and from other connected cable modems on the same subnet as me.

---

Well routing is often asymmetric, so that offers a possible explanation. But I don't fully buy that. I'd expect this to affect other people in one way or another if it was peering.

If it was the hub, I can't understand why some sites present the issue and others don't. The one time I dealt with a problem like this in the wild, it was with a small commercial ISP. They made routing changes and the problem went away.

How often does the problem occur and how long did you run the hub in router mode for?

[heliosfa]

---

@BaldrickBravo wrote:

Well routing is often asymmetric, so that offers a possible explanation. But I don't fully buy that. I'd expect this to affect other people in one way or another if it was peering.

---

Indeed asymmetric routing is common, but not between me and the CMTS as it is the only gateway option I have.

---

@BaldrickBravo wrote:

If it was the hub, I can't understand why some sites present the issue and others don't. The one time I dealt with a problem like this in the wild, it was with a small commercial ISP. They made routing changes and the problem went away.

How often does the problem occur and how long did you run the hub in router mode for?

---

If it was upstream routing, I would not expect to see a problem intermittently to the CMTS.

On average, the problem occurs at least a couple of times per day. Some days, it is more frequent and others it does not happen at all. Yesterday everything was fine for most of the day and then I had several hours of on-and-off issues, even with hub reboots. A couple of days ago (when I had a load of diagnostics to run...) I had no problem periods at all.

The periods of upset are also inconsistent - sometimes, it is transient and the problem sites have a "hiccup" and will work when refreshed. Other times it will take a few minutes - with the diagnostics I have been doing, I have seen a site testing with an MTU of 1482 consistently for five-10 minutes and then just suddenly go back to an MTU of 1500. Other times still, it can take a reboot to get things back to "normal".

I ran the hub in router mode for a about three days with no issue. I put it into router mode on a day when I had rebooted it multiple times. The day I took it out of router mode, issues started again.

[heliosfa]

So, I wrote a quick script to keep an eye on the achievable MTU from something in my network to a few different websites. Here is the current view from it

Yes, simultaneously I have an MTU of 1482 to my upstream gateway and an MTU of 1500 to Google's DNS.

Full graphs can be seen here: https://iotplotter.com/user/feed/973010392123870620

[heliosfa]

Today has been particularly bad with the problem reoccurring far more regularly than normal. My MTU monitoring has been picking it up quite nicely:

I called Virgin and the tech I spoke to understood the issue fairly well. He want it leaving in router mode for a few days and then "we'll see".

Interestingly, he seems to have had issues accessing the hub management functions from their end.

Let's see what the monitoring picks up...

They cannot currently replace the Hub 4 as they have no stock of the 4 or 5 (I am guessing that is just for "support" stock...).

He did say that they have been having lots of problems with the 4 compared to the 2 and 3?!

[BaldrickBravo]

I do like a good graph.
And that has to be the first time I've seen MTU size plotted on a graph, so that's doubly nice.

Good luck getting the problem resolved.