Hub 3.0 random rebooting / restarting (daily)

[timmyb]

Hello.

I am documenting this issue here because I have noticed, while this forum contains a lot of problems, there are very few (if any) documented solutions. Flagged "Answers" generally are not answers at all.

The problem I am experiencing seems to be very well documented here (although no solution is listed):

https://community.virginmedia.com/t5/Networking-and-WiFi/Hub-3-0-Random-Reboots/m-p/4816684

My experience with VM support has ranged from "OK" to diabolical, but I will not go into details unless necessary because I think the (poor) level of support is already well documented.

So here goes:

My VM account is about 5 months old although only used "in anger" since January. No problems at all. I run various systems at 100% uptime so I kind of know if anything goes down. Approximately one month ago, I noticed my remote connections (hard wired) were reseting and the wifi was weird. After the first support call I was assigned an engineer but the problem appeared to resolve so I cancelled the visit. (VM tried to charge me £25, so I yelled at them. Disgusting.).

A few days later it became apparent this was not a glitch. In fact, I observed the Hub 3.0 rebooting. White light off, then flashing, then green service lights flashing as the services return. Then everything fine again. The "up time" was reset to zero in the Hub settings (under Admin).

*** There is no problem at all when the Hub is up. The problem is random rebooting. *** Nothing to do with wifi, nothing to do with "drop outs", nothing to do with low speeds, nothing to do with hub errors. *** Random rebooting occurs almost everyday, sometimes many times per day. ***

Engineer 1 visit (02/04/2022): Cleaned up the local connections, added an attenuator (pretty sure this wasn't necessary), checked the cabinet, admitted couldn't find anything substantial and left. Called me two days later (as promised) to check and I said the problem was still occurring (it was). He said he would swap the Hub 3.0 one week later, and he never showed up again. That was the last I heard of him. Hopeless.

Engineer 2 visit (14/04/2022):
Hub 3.0 and PSU swapped. Both he, and I thought this would probably solve the problem. Of course, it didn't. He was supposed to call on Saturday (16/04/2022) to verify the solution but, of cours, he didn't. The stats on both hubs look fine and both operate just fine if they're not rebooting. I have posted the stats from the recent hub (yes, the power levels look high but on the old hub they were much lower with the same behavior. Pretty sure this is irrelevant. Also the power levels, errors, etc are all very stable over time.)

So what is it? Two hubs, both looks fine, same behavior. I have done my best to remove local potential issues, such as with and without a UPS and I have observed zero problems on any of my other hardware.

My educated guess is that there is a hardware or cable line failure somewhere in VM cabinets and sytems, but probably tricky to track down. Regardless, I will obviously be booking yet another engineer to do more work. I am no stranger to IT equipment management electrical failure but it is very weird when kit reboots, and apparently isn't overheating or faulty.

Notes to VM forum staff. I appreciate your help, but:
1. Please don't spam my forum inbox with nonsense about my "ranking". I need ansnwers, not spam.
2. Please don't ask if the problem has improved lately. It hasn't, and it won't until an engineer correctly identifies the problem.
3. If there is not a resolution to this problem fairly soon, I will be seeking early cancellation since this is inadequate service and has wasted hours of my time, and some money too.

[timmyb]

The following stats were taken after the most recent reboot, only approx 2 hours ago.

Downstream bonded channels

Channel Frequency (Hz) Power (dBmV) SNR (dB) Modulation Channel ID

1	138750000	9.8	40	256 qam	1
2	146750000	9.1	38	256 qam	2
3	154750000	8.5	38	256 qam	3
4	162750000	8.3	40	256 qam	4
5	170750000	8.5	38	256 qam	5
6	178750000	8.4	38	256 qam	6
7	186750000	8.5	38	256 qam	7
8	194750000	8	40	256 qam	8
9	202750000	7.9	38	256 qam	9
10	210750000	7.9	38	256 qam	10
11	218750000	7.9	40	256 qam	11
12	226750000	7.9	40	256 qam	12
13	234750000	8	40	256 qam	13
14	242750000	8	40	256 qam	14
15	250750000	8	40	256 qam	15
16	258750000	8	40	256 qam	16
17	266750000	7.9	38	256 qam	17
18	274750000	7.8	40	256 qam	18
19	282750000	8	40	256 qam	19
20	290750000	8	38	256 qam	20
21	298750000	8.3	38	256 qam	21
22	306750000	8.1	40	256 qam	22
23	314750000	8	38	256 qam	23
24	322750000	7.9	40	256 qam	24

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	40.3	6	0
2	Locked	38.6	3	0
3	Locked	38.9	4	0
4	Locked	40.3	5	0
5	Locked	38.9	5	0
6	Locked	38.9	5	0
7	Locked	38.9	9	0
8	Locked	40.3	5	0
9	Locked	38.6	6	0
10	Locked	38.9	5	0
11	Locked	40.9	6	0
12	Locked	40.3	5	0
13	Locked	40.3	5	0
14	Locked	40.3	5	0
15	Locked	40.3	6	0
16	Locked	40.3	6	0
17	Locked	38.9	6	0
18	Locked	40.3	5	0
19	Locked	40.3	9	0
20	Locked	38.6	5	0
21	Locked	38.6	5	0
22	Locked	40.3	7	0
23	Locked	38.9	4	0
24	Locked	40.3	5	0

[timmyb]

Upstream bonded channels

Channel Frequency (Hz) Power (dBmV) Symbol Rate (ksps) Modulation Channel ID

1	25800000	44	5120	64 qam	6
2	32600000	44.3	5120	64 qam	5
3	39400000	44.3	5120	64 qam	4
4	46200000	44.5	5120	64 qam	3

Upstream bonded channels

Channel Channel Type T1 Timeouts T2 Timeouts T3 Timeouts T4 Timeouts

1	ATDMA	0	0	0	0
2	ATDMA	0	0	0	0
3	ATDMA	0	0	0	0
4	ATDMA	0	0	0	0

[timmyb]

Network Log (MAC redacted)

Time Priority Description

17/04/2022 18:55:10	notice	LAN login Success;CM-MAC=xxx;CMTS-MAC=xxx;CM-QOS=1.1;CM-VER=3.0;
01/01/1970 00:01:37	critical	No Ranging Response received - T3 time-out;CM-MAC=xxx;CMTS-MAC=xxx:bd;CM-QOS=1.1;CM-VER=3.0;
16/04/2022 13:23:40	notice	LAN login Success;CM-MAC=xxx;CMTS-MAC=xxx;CM-QOS=1.1;CM-VER=3.0;
15/04/2022 16:36:49	critical	No Ranging Response received - T3 time-out;CM-MAC=xxx;CMTS-MAC=xxx;CM-QOS=1.1;CM-VER=3.0;
15/04/2022 11:22:20	notice	LAN login Success;CM-MAC=xxx;CMTS-MAC=xxx;CM-QOS=1.1;CM-VER=3.0;
15/04/2022 03:11:31	critical	No Ranging Response received - T3 time-out;CM-MAC=xxx;CMTS-MAC=xxx;CM-QOS=1.1;CM-VER=3.0;
14/04/2022 18:33:34	notice	LAN login Success;CM-MAC=xxx:34;CMTS-MAC=xxx;CM-QOS=1.1;CM-VER=3.0;
14/04/2022 10:48:7	Error	DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=xxx;CMTS-MAC=xxx;CM-QOS=1.1;CM-VER=3.0;
14/04/2022 10:36:29	notice	LAN login Success;CM-MAC=xxx;CMTS-MAC=xxx;CM-QOS=1.1;CM-VER=3.0;
01/01/1970 00:01:41	critical	No Ranging Response received - T3 time-out;CM-MAC=xxx;CMTS-MAC=xxx:bd;CM-QOS=1.1;CM-VER=3.0;

[timmyb]

Update:

Engineer 2 called me back on 18/04 (better late than never) and also left a contact number (not sure if this was intentional). He said he would authorise another visit asap. In my despare I had not got around to attempting to arrange a follow up through the dreadful helpdesk. I chased him again 2 days later (20/04) and I guess I got "lucky" because an experienced technician happened to be available later the same day, when I was free.

Engineer 3 seemed thorough. Further testing and inspection did not reveal anything more obvious but I pointed out that the cable outside the house looked pretty old, and possibly even damaged (I went to look at this after the router swap did not help the situation). He agreed and ordered a "repull"(?) - basically rewire my place to the cabinet. This will be performed on 04 May. Since there is nothing else obvious, it is assumed that this will fix the problem.

In the interim, I have switched the Hub 3.0 into modem mode and purchased a proprietary router. Hub 3.0 resets have now largely ceased, although not entirely. Furthermore, I have been more closely monitoring network performance and I can see there are frequent periods of high packet loss (up to 30%). Speeds are generally still good but things grind to a halt when the disruption occurs. It seems likely that the Hub 3.0 cannot handle continuous traffic during disruption, and simply falls over. Modem mode seems ot have eased this problem somewhat - now it just gets slow but doesn't lose all connections.

Note also that the connection performance between the proprietary router (TPLink) and my (wired) hardware is *vastly* superior to the Hub 3.0. Specifically, ping times to the Hub 3.0 (under zero load) are ~ 2 ms which is very slow on a LAN. By comparison, ping times to the TPLink are 0.2 ms. (i.e. 10 times faster).

[Tudor]

I too suspect a cable or cabinet fault and yes the VM hubs always work better in modem mode when there are circuit problems. 
What happens is when the hub loses connection (for whatever reason) with the CMTS (the VM device at the other end of your cable) it goes into a loop until it can reconnect to the CMTS. This externally looks like the hub is rebooting, it is not. Because the hubs basically have underpowered CPU while they are in a retry loop all other functions (if they are in router mode) are paused. 

[alexgrist]

The Hub 3 uses a Puma 6 CPU which suffers a major flaw and instability, you can find articles about it pretty easily. Virgin Media have been well aware of it for years, the Hub 5 doesn't use a Puma CPU so it shouldn't suffer the issues but good luck getting one.

Been having the same issues as yourself on and off for at least 3 years now.

[timmyb]

Thanks for the input. It is very frustrating being tied to VM hardware, although I am very interested to see how the connection performs once the line is replaced. However, as of next week I am testing driving a full fibre connection that is now available in my area, and I suspect I won't be looking back.

[DDMuhrcu]

I've been having these very same issues for the past 2 months. It started when there was a major outage on the service two days in one week. Since then the service has been behaving very much like yours. Very similar story to yours, multiple engineer visits, I'm sure you also had the helpline try and blame things on your side too. I'm seeing the problem is rife from other threads much like this one. Chatting to the engineers they were sure the issue was upstream from the box. I'm in central London so would be very surprised that this is something that is only effecting us.

[Kain_W]

Hi timmyb,

Thanks for your post and welcome back to the community!
Apologies for all the issues faced, from checking our service I can that an appointment is booked in for a cable pull.

Let us know whether that helps with this.

Thanks,