RPC Timeouts

[Vintageden]

We’ve never had any problems with the hub 3 until recently when the network log shows an RPC time out error and we lose our broadband connection. This can happen every day but there is no set pattern. On occasion we lose connectivity following a Kernel/oops panic error, but generally connectivity comes back without any manual intervention. I always check the Virgin Media service status but there are never any outages in our area at the time.

At first we thought it was the hub, which we’ve had for a considerable time, and arranged for an engineer to install a new hub but it is still happening.

I’ve attached the downstream and upstream stats, network error logs and the BQM. Can anyone shed any light as to why this is happening?

Where I’ve rebooted the hub, the network log shows the date as 01/01/1970 which I understand is perfectly normal.

Downstream bonded channels

Channel	Frequency (Hz)	Power (dBmV)	SNR (dB)	Modulation	Channel ID
1	139000000	2.2	37	256 qam	1
2	147000000	2.7	37	256 qam	2
3	155000000	2.5	38	256 qam	3
4	163000000	2.7	38	256 qam	4
5	171000000	2.9	38	256 qam	5
6	179000000	3.2	38	256 qam	6
7	187000000	3	38	256 qam	7
8	195000000	3	38	256 qam	8
9	203000000	3	38	256 qam	9
10	211000000	3	38	256 qam	10
11	219000000	3.2	38	256 qam	11
12	227000000	3.4	38	256 qam	12
13	235000000	3.4	38	256 qam	13
14	243000000	3.7	38	256 qam	14
15	251000000	3.5	38	256 qam	15
16	259000000	3.5	38	256 qam	16
17	267000000	3.7	38	256 qam	17
18	275000000	4	38	256 qam	18
19	283000000	3.9	38	256 qam	19
20	291000000	4.6	38	256 qam	20
21	299000000	5	38	256 qam	21
22	307000000	4.9	38	256 qam	22
23	315000000	5.4	38	256 qam	23
24	323000000	5.6	38	256 qam	24

Downstream bonded channels

Channel	Locked Status	RxMER (dB)	Pre RS Errors	Post RS Errors
1	Locked	37.6	17	0
2	Locked	37.6	6	0
3	Locked	38.6	6	0
4	Locked	38.9	4	0
5	Locked	38.9	4	0
6	Locked	38.9	18	0
7	Locked	38.9	5	0
8	Locked	38.6	4	0
9	Locked	38.6	5	0
10	Locked	38.9	7	0
11	Locked	38.9	5	0
12	Locked	38.6	6	0
13	Locked	38.9	5	0
14	Locked	38.9	6	0
15	Locked	38.6	6	0
16	Locked	38.9	6	0
17	Locked	38.6	5	0
18	Locked	38.9	6	0
19	Locked	38.9	0	0
20	Locked	38.9	6	0
21	Locked	38.9	0	0
22	Locked	38.9	6	0
23	Locked	38.9	3	0
24	Locked	38.9	5	0

Upstream bonded channels

Channel	Frequency (Hz)	Power (dBmV)	Symbol Rate (ksps)	Modulation	Channel ID
1	36600000	48	5120	64 qam	3
2	30100000	47.8	5120	64 qam	4
3	43100000	48	5120	64 qam	2
4	23600000	47.5	5120	16 qam	5
5	49600000	48.3	5120	64 qam	1

Upstream bonded channels

Channel	Channel Type	T1 Timeouts	T2 Timeouts	T3 Timeouts	T4 Timeouts
1	ATDMA	0	0	1	0
2	ATDMA	0	0	1	0
3	ATDMA	0	0	0	0
4	ATDMA	0	0	0	0
5	ATDMA	0	0	0	0

Network Log

Time	Priority	Description
14/02/2023 10:08:4	notice	LAN login Success;CM-MAC=**:**:**:**:**:**;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
01/01/1970 00:01:41	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
14/02/2023 09:07:38	notice	NOTICE ATOM is restarted due to RPC timeout as part of Self Healing Mechanism;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
13/02/2023 12:49:29	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
13/02/2023 12:49:29	Warning!	RCS Partial Service;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
12/02/2023 17:27:31	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
12/02/2023 11:55:4	notice	NOTICE ATOM is restarted due to Kernel/oops panic as part of Self Healing Mechanism ;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
11/02/2023 20:57:16	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
11/02/2023 15:36:35	notice	LAN login Success;CM-MAC=xx:db;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
11/02/2023 15:16:59	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
11/02/2023 15:14:36	notice	LAN login Success;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
01/01/1970 00:01:40	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
11/02/2023 14:37:44	notice	NOTICE ATOM is restarted due to RPC timeout as part of Self Healing Mechanism;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
10/02/2023 18:18:22	notice	NOTICE ATOM is restarted due to Kernel/oops panic as part of Self Healing Mechanism ;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
09/02/2023 16:50:1	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
09/02/2023 07:29:4	notice	NOTICE ATOM is restarted due to Kernel/oops panic as part of Self Healing Mechanism ;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
01/01/1970 00:01:40	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
07/02/2023 20:41:58	Error	DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
01/01/1970 00:01:40	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
04/02/2023 10:34:30	notice	NOTICE ATOM is restarted due to RPC timeout as part of Self Healing Mechanism;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;

[Kath_F]

Hi Vintageden, 

Thanks for taking the time to contact us via the Community. It's lovely having you on board with us in the Forums. ⭐

We're sorry to hear you've been having an issue with losing your broadband connection. Checking things this end, the hub is showing online and all power and signal levels are spot on. There is nothing to suggest an area fault is affecting you either. 

I can see a few spikes in the BQM graph but are you able to post the live URL so we can monitor things this end? 

Are you noticing the drop outs on all devices at the same time or just one? 

How are you connecting to the broadband too? 

When the dropouts occur, do you notice a different light pattern on the hub? 

Pop back and let us know so we can help further. 

Many thanks, 

[Vintageden]

Thank you for your reply.

Here is the link to the BQM:

https://www.thinkbroadband.com/broadband/monitoring/quality/share/650d10f0b9d3628d2482ff98cf857f6ce0...

The drop outs occur on all devices and for both hardwired devices and wireless. We have tried to work out if a particular device is causing the outage but there is no obvious pattern. Also when there is an outage my phone stays connected to Wi-Fi but under >Wi-Fi>settings it says ‘no internet connection.’

The hub light remains the same colour which is amber (as did the last hub we had).

[Ayisha_B]

Hi @Vintageden,

Thanks for coming back to us.

I have ran some more tests and although Sam Knows reported 7 disconnections in the last week, there's been no drop outs since Friday. 

Have you noticed any improvements over the weekend?

[Vintageden]

So far so good, no disconnects since Friday but it is probably too early to say whether its been resolved. I'll monitor it during next week and give you an update then. Has the issue been fixed? What was the problem? 

[Vintageden]

It might be worth mentioning that I've noticed a few other errors in the network log now which I haven't seen before, namely T3 timeout errors, RCS partial service and  SYNC timing error. The usual RPC timeout error which normally precedes the disconnect isn't in the log at all for the outage on Friday.

Time	Priority	Description
18/02/2023 18:31:10	notice	LAN login Success;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
01/01/1970 00:01:39	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
15/02/2023 09:45:47	notice	NOTICE ATOM is restarted due to RPC timeout as part of Self Healing Mechanism;CM-MAC=xx:db;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
14/02/2023 23:08:4	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
14/02/2023 13:50:12	Warning!	RCS Partial Service;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
14/02/2023 13:45:34	critical	SYNC Timing Synchronization failure - Loss of Sync;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
14/02/2023 12:41:7	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;

[Adduxi · Accepted solution]

Your last Upstream log showed a channel on 16 QAM. These should all be 64 QAM. Keep on eye on that as it will cause problems. 
The ATOM restart is usually when the CPU is overloaded and it restarts to reset the load. 

[Vintageden]

Thanks for your input.

I've checked the hub and there is one channel on 32 qam. I'll keep monitoring it.

Upstream bonded channels

Channel	Frequency (Hz)	Power (dBmV)	Symbol Rate (ksps)	Modulation	Channel ID
1	36600000	47.8	5120	64 qam	3
2	30100000	47.5	5120	64 qam	4
3	43100000	47.8	5120	64 qam	2
4	49600000	48	5120	64 qam	1
5	23600000	47	5120	32 qam	5

[Adri_G]

Hey Vintageden, thanks for posting on our help forum and glad to see you've joined to the community.

Sorry to see you've been experiencing issues with your connection and internet access recently.
How are things looking since your last post on Sunday?

Has the issue re-occurred and have you noticed any improvements since you've located the channel issue?

Here to help, please give us a shout here if you need further support.

[Vintageden]

Its been working well since the last outage until the network went down in the early hours at 00:17. It came back without any intervention my end about 10 minutes later. I've pulled off the network logs and I'm seeing a host of new error messages now. 

Time	Priority	Description
23/02/2023 09:58:46	notice	LAN login Success;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
23/02/2023 00:21:45	critical	No Ranging Response received - T3 time-out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
23/02/2023 00:19:29	critical	Received Response to Broadcast Maintenance Request, But no Unicast Maintenance opportunities received - T4 time out;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
23/02/2023 00:17:36	Warning!	Lost MDD Timeout;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
23/02/2023 00:17:32	critical	SYNC Timing Synchronization failure - Loss of Sync;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
23/02/2023 00:17:31	Warning!	RCS Partial Service;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
23/02/2023 00:17:31	critical	SYNC Timing Synchronization failure - Loss of Sync;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;
22/02/2023 23:14:49	Error	DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=xx;CMTS-MAC=xx;CM-QOS=1.1;CM-VER=3.0;