CF23 Internet stability and packet loss

[RedDragonTaff]

Hi,

I am at the end of my tether.  Since the 14 Sept 20 the VM network has really gone bad.  Even today this is still ongoing.  All other levels of communication Facebook, Customer Support 150/151 has fallen on deaf ears.

So tonight here's what I see on the Hub:

Network Log:
27/09/2020 21:35:13 GMT 27/09/2020 21:35:13 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:34:59 GMT 27/09/2020 21:34:59 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:34:37 GMT 27/09/2020 21:34:37 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:34:19 GMT 27/09/2020 21:34:19 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:34:00 GMT 27/09/2020 21:34:00 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:33:39 GMT 27/09/2020 21:33:39 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:33:20 GMT 27/09/2020 21:33:20 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:33:00 GMT 27/09/2020 21:33:00 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:32:40 GMT 27/09/2020 21:32:40 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:32:20 GMT 27/09/2020 21:32:20 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:32:01 GMT 27/09/2020 21:32:01 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:31:44 GMT 27/09/2020 21:31:44 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:31:21 GMT 27/09/2020 21:31:21 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:31:13 GMT 27/09/2020 21:31:13 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:31:03 GMT 27/09/2020 21:31:03 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:30:27 GMT 27/09/2020 21:30:27 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:30:18 GMT 27/09/2020 21:30:18 GMT Error (4) 68010302 DHCP WAN IP - removed
27/09/2020 21:30:04 GMT 27/09/2020 21:30:04 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:29:45 GMT 27/09/2020 21:29:45 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:29:38 GMT 27/09/2020 21:29:38 GMT Warning (5) 84020200 Lost MDD Timeout

Downstream:
DS-1 DS-2 DS-3 DS-4 DS-5 DS-6 DS-7 DS-8
Frequency (Hz) 139000000 147000000 155000000 163000000 N/A 179000000 187000000 195000000
Lock Status(QAM Lock/FEC Sync/MPEG Lock) Locked Locked Locked Locked Unlocked Locked Locked Locked
Channel ID 1 2 3 4 N/A 6 7 8
Modulation 256QAM 256QAM 256QAM 256QAM Unknown 256QAM 256QAM 256QAM
Symbol Rate (Msym/sec) 6.952000 6.952000 6.952000 6.952000 N/A 6.952000 6.952000 6.952000
Interleave Depth I=12
J=17 I=12
J=17 I=12
J=17 I=12
J=17 N/A I=12
J=17 I=12
J=17 I=12
J=17
Power Level (dBmV) -2.30 -2.44 -2.64 -2.56 N/A -2.69 -2.95 -2.87
RxMER (dB) 36.84 37.09 36.84 36.84 N/A 38.26 38.26 36.39
Pre RS Errors 327 319 297 276 N/A 295 302 966
Post RS Errors 289 287 297 275 N/A 295 302 966

Upstream:
US-1 US-2 US-3 US-4
Channel Type 2.0 2.0 2.0 2.0
Channel ID 1 4 2 3
Frequency (Hz) 60300000 39400000 53700000 46200000
Ranging Status Success Success Success Success
Modulation 64QAM 64QAM 64QAM 64QAM
Symbol Rate (Sym/sec) 5120000 5120000 5120000 5120000
Mini-Slot Size 2 2 2 2
Power Level (dBmV) 46.50 45.50 46.00 45.75
T1 Timeouts 0 0 0 0
T2 Timeouts 0 0 0 0
T3 Timeouts 1 0 0 0
T4 Timeouts 0 0 0 0

Upstream Burst:
Req
(1) Init Maint
(3) Per Maint
(4) Adv Short
(9) Adv Long
(10) Adv UGS
(11)
Modulation Type 16QAM QPSK 16QAM 64QAM 64QAM 16QAM
Differential Encoding OFF OFF OFF OFF OFF OFF
Preamble Length 56 640 384 104 104 104
Preamble Value Offset 652 0 0 716 716 716
FEC Error Correction (T) 0 5 5 10 16 5
FEC Codeword Information Bytes (K) 16 34 34 81 223 86
Maximum Burst Size 0 0 0 3 255 255
Guard Time Size 8 48 48 8 8 8
Last Codeword Length Fixed Fixed Fixed Shortened Shortened Shortened
Scrambler On/Off ON ON ON ON ON ON

Attached is the BQM graphs daily since 14th Sept.  Can someone initiate an investigation please?

Thanks

AL

[MOD EDIT: Personal and private information has been removed from this post. Please do not post personal or private information in your public posts. Please review the Forum Guidelines]

[RedDragonTaff]

Some new logs, this morning (Tue 15th June not off to a great start):

Upstream bonded channels

Channel	Frequency (Hz)	Power (dBmV)	Symbol Rate (ksps)	Modulation	Channel ID
1	32600000	50.8	5120	32 qam	5
2	53700000	51	5120	64 qam	2
3	39400000	50	5120	32 qam	4
4	46200000	51	5120	64 qam	3

Upstream bonded channels

Channel	Channel Type	T1 Timeouts	T2 Timeouts	T3 Timeouts	T4 Timeouts
1	ATDMA	0	0	0	0
2	ATDMA	0	0	1	0
3	ATDMA	0	0	0	0
4	ATDMA	0	0	2	0

Downstream bonded channels

Channel	Frequency (Hz)	Power (dBmV)	SNR (dB)	Modulation	Channel ID
1	203000000	-1.2	38	256 qam	9
2	139000000	-1.5	38	256 qam	1
3	147000000	-1.7	37	256 qam	2
4	155000000	-1.5	38	256 qam	3
5	171000000	-0.7	38	256 qam	5
6	179000000	-0.5	38	256 qam	6
7	187000000	-0.7	38	256 qam	7
8	195000000	-0.9	38	256 qam	8
9	211000000	-1.5	38	256 qam	10
10	219000000	-1.9	38	256 qam	11
11	227000000	-2.2	38	256 qam	12
12	235000000	-2.5	38	256 qam	13
13	243000000	-2.5	38	256 qam	14
14	251000000	-2.7	39	256 qam	15
15	259000000	-2.7	38	256 qam	16
16	267000000	-2.5	39	256 qam	17
17	275000000	-2.7	38	256 qam	18
18	283000000	-2	38	256 qam	19
19	291000000	-1.5	39	256 qam	20
20	299000000	-1.2	38	256 qam	21
21	307000000	-0.7	39	256 qam	22
22	315000000	-0.4	40	256 qam	23
23	323000000	0	39	256 qam	24

Downstream bonded channels

Channel	Locked Status	RxMER (dB)	Pre RS Errors	Post RS Errors
1	Locked	38.9	19682	15634
2	Locked	38.2	19549	33334
3	Locked	37.6	19740	22380
4	Locked	38.2	3095	4199
5	Locked	38.6	19258	16182
6	Locked	38.9	19497	15263
7	Locked	38.9	19887	17221
8	Locked	38.9	19879	16275
9	Locked	38.9	19422	15958
10	Locked	38.9	19333	17295
11	Locked	38.9	19512	19253
12	Locked	38.9	19338	18685
13	Locked	38.9	18993	18534
14	Locked	39.3	19000	16669
15	Locked	38.9	18998	16728
16	Locked	39.3	19357	18511
17	Locked	38.9	19292	19279
18	Locked	38.6	19372	16556
19	Locked	39.3	19069	15615
20	Locked	38.9	19113	13874
21	Locked	39.3	19043	14204
22	Locked	40.3	19375	15172
23	Locked	39.8	19482	13565

Network Log

Time	Priority	Description
15/06/2021 04:19:31	Warning!	RCS Partial Service;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 04:19:31	critical	No Ranging Response received - T3 time-out;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 04:19:27	Warning!	RCS Partial Service;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 04:19:27	critical	SYNC Timing Synchronization failure - Loss of Sync;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 04:19:22	Warning!	RCS Partial Service;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 04:19:21	critical	No Ranging Response received - T3 time-out;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 04:19:17	critical	SYNC Timing Synchronization failure - Loss of Sync;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 04:19:7	Warning!	RCS Partial Service;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 04:19:7	critical	SYNC Timing Synchronization failure - Loss of Sync;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 04:18:44	Warning!	RCS Partial Service;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;

[RedDragonTaff]

Downstream power looks to have dropped and Upstream power looks to have surged compared to earlier logs.

Not surprised that that errors on the downstream have shot through the roof as the signal is low.  Upstream we are seeing errors on channel ID #2 and #3 (53700000Hz and 46200000Hz respectively).

[lotharmat]

Yup - Upstream definitely needs rectifying!

[Zach_R]

Hi @RedDragonTaff,

Thank you for your posts, we're here to help.

I'm so sorry to hear that you've been having some problems with your broadband connection recently. Would you mind clarifying what diagnostics and troubleshooting you've carried out thus far? This is so we don't repeat anything where possible, but also don't miss anything either.

Thanks,
 

[RedDragonTaff]

Hi Zach,

Not sure your reason for your innocent or novice question, but to humour your question I have created a flowchart of my process.  Beyond that I don't know what formal VM process you guys follow for diagnosing problems on your network to the site, if any.

Anyway this is what I do when your network fails to respond to internet requests and when external devices/platforms can't see the external side of the SH3 served by VM:

Let me know if you still have trouble understanding this.  Tip start at the top and you finish (hopefully) at the green box.

[RedDragonTaff]

Sixth time the router has reset itself.

BQM:

Router Network logs like I mention earlier don't seem to list everything?  Can anyone from VM look at the reasons for reboot beyond what I see in these logs?

15/06/2021 07:50:14	notice	LAN login Success;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 05:15:51	critical	No Ranging Response received - T3 time-out;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;

Counters in the Upstream and Downstream have been reset so hard to know the values at the time of the failure, but the current ones are:

Upstream bonded channels

Channel Frequency (Hz) Power (dBmV) Symbol Rate (ksps) Modulation Channel ID

1	32600000	46.5	5120	32 qam	5
2	53700000	47.8	5120	64 qam	2
3	39400000	47	5120	32 qam	4
4	46200000	47.3	5120	64 qam	3

Upstream bonded channels

Channel Channel Type T1 Timeouts T2 Timeouts T3 Timeouts T4 Timeouts

1	ATDMA	0	0	0	0
2	ATDMA	0	0	0	0
3	ATDMA	0	0	0	0
4	ATDMA	0	0	0	0

Downstream bonded channels

Channel Frequency (Hz) Power (dBmV) SNR (dB) Modulation Channel ID

1	203000000	3.7	40	256 qam	9
2	211000000	3.5	40	256 qam	10
3	219000000	3.4	40	256 qam	11
4	227000000	3.2	40	256 qam	12
5	243000000	3	40	256 qam	14
6	251000000	3	40	256 qam	15
7	259000000	2.7	40	256 qam	16
8	267000000	2.5	40	256 qam	17
9	275000000	1.4	40	256 qam	18
10	283000000	3.2	40	256 qam	19
11	291000000	4	40	256 qam	20
12	299000000	4	40	256 qam	21
13	307000000	4.3	40	256 qam	22
14	315000000	4.4	40	256 qam	23
15	323000000	4.5	40	256 qam	24
16	331000000	4.5	40	256 qam	25
17	371000000	5	40	256 qam	26
18	379000000	5	40	256 qam	27
19	387000000	3.7	40	256 qam	28
20	395000000	2.5	40	256 qam	29
21	403000000	2.9	40	256 qam	30
22	411000000	3.2	40	256 qam	31
23	419000000	3.4	40	256 qam	32
24	235000000	3	40	256 qam	13

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	40.3	4	0
2	Locked	40.3	4	0
3	Locked	40.3	5	0
4	Locked	40.3	6	0
5	Locked	40.3	0	0
6	Locked	40.3	4	0
7	Locked	40.9	13	0
8	Locked	40.3	12	0
9	Locked	40.3	0	0
10	Locked	40.3	6	0
11	Locked	40.3	5	0
12	Locked	40.3	19	0
13	Locked	40.3	7	0
14	Locked	40.3	3	0
15	Locked	40.3	6	0
16	Locked	40.3	0	0
17	Locked	40.9	0	0
18	Locked	40.9	0	0
19	Locked	40.3	5	0
20	Locked	40.9	0	0
21	Locked	40.3	5	0
22	Locked	40.3	5	0
23	Locked	40.9	6	0
24	Locked	40.9	5	0

[RedDragonTaff]

Seventh drop out.

BQM:

Again Router Network log isn't showing any new entries (odd)

Upstream and Downstream counters are reset back to zero by the router:

Upstream bonded channels

Channel Frequency (Hz) Power (dBmV) Symbol Rate (ksps) Modulation Channel ID

1	32600000	46.8	5120	32 qam	5
2	53700000	48	5120	64 qam	2
3	39400000	47.3	5120	32 qam	4
4	46200000	47.5	5120	64 qam	3

Upstream bonded channels

Channel Channel Type T1 Timeouts T2 Timeouts T3 Timeouts T4 Timeouts

1	ATDMA	0	0	0	0
2	ATDMA	0	0	0	0
3	ATDMA	0	0	0	0
4	ATDMA	0	0	0	0

Downstream bonded channels

Channel Frequency (Hz) Power (dBmV) SNR (dB) Modulation Channel ID

1	203000000	2.9	40	256 qam	9
2	211000000	2.7	40	256 qam	10
3	219000000	2.5	40	256 qam	11
4	227000000	2.4	40	256 qam	12
5	235000000	2.4	40	256 qam	13
6	243000000	2.2	40	256 qam	14
7	251000000	2.2	40	256 qam	15
8	259000000	2	40	256 qam	16
9	267000000	1.5	40	256 qam	17
10	275000000	0.5	40	256 qam	18
11	283000000	2.5	40	256 qam	19
12	291000000	3.4	40	256 qam	20
13	299000000	3.2	40	256 qam	21
14	307000000	3.5	40	256 qam	22
15	315000000	3.5	40	256 qam	23
16	323000000	3.7	40	256 qam	24
17	331000000	3.7	40	256 qam	25
18	371000000	4.3	40	256 qam	26
19	379000000	4.4	40	256 qam	27
20	387000000	2.7	40	256 qam	28
21	395000000	1.9	40	256 qam	29
22	403000000	2.2	40	256 qam	30
23	411000000	2.5	40	256 qam	31
24	419000000	2.7	40	256 qam	32

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	40.3	4	0
2	Locked	40.3	5	0
3	Locked	40.9	4	0
4	Locked	40.3	7	0
5	Locked	40.3	4	0
6	Locked	40.9	0	0
7	Locked	40.9	4	0
8	Locked	40.3	4	0
9	Locked	40.3	10	0
10	Locked	40.3	9	0
11	Locked	40.3	4	0
12	Locked	40.3	5	0
13	Locked	40.3	3	0
14	Locked	40.3	4	0
15	Locked	40.3	16	0
16	Locked	40.3	4	0
17	Locked	40.3	0	0
18	Locked	40.3	0	0
19	Locked	40.3	0	0
20	Locked	40.3	0	0
21	Locked	40.3	0	0
22	Locked	40.3	5	0
23	Locked	40.9	5	0
24	Locked	40.3	6	0

Nothing much to see but the BQM.  The Router did reboot as we saw the lights go RED and then GREEN and after a few minutes the white bar.

[RedDragonTaff]

Any comment @Zach_R after my response to your question?  You seemed interested but quickly dumped this from your radar!  I don't think that's very engaging or demonstating your effective customer support skills.

[Martin_N]

Thank you for all the information provided RedDragonTaff.

I have taken a look on our side and I can see you have engineer booked in regarding this. 

At this point we would advise waiting for the visit so the engineer can investigate this further. 

^Martin

[RedDragonTaff]

10th reset today.

BQM:

New entries in the router network log since the last post:

Network Log

Time	Priority	Description
15/06/2021 17:48:16	Warning!	Lost MDD Timeout;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 17:48:15	Warning!	RCS Partial Service;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 17:48:15	critical	SYNC Timing Synchronization failure - Loss of Sync;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 17:48:7	Warning!	RCS Partial Service;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 17:48:7	critical	SYNC Timing Synchronization failure - Loss of Sync;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;
15/06/2021 17:48:7	Warning!	RCS Partial Service;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;

Upstream bonded channels

Channel	Frequency (Hz)	Power (dBmV)	Symbol Rate (ksps)	Modulation	Channel ID
1	32600000	47	5120	64 qam	5
2	53700000	48	5120	64 qam	2
3	39400000	47.3	5120	32 qam	4
4	46200000	47.5	5120	64 qam	3

Upstream bonded channels

Channel	Channel Type	T1 Timeouts	T2 Timeouts	T3 Timeouts	T4 Timeouts
1	ATDMA	0	0	0	0
2	ATDMA	0	0	0	0
3	ATDMA	0	0	0	0
4	ATDMA	0	0	0	0

Downstream bonded channels

Channel	Frequency (Hz)	Power (dBmV)	SNR (dB)	Modulation	Channel ID
1	203000000	2.7	40	256 qam	9
2	211000000	2.5	40	256 qam	10
3	219000000	2.4	40	256 qam	11
4	227000000	2.2	40	256 qam	12
5	235000000	2.2	40	256 qam	13
6	243000000	2	40	256 qam	14
7	251000000	2.2	40	256 qam	15
8	259000000	1.7	40	256 qam	16
9	267000000	1.4	40	256 qam	17
10	275000000	0.5	38	256 qam	18
11	283000000	2.5	40	256 qam	19
12	291000000	3.2	40	256 qam	20
13	299000000	3.2	40	256 qam	21
14	307000000	3.2	40	256 qam	22
15	315000000	3.4	40	256 qam	23
16	323000000	3.5	40	256 qam	24
17	331000000	3.5	40	256 qam	25
18	371000000	4	40	256 qam	26
19	379000000	4.3	40	256 qam	27
20	387000000	2.5	40	256 qam	28
21	395000000	1.7	40	256 qam	29
22	403000000	2	40	256 qam	30
23	411000000	2.4	40	256 qam	31
24	419000000	2.5	40	256 qam	32

Downstream bonded channels

Channel	Locked Status	RxMER (dB)	Pre RS Errors	Post RS Errors
1	Locked	40.3	0	0
2	Locked	40.3	4	0
3	Locked	40.3	4	0
4	Locked	40.3	5	0
5	Locked	40.3	5	0
6	Locked	40.3	5	0
7	Locked	40.3	0	0
8	Locked	40.9	5	0
9	Locked	40.3	7	0
10	Locked	40.9	11	0
11	Locked	38.9	0	0
12	Locked	40.3	6	0
13	Locked	40.3	0	0
14	Locked	40.3	5	0
15	Locked	40.3	17	0
16	Locked	40.3	4	0
17	Locked	40.3	5	0
18	Locked	40.3	0	0
19	Locked	40.9	0	0
20	Locked	40.9	0	0
21	Locked	40.3	0	0
22	Locked	40.3	0	0
23	Locked	40.3	6	0
24	Locked	40.3	5	0