CF23 Internet stability and packet loss

[RedDragonTaff]

Hi,

I am at the end of my tether.  Since the 14 Sept 20 the VM network has really gone bad.  Even today this is still ongoing.  All other levels of communication Facebook, Customer Support 150/151 has fallen on deaf ears.

So tonight here's what I see on the Hub:

Network Log:
27/09/2020 21:35:13 GMT 27/09/2020 21:35:13 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:34:59 GMT 27/09/2020 21:34:59 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:34:37 GMT 27/09/2020 21:34:37 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:34:19 GMT 27/09/2020 21:34:19 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:34:00 GMT 27/09/2020 21:34:00 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:33:39 GMT 27/09/2020 21:33:39 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:33:20 GMT 27/09/2020 21:33:20 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:33:00 GMT 27/09/2020 21:33:00 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:32:40 GMT 27/09/2020 21:32:40 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:32:20 GMT 27/09/2020 21:32:20 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:32:01 GMT 27/09/2020 21:32:01 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:31:44 GMT 27/09/2020 21:31:44 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:31:21 GMT 27/09/2020 21:31:21 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:31:13 GMT 27/09/2020 21:31:13 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:31:03 GMT 27/09/2020 21:31:03 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:30:27 GMT 27/09/2020 21:30:27 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:30:18 GMT 27/09/2020 21:30:18 GMT Error (4) 68010302 DHCP WAN IP - removed
27/09/2020 21:30:04 GMT 27/09/2020 21:30:04 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:29:45 GMT 27/09/2020 21:29:45 GMT Warning (5) 84020200 Lost MDD Timeout
27/09/2020 21:29:38 GMT 27/09/2020 21:29:38 GMT Warning (5) 84020200 Lost MDD Timeout

Downstream:
DS-1 DS-2 DS-3 DS-4 DS-5 DS-6 DS-7 DS-8
Frequency (Hz) 139000000 147000000 155000000 163000000 N/A 179000000 187000000 195000000
Lock Status(QAM Lock/FEC Sync/MPEG Lock) Locked Locked Locked Locked Unlocked Locked Locked Locked
Channel ID 1 2 3 4 N/A 6 7 8
Modulation 256QAM 256QAM 256QAM 256QAM Unknown 256QAM 256QAM 256QAM
Symbol Rate (Msym/sec) 6.952000 6.952000 6.952000 6.952000 N/A 6.952000 6.952000 6.952000
Interleave Depth I=12
J=17 I=12
J=17 I=12
J=17 I=12
J=17 N/A I=12
J=17 I=12
J=17 I=12
J=17
Power Level (dBmV) -2.30 -2.44 -2.64 -2.56 N/A -2.69 -2.95 -2.87
RxMER (dB) 36.84 37.09 36.84 36.84 N/A 38.26 38.26 36.39
Pre RS Errors 327 319 297 276 N/A 295 302 966
Post RS Errors 289 287 297 275 N/A 295 302 966

Upstream:
US-1 US-2 US-3 US-4
Channel Type 2.0 2.0 2.0 2.0
Channel ID 1 4 2 3
Frequency (Hz) 60300000 39400000 53700000 46200000
Ranging Status Success Success Success Success
Modulation 64QAM 64QAM 64QAM 64QAM
Symbol Rate (Sym/sec) 5120000 5120000 5120000 5120000
Mini-Slot Size 2 2 2 2
Power Level (dBmV) 46.50 45.50 46.00 45.75
T1 Timeouts 0 0 0 0
T2 Timeouts 0 0 0 0
T3 Timeouts 1 0 0 0
T4 Timeouts 0 0 0 0

Upstream Burst:
Req
(1) Init Maint
(3) Per Maint
(4) Adv Short
(9) Adv Long
(10) Adv UGS
(11)
Modulation Type 16QAM QPSK 16QAM 64QAM 64QAM 16QAM
Differential Encoding OFF OFF OFF OFF OFF OFF
Preamble Length 56 640 384 104 104 104
Preamble Value Offset 652 0 0 716 716 716
FEC Error Correction (T) 0 5 5 10 16 5
FEC Codeword Information Bytes (K) 16 34 34 81 223 86
Maximum Burst Size 0 0 0 3 255 255
Guard Time Size 8 48 48 8 8 8
Last Codeword Length Fixed Fixed Fixed Shortened Shortened Shortened
Scrambler On/Off ON ON ON ON ON ON

Attached is the BQM graphs daily since 14th Sept.  Can someone initiate an investigation please?

Thanks

AL

[MOD EDIT: Personal and private information has been removed from this post. Please do not post personal or private information in your public posts. Please review the Forum Guidelines]

[RedDragonTaff]

Can I suggest at least answering these technical questions?

1) Why do SH3 or may particular SH3 seem to have lack of events on the Network log?

2) For the Downstream power levels, why would they seemingly dropped from 4-5dBmV to 0-3dBmV a day after the engineers visit?

AL

[Martin_N]

I'm unsure why they would drop like you have advised and why your hub is lacking in regards to network logs. 

When did you last reboot your hub? Did you make any hub setting amendments after the visit?

^Martin

[RedDragonTaff]

The last physical reboot was on Wednesday of the engineers visit.  It has not been rebooted (powered off/on), nor reset to factory defaults since the engineer did this.  It has by its own accord reset itself many times as per the BQM logs since which you also have a copy of.

No settings were changed on the Router.  It still has the default Wifi password (I don't use the wifi), and it still has the same router admin password.  I did not change that either.

I just log into it to see the stupid logs!

[RedDragonTaff]

I can guess your default answer to ask for it to be reset.  I'll do that first thing at 00:00.  Unless you recommend a factory reset.  The engineer recommended resetting the router once every week but we haven't reached that point as yet @Martin_N

[RedDragonTaff]

So I powered off the router on 22 June 2021 @ 1:40am approx for 1 minute and then powered back on.

BQM for Day 5 post engineers visit:

Logs will be uninteresting since they are post the above manual reboot

No new entries since the 19th June regardless of BQM actual events

Network Log

Time Priority Description

19/06/2021 09:02:13

critical

No Ranging Response received - T3 time-out;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;

[RedDragonTaff]

Upstream bonded channels

Channel Frequency (Hz) Power (dBmV) Symbol Rate (ksps) Modulation Channel ID

1	32600000	49	5120	64 qam	5
2	53700000	50.3	5120	64 qam	2
3	39400000	49.3	5120	64 qam	4
4	46200000	49.5	5120	64 qam	3
5	60300000	50.5	5120	64 qam	1

Upstream bonded channels

Channel Channel Type T1 Timeouts T2 Timeouts T3 Timeouts T4 Timeouts

1	ATDMA	0	0	0	0
2	ATDMA	0	0	0	0
3	ATDMA	0	0	0	0
4	ATDMA	0	0	0	0
5	ATDMA	0	0	0	0

Downstream bonded channels

Channel Frequency (Hz) Power (dBmV) SNR (dB) Modulation Channel ID

1	203000000	1.4	40	256 qam	9
2	139000000	1.2	40	256 qam	1
3	147000000	1.2	40	256 qam	2
4	155000000	1.4	40	256 qam	3
5	163000000	1.5	40	256 qam	4
6	171000000	1.7	40	256 qam	5
7	179000000	1.5	40	256 qam	6
8	187000000	1.5	40	256 qam	7
9	195000000	1.5	40	256 qam	8
10	211000000	1.2	40	256 qam	10
11	219000000	1.2	40	256 qam	11
12	227000000	1	40	256 qam	12
13	235000000	1	40	256 qam	13
14	243000000	1	40	256 qam	14
15	251000000	1	40	256 qam	15
16	259000000	1	40	256 qam	16
17	267000000	0.5	40	256 qam	17
18	275000000	-0.5	40	256 qam	18
19	283000000	1.5	40	256 qam	19
20	291000000	2.2	40	256 qam	20
21	299000000	2.4	40	256 qam	21
22	307000000	2.5	40	256 qam	22
23	315000000	2.7	40	256 qam	23
24	323000000	2.9	40	256 qam	24

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	40.3	12	0
2	Locked	40.3	8	0
3	Locked	40.9	7	0
4	Locked	40.3	9	0
5	Locked	40.3	8	0
6	Locked	40.3	8	0
7	Locked	40.3	7	0
8	Locked	40.3	6	0
9	Locked	40.9	11	0
10	Locked	40.3	6	0
11	Locked	40.3	15	0
12	Locked	40.3	6	0
13	Locked	40.3	22	0
14	Locked	40.3	7	0
15	Locked	40.9	14	0
16	Locked	40.3	8	0
17	Locked	40.9	5	0
18	Locked	40.3	2	0
19	Locked	40.9	2	0
20	Locked	40.3	9	0
21	Locked	40.3	7	0
22	Locked	40.9	6	0
23	Locked	40.3	6	0
24	Locked	40.9	8	0

[lotharmat]

5 Upstream channels????

That's just plain greedy! 😉

[RedDragonTaff]

Not sure what happened but no longer showing 5 upstream, back to 4.  Not sure why the router reported that at the time.  BQM showing latency issues, and logs show T3 errors since the last router power cycle:

Network Log

Time Priority Description

22/06/2021 01:37:0

critical

No Ranging Response received - T3 time-out;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.0;

Upstream bonded channels

Channel Frequency (Hz) Power (dBmV) Symbol Rate (ksps) Modulation Channel ID

1	53700000	50.3	5120	64 qam	2
2	39400000	49.3	5120	64 qam	4
3	46200000	50	5120	16 qam	3
4	60300000	50.5	5120	64 qam	1

Upstream bonded channels

Channel Channel Type T1 Timeouts T2 Timeouts T3 Timeouts T4 Timeouts

1	ATDMA	0	0	3	0
2	ATDMA	0	0	1	0
3	ATDMA	0	0	0	0
4	ATDMA	0	0	2	0

Downstream bonded channels

Channel Frequency (Hz) Power (dBmV) SNR (dB) Modulation Channel ID

1	203000000	1.2	40	256 qam	9
2	139000000	1.2	40	256 qam	1
3	147000000	1.2	40	256 qam	2
4	155000000	1.2	40	256 qam	3
5	163000000	1.4	40	256 qam	4
6	171000000	1.5	40	256 qam	5
7	179000000	1.5	40	256 qam	6
8	187000000	1.4	40	256 qam	7
9	195000000	1.2	40	256 qam	8
10	211000000	1	40	256 qam	10
11	219000000	1	40	256 qam	11
12	227000000	0.9	40	256 qam	12
13	235000000	0.9	40	256 qam	13
14	243000000	0.9	40	256 qam	14
15	251000000	1	40	256 qam	15
16	259000000	0.9	40	256 qam	16
17	267000000	0.5	40	256 qam	17
18	275000000	0	40	256 qam	18
19	283000000	1.2	40	256 qam	19
20	291000000	1.9	40	256 qam	20
21	299000000	2.2	40	256 qam	21
22	307000000	2.5	40	256 qam	22
23	315000000	2.5	40	256 qam	23
24	323000000	2.7	40	256 qam	24

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	40.3	6	0
2	Locked	40.3	5	0
3	Locked	40.3	6	0
4	Locked	40.3	5	0
5	Locked	40.9	4	0
6	Locked	40.9	5	0
7	Locked	40.9	9	0
8	Locked	40.3	4	0
9	Locked	40.3	5	0
10	Locked	40.9	7	0
11	Locked	40.3	9	0
12	Locked	40.3	6	0
13	Locked	40.3	4	0
14	Locked	40.9	6	0
15	Locked	40.3	6	0
16	Locked	40.9	4	0
17	Locked	40.3	5	0
18	Locked	40.3	3	0
19	Locked	40.3	6	0
20	Locked	40.3	6	0
21	Locked	40.3	25	0
22	Locked	40.3	4	0
23	Locked	40.3	5	0
24	Locked	40.3	8	0

[RedDragonTaff]

So far since the router power cycle:

[Andrew-G · Accepted solution]

It would seem evident to me that (from this and a small number of intractable "field" problems) that VM do not have any effective fault finding process, are overly reliant upon some very crude diagnostics, and have little or no access to the tools necessary to maintain and troubleshoot on an RF network.  Everything is divided into silos (eg between customer services, the field team, the network team), each sub-element is cheese pared down with the lowest cost labour available for each step, and there are procedures in place that are DESIGNED to stop employees taking ownership and sorting out problems that don't fit into the basic scripts and diagnostics.  Usually this does work for simple problems, but anything out of the ordinary and it fails spectacularly.

So @RedDragonTaff given what's gone on, is it time to cut your losses and leave?  On the one hand it looks to be that this latest bout of problems has only been going on for two weeks(?), on the other it has been a depressing read where the company have flunked each of multiple opportunities to diagnose and fix what ought to be an easily found and resolved problem.   

If you're in a fixed term contract then you'll need to formally invoke VM's complaint policy and in the complaint give them 30 days to fix the problem or release you from contract without penalty.   This might even be enough to get the field team to allocate the resources to find and fix, but looking at other comments in the forum, VM's complaints process appears to be an exercise in fobbing off complaints by the same sort of poorly trained offshore staff that do first line "support".  But you'll need to go through that stage if it becomes necessary to escalate the complaint to arbitration by CISAS (and even if they do fix this, I'd go down that route to seek compensation).