Packet loss, high latency since 15th Sept.

[kej999]

Hi Folks,

  I've just called the support team but have been asked to wait 24 hours due to there being a potential transitory error.  I thought I'd better just check with knowledgable people first to see if there's something to try. More to the point, does it look like I have a separate problem to any ongoing issues? Any advice would be appreciated.

 Since some planned maintenance in my area (15th Sept), we've been getting notably intermittant connections. The majority of the time, speed is good (200+ down, 20+ up, M200 service). Recently though, things have becoming intermittent and will grind to a halt for a number of seconds/minutes, multiple times an hour.

My original setup;

• SH3 (modem mode, 1st floor feed) feeding TP-Link Deco M9+ mesh by ethernet, my computer on ethernet from a slave M9+.

After recabling changes (retightened connections, swapped cables,swapped cable feed etc etc) I set it back up to something more standard for the phone support team prior to reporting it;

• SH3 (router mode, wifi off, gnd floor feed) feeding TP-Link Deco M9+ mesh by ethernet, my computer on ethernet from the SH3.

  VM's automated tests say it's fine (I got one referral to book an engineer last week but it's not been offered since). Today's phone call gave no errors when tested by the support team member. However, I'm still getting dropped packets and spiky ping response times though.  Large quantities of post-RS errors are turning up with "RCS Partial Service" and "Lost MDD Timeout" errors in log files. The SH3 internet status now says, "Partial Service (DS only)".

BQM;

• Normal (prior to maintenance) 
• Problems started? (day of maintenance)
• Problems continue (kids noticed) 
• Good day (relatively speaking) 
• Moved everything downstairs (clear for a bit)
• Still wierd 
• Laggy day
• Back to router mode (new IP issued!)
• Fixed? (nope)

SH3:

Downstream bonded channels

Channel Frequency (Hz) Power (dBmV) SNR (dB) Modulation Channel ID

1	138750000	4.3	37	256 qam	1
2	146750000	4	37	256 qam	2
3	154750000	4.1	37	256 qam	3
4	162750000	4.1	36	256 qam	4
5	170750000	4.5	37	256 qam	5
6	178750000	4	37	256 qam	6
7	186750000	4	37	256 qam	7
8	194750000	3.5	37	256 qam	8
9	202750000	3	37	256 qam	9
10	226750000	2.5	30	256 qam	12
11	234750000	1.7	37	256 qam	13
12	242750000	1.7	37	256 qam	14
13	250750000	2.2	37	256 qam	15
14	258750000	2.7	37	256 qam	16
15	266750000	2.5	37	256 qam	17
16	274750000	2.9	37	256 qam	18
17	282750000	3	38	256 qam	19
18	290750000	4.3	38	256 qam	20
19	298750000	5.5	38	256 qam	21
20	306750000	6.1	38	256 qam	22
21	314750000	6.5	40	256 qam	23
22	322750000	6.5	40	256 qam	24

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	37.3	12288	113
2	Locked	37.6	8998	85
3	Locked	37.3	11080	93600
4	Locked	36.6	10152	86
5	Locked	37.6	138	88
6	Locked	37.6	126	90
7	Locked	37.6	173	85
8	Locked	37.3	432	239
9	Locked	37.3	160	60
10	Locked	30.4	124048954	297708698
11	Locked	37.3	132	53
12	Locked	37.6	167	59
13	Locked	37.6	147	58
14	Locked	37.6	151	55
15	Locked	37.6	107	53
16	Locked	37.6	86	55
17	Locked	38.6	99	63
18	Locked	38.9	55	57
19	Locked	38.6	39	50
20	Locked	38.9	48	62
21	Locked	40.3	45	52
22	Locked	40.3	47	36

Channels 10-12 seem to be the ones consistently building up errors in other logs I've taken. Any suggestions on things to try?

Regards,

Keith

[jbrennand]

Down channel 10 looks borked. Can you start from a clean sheet by doing the below. Also post the BQM link to "share live graph" so its the actual most recent one.
_____________________________________________________

Switch the Hub off and unplug it from the mains supply for five minutes. Whilst it's off, do a quick check that all of your coax cable connections are in nice and "finger" tight - at the Hub and wall box and also at any connectors etc. Ensure there are no “unterminated cable loose ends. Disconnect all the connections and reconnect to be sure. Also check that the internal wiring is ok with no kinking or chaffing, check that all looks good with the outside cabling and wall box (no “staples, etc.,) piercing the cables.. Then switch the Hub back on and leave ~5 minutes

When all done, check back in the settings and ensure that the RS error counts and T3 errors have all reset to 0. Also that channel 10 SNR has reverted to >34dB. Then check every hour or so to see if the RS errors start reappearing - they shouldn't.

If they do (particularly the postRS ones) you have a problem (noise) that only a Tech visit will sort.

[Travis_M]

Hi @kej999

Thanks for posting on our community forum!

Could you follow the above advice from jbrennand please?

Once this has been followed I can check on my end and if needed we'll have to get a tech booked to review the noise issues if so.

Regards

[kej999]

Hi jbrennand,

Thanks for the help. It is much appreciated!

I'm sorry I didn't respond promptly. Today ended up as a write-off (brother-in-law ill) and I haven't even phoned VM yet. Strangely enough though, I had to do 90% of that yesterday because the SH3 had a bit of a wobbly. I'll do another re-cables/reset in a minute. I'll check the external cabling tomorrow am.

Regards,

Keith

.

[kej999]

Right okay. Additional stuff.

Here's the BQM graphs for the last two days and a live one for today onwards.

Tuesday 5/10/2021

Wednesday 6/10/2021

Live 

On the Wednesday graph you can see the wobble it had at 12:00am-2:00am. There's an initial red spike where the SH3 rebooted itself. It does that every night so I'm presuming its preventative maintenance. This time it didn't recover normally. Rather than the usual few disconnects you get as the ethernet ports get reconfigured, my PC just carried on getting disconnected. Checking that, you could see it attempting to autonegotiate a port speed, accepting 10Mb/s and then giving up and disconnecting again. I rebooted my PC but that didn't help so I powered the SH3 down and gave it a good reset (holding reset button for 50+ second etc). That seemed to do something but the stats still showed problems hours later

 ...Just *before* I reset everything again for the clean slate tonight I grabbed stats...Not only was 10 borked but channel 11 and 12 had joined the party 🙂

Downstream bonded channels
Channel	Locked Status	RxMER (dB)	Pre RS Errors	Post RS Errors
... etc etc
10	Locked	32.9	3299720	2836541
11	Locked	32.6	2256645	79327
12	Locked	33.8	220409	33334
... etc etc

Currently it's looking okay-ish but it's looked okay many times before. Here's the fresh stats and the modem says it's online

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	36.6	20	0
2	Locked	37.3	15	0
3	Locked	37.3	7	0
4	Locked	37.3	7	0
5	Locked	37.6	20	0
6	Locked	37.3	20	0
7	Locked	37.3	4	202
8	Locked	37.6	19	0
9	Locked	37.6	4	0
10	Locked	35.5	24	0
11	Locked	35.7	17	0
12	Locked	35.5	25	0
13	Locked	37.3	0	0
14	Locked	37.3	6	0
15	Locked	37.6	2	0
16	Locked	37.6	7	0
17	Locked	37.6	8	0
18	Locked	37.6	5	0
19	Locked	37.6	1	0
20	Locked	38.9	0	0
21	Locked	38.6	5	0
22	Locked	38.9	5	0
23	Locked	38.6	5	0
24	Locked	38.9	18	0

Time for sleep. I'm not holding my breath though 😞

Regards,

Keith

[kej999]

Thanks Travis_M. Much obliged 🙂

I'll report back after a snooze.

Regards,

Keith

[kej999]

A quick update. I still haven't phoned as there's nothing really there to report (I'm not sure if I should put smiley or frownie for that... ha ha!).

The cables outside look okay-ish, there no additional kinks on anything exposed (at least). The majority of cabling got concreted in many months ago (after being run over and fixed multiple times - another story!). The termination box is tatty but intact (to be expected after 10-15 years).

All-in-all there doesn't appear to be anything obviously wrong. The stats are hanging in there. Channels 10-12 have all dropped in power since this morning but there's only a few 10,000's of pre-rs errors so far.

Downstream bonded channels

Channel Frequency (Hz) Power (dBmV) SNR (dB) Modulation Channel ID

1	138750000	2.5	36	256 qam	1
2	146750000	2.2	36	256 qam	2
3	154750000	2.2	36	256 qam	3
4	162750000	2.2	36	256 qam	4
5	170750000	2.5	37	256 qam	5
6	178750000	2.2	37	256 qam	6
7	186750000	1.9	36	256 qam	7
8	194750000	1.5	37	256 qam	8
9	202750000	1.2	37	256 qam	9
10	210750000	0.2	34	256 qam	10
11	218750000	0	34	256 qam	11
12	226750000	0	34	256 qam	12
13	234750000	-0.5	36	256 qam	13
14	242750000	0	37	256 qam	14
15	250750000	-0.2	37	256 qam	15
16	258750000	0.4	37	256 qam	16
17	266750000	0.5	37	256 qam	17
18	274750000	0.5	37	256 qam	18
19	282750000	0.7	37	256 qam	19
20	290750000	2.2	38	256 qam	20
21	298750000	3	38	256 qam	21
22	306750000	3.5	38	256 qam	22
23	314750000	3.7	38	256 qam	23
24	322750000	3.5	38	256 qam	24

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	36.3	135	0
2	Locked	36.6	80	0
3	Locked	36.6	28	0
4	Locked	36.6	28	0
5	Locked	37.3	26	0
6	Locked	37.3	24	0
7	Locked	36.6	14	202
8	Locked	37.3	22	0
9	Locked	37.3	23	0
10	Locked	34.4	16959	0
11	Locked	34.4	4718	0
12	Locked	34.3	5537	0
13	Locked	36.6	3	0
14	Locked	37.6	7	0
15	Locked	37.3	5	0
16	Locked	37.3	27	0
17	Locked	37.6	9	0
18	Locked	37.6	8	0
19	Locked	37.6	8	0
20	Locked	38.6	14	0
21	Locked	38.6	8	0
22	Locked	38.9	10	0
23	Locked	38.9	26	0
24	Locked	38.6	23	0

I guess I'd better hang on for a while and see if something happens...

Regards,

Keith

[jbrennand]

BQM still looks poor - RS errors piling up is not right - @Travis_M  needs to take a look.

[kej999]

Hi John,

 Thank you for your time. Yes,it was looking good for a while but just started failing at a slow rate instead :-/. I've just grabbed stats again and...

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	36.6	693	634
2	Locked	36.3	321	557
3	Locked	36.6	220	270
4	Locked	36.6	157	354
5	Locked	37.6	117	326
6	Locked	36.6	143	633
7	Locked	36.6	182	560
8	Locked	36.3	99	231
9	Locked	36.6	94	190
10	Locked	33	13765674	23297418
11	Locked	32.9	20292473	32711423
12	Locked	33.9	242643	42713
13	Locked	36.3	117	195
14	Locked	37.3	147	215
15	Locked	36.6	140	231
16	Locked	37.3	137	233
17	Locked	37.3	102	199
18	Locked	37.3	113	220
19	Locked	37.6	135	196
20	Locked	38.6	88	171
21	Locked	37.6	109	236
22	Locked	38.9	77	145
23	Locked	38.6	90	144
24	Locked	38.9	95	112

Upstream bonded channels

Channel Channel Type T1 Timeouts T2 Timeouts T3 Timeouts T4 Timeouts

1	ATDMA	0	0	0	0
2	ATDMA	0	0	1	0
3	ATDMA	0	0	0	0
4	ATDMA	0	0	0	0

@Travis_M - The automated testing has noticed "something" wrong this time but it's suggesting a reboot which will just make it look good for a while. If you could book me an engineer I'd really appreciate it! I don't think I'd get very far via the phone.

Thanks again for the help both of you! I'm sorry this was an awkward one.

Regards,

Keith

[jbrennand]

Thats dreadful. I will flag for someone from VM to come here and comment