T3 timeouts and other fun

[Icantcrabhere]

Hi there,
noticed over the past few weeks that i'm getting random bursts of T3 timeouts,
also yesterday in the morning, I lost a whole upstream channel - rebooting restored this, however it still isn't a good sign.

I've checked both the online fault checker and the phone version:

Yesterday the online fault checker showed no issues in my area. - then I clicked the "test your hub" option.
It showed "intermittent signal in your area" - but today it's back to normal, showing just the standard "We couldn't find an issue, let's try resetting the hub!" style message.
However this is far from the case, clearly.

I'll share my fresh stats from the restart yesterday, these are from after 12PM yesterday afternoon.

I'll have to split the post due to character limits.

Firstly, BQM live graph:(Nothing much to see here, just for future reference)

https://www.thinkbroadband.com/broadband/monitoring/quality/share/1700d27c199ce3ea214d66ce063ff9f395507b3e

Downstream bonded channels

Channel Frequency (Hz) Power (dBmV) SNR (dB) Modulation Channel ID

1	539000000	0.5	38	256 qam	17
2	547000000	0.2	38	256 qam	18
3	555000000	0	38	256 qam	19
4	563000000	-0.2	38	256 qam	20
5	571000000	-0.5	38	256 qam	21
6	579000000	-0.4	38	256 qam	22
7	587000000	0	38	256 qam	23
8	595000000	0.5	38	256 qam	24
9	603000000	0.4	38	256 qam	25
10	611000000	0.2	38	256 qam	26
11	619000000	-0.4	38	256 qam	27
12	627000000	-0.5	38	256 qam	28
13	635000000	-0.5	37	256 qam	29
14	643000000	-0.2	38	256 qam	30
15	651000000	0.2	38	256 qam	31
16	659000000	0.5	38	256 qam	32
17	667000000	0.5	38	256 qam	33
18	675000000	0.2	38	256 qam	34
19	683000000	-0.2	38	256 qam	35
20	691000000	-0.2	38	256 qam	36
21	699000000	-0.4	37	256 qam	37
22	707000000	0	37	256 qam	38
23	715000000	-0.4	37	256 qam	39
24	723000000	-0.5	38	256 qam	40

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	38.6	30	0
2	Locked	38.6	38	0
3	Locked	38.9	44	0
4	Locked	38.9	43	0
5	Locked	38.6	68	0
6	Locked	38.9	47	0
7	Locked	38.6	37	0
8	Locked	38.6	22	0
9	Locked	38.9	18	0
10	Locked	38.6	61	0
11	Locked	38.6	90	0
12	Locked	38.6	125	0
13	Locked	37.6	121	0
14	Locked	38.6	78	0
15	Locked	38.9	58	0
16	Locked	38.6	61	0
17	Locked	38.6	48	0
18	Locked	38.6	78	0
19	Locked	38.6	126	0
20	Locked	38.6	136	0
21	Locked	37.6	110	0
22	Locked	37.6	78	0
23	Locked	37.6	108	0
24	Locked	38.6	124	0

Upstream bonded channels

Channel Frequency (Hz) Power (dBmV) Symbol Rate (ksps) Modulation Channel ID

1	49600258	45	5120	64 qam	1
2	36600000	44.3	5120	64 qam	3
3	23600000	43.8	5120	64 qam	5
4	30100258	44	5120	64 qam	4
5	43100000	44.5	5120	64 qam	2

Upstream bonded channels

Channel Channel Type T1 Timeouts T2 Timeouts T3 Timeouts T4 Timeouts

1	ATDMA	0	0	1	0
2	ATDMA	0	0	1	0
3	ATDMA	0	0	0	0
4	ATDMA	0	0	0	0
5	ATDMA	0	0	2	0

[Zach_R]

Hi @Icantcrabhere, 

Thank you for expanding. We're not detecting any fault on our systems that would explain this.

Can you clarify what issues with the service you're experiencing when this happens? Are you losing your connection completely?

Thanks,

[Icantcrabhere]

Hi there,
Firstly, thanks for all your replies! (seph, IPF, forum staff)
Sorry for my delayed replies to some of you,
You guys and girls are wonderful 🙂


No, the connection doesn't drop completely.

a single t3 isn't an issue,
it's when they come in a big burst and cause a temporary "Lag spike" that would be more of the issue.
Weirdly, there's no evidence of packet loss on the ThinkBroadband graph when this happens, either.

Its been so random the last week or so, that it isn't occuring every day.

I was more so worried about losing a full upstream channel the other day(As shown by the errors in my original post)
Although that hasn't happened again since, but I figured it was worth reporting as it's not exactly normal or ideal.

Maybe I'm overthinking it,
I'm just no stranger to SNR/Noise issues, so i thought It'd be worth posting about before it became(becomes) worse.

Also seems that my PRE-RS error count has reset itself again without any input from myself:

Downstream bonded channels

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors

1	Locked	38.9	162	0
2	Locked	38.6	5	0
3	Locked	38.9	6	0
4	Locked	38.9	6	0
5	Locked	38.9	6	0
6	Locked	38.9	5	0
7	Locked	38.9	8	0
8	Locked	38.6	6	0
9	Locked	38.9	4	0
10	Locked	38.6	7	0
11	Locked	38.9	9	0
12	Locked	38.6	9	0
13	Locked	38.6	13	0
14	Locked	38.9	13	0
15	Locked	38.9	4	0
16	Locked	38.6	6	0
17	Locked	38.9	6	0
18	Locked	38.6	7	0
19	Locked	38.9	13	0
20	Locked	38.9	11	0
21	Locked	38.6	10	0
22	Locked	38.6	12	0
23	Locked	38.6	8	0
24	Locked	38.6	8	0

[unisoft]

---

@Icantcrabhere wrote:

Hi there, cheers for the reply - i appreciate it! 🙂
the RS coun't shouldn't have reset, i've had no full disconnects - which is why I was confused by this

I assumed it'd be noise of some sort, hadn't seen any upstream channels fall below 64QAM so I was truly unsure.

---

It could manifest itself as short strong bursts of noise. They can cause T3 timeouts without QAM degrading as they don't last long; this could be en route to cab or at the cab itself. Can also get T3 timeouts if optical/amplifier is faulty at the cab. Years ago, I had this specific reason.

[Sephiroth]

Just to make sure that the T3 timeout event is understood. It occurs if an expected keep-alive message is not received at the CMTS (the VM end).  If there are 16x of these on the trot, the hub will reset (a T4 event).

[Icantcrabhere]

@Sephiroth,
Yeah, I understand that 🙂
more so just worried about the bursts they've been coming in - as seeing a few every so often is fine, but the bursts that had been happening of multiple T3's on multiple channels at once was quite concerning.

Thank you everyone for your help and input thus far, I really do appreciate it! 🙂

[Sephiroth]

If you've had an instance of losing an upstream channel, then the problem's in the Network.   Quite how you get the VM bods on this forum to get that dealt with is quite a challenge because the instants that they look at your situation, everything looks normal.    Of course the data's all there in the CMTS logs, to which, AFAIK, the Forum Team don't have access.

[Icantcrabhere]

And here comes the Modulation drops

Channel Frequency (Hz) Power (dBmV) Symbol Rate (ksps) Modulation Channel ID

1	49600268	44.8	5120	64 qam	1
2	36600000	44.3	5120	32 qam	3
3	23600000	43.8	5120	64 qam	5
4	30100000	44	5120	64 qam	4
5	43100000	45.5	5120	16 qam	2

Upstream bonded channels

Channel Channel Type T1 Timeouts T2 Timeouts T3 Timeouts T4 Timeouts

1	ATDMA	0	0	9	0
2	ATDMA	0	0	2	0
3	ATDMA	0	0	1	0
4	ATDMA	0	0	0	0
5	ATDMA	0	0	13	0

Well, I was prepared for this!

[Ilyas_Y]

Thanks for the reply @Icantcrabhere 🧾

I've had a look and I can't see any SNR issues on our end nor can I see any faults in the area.
The hub is showing as online on our end and everything is showing as fine.

Are you able to setup a BQM graph here and show us the results?
This will help us see where the dropouts are.

Kind regards,
Ilyas.

[Icantcrabhere]

Hi there,
it seems super intermittent, currently.
I'll continue monitoring over the next few days,

https://www.thinkbroadband.com/broadband/monitoring/quality/share/1700d27c199ce3ea214d66ce063ff9f395507b3e

Here's the live BQM.
You wont see much here as there's not any full dropouts.

[Sephiroth]

Here's the potential problem with the Forum Team reply.

If you are still seeing the reduced upstream modulation (as in 16QAM, 32QAM), then your upstream is experiencing noise on those channels.  The CMTS at the VM end adjusts the modulation downwards so that the upstream data received can be used.  If 16QAM doesn't cut it, a modem rest will occur.

The Forum Team will have been briefed that this is normal behaviour (which, in DOCSIS terms it is).  Never mind the disrupted upstream, poor performance and so on.

So, until Networks resolve the noise issue (and who's gonna invoke that effort), you're stuffed, imo.