Network flapping post VM “area downtime”

[Kaz911]

Yesterday - our area (Brighton BN2) was impacted by a VM repair. It lasted from 12 to 5pm - and during that time - our internet flapped in/out every 2 minutes. 

At 5pm - internet stabilised and was fine - and repair notice on VM website went away.

Today at 3:15pm it started again. There is NO repair notice for our area but our internet is flapping in/out every 2 minutes.

I have been waiting on chat since 16:30 - and so far not a single response.

I have previously run my own ISP - and am certified in multiple router and firewall systems. There is nothing wrong in ‘our end’  

HUB shows green and connected - but IP transport is being “disturbed on Virgins side leading to 100% packet loss post our router. (HUB in Bridge mode - router UBNT UDM SE.)

Hub has been rebooted - router have been rebooted - no change. Nothing have changed in configuration apart from HUB downloaded and installed a firmware update on the 30/1/2023. All problems started the next day.

So who do we contact to get the thing fixed yesterday “fixed” again? 

[Tudor]

Check with Area faults on 0800 561 0061 If you have a VM landline 150 this goes down to post code level. You could also try the web page status, but this is not recommended as it only covers issues that affect a very large number of customers.

VM will not dispatch any technicians while an area fault exists.

If no area faults found:

The primary place to report faults or for service requests is Customer Services on 0345 454 1111/150 if you have a VM landline or wait two or three days for a VM staff member to get to your post.

[Kaz911]

I called the 0800 number - and by magic they said the area is impacted. “But is is a very complex problem so it might take time” 

Ohh well it is so complex 

I do look forward to having other options where we are…

[Sephiroth]

Was the Hub resetting? Important to know that. 

you could look at your modem stats and see whether any of the levels are too high/low and you could look at the downstream post-Rs errors. the upstream modulation is also worth a look. If it’s moving around and not staying on 64QAM then you have upstream noise. 

Good luck with the VM bods on the forum!  Totally scripted.  They are useful for identifying area fault's, though. 

[Kaz911]

No, hub is not resetting. I have manually reset the hub.

And I have checked channel data - only one T3 timeout on upstream - but that is quite normal when VM stick their fingers in the mud. No downstream timeouts. 

But the most disappointing thing is I have been on chat since 4:30. No-one has bothered to even reply. I update chat every 30 min with current status. It is a customer service let down of proportions that should not happen if the ISP was professional.  (I have designed CS systems for MVNO’s and ISPs when I was in that business) - a 1 hour client wait was = “instant termination” of supervisor or support director.

[Sephiroth]

Thing is, if the levels etc are fine then exactly what is flapping?  If it’s not the circuit (and you don’t have a Hub 3), then that leaves your router.  The first thing the VM bods will ask you to do is to put the Hub into router mode to eliminate your router.

The other thing is than in flapping circumstances, as I’m sure you know, levels can be fine until it flaps.  But flapping usually means a disconnection and that would lead to a hub reset.

So, if there’s no area fault, no sequential T3 events, not T4 event, then be prepared for the VM bods to say they can find nothing wrong.

[Kaz911]

Well my neighbour has the same issue 🙂 

But I have put my notebook on - and same issue. 

The flapping is IP availability - huge amount of packet loss. Then for 10-60 seconds - everything clears and no packetloss - and then 100% packetloss again - all while HUB 4 is happy as Larry.

[Sephiroth]

Sounds to me like something flapping in the upstream. The neighbour’s problem kills the need to try router mode.

As you may (or not) know, the modem sends regular keep-Alice’s to the CMTS. These are obviously upstream semaphores. The upstream has to be unreadable for quite a few minutes before the CMTS takes action via a T4 event (reset). So this flapping is of short duration and it’ll likely be a bustard fir them to localise.

[Kaz911]

Hmm I tried pinging the hub

every 10 seconds it responds really slow for 2-3 pings - so from 1-4 ms to suddenly 50-150ms 

[Kaz911]

ok the issue continued through the night - and no answer at all from Virgin Chat.

So this morning I decided to "BIG 60+ sec reset" the hub. 

First with PC on - different MAC on PC gave different IP post reset (vs old IP via Router) - no problems !! 

Then in with the Router Cable, REBOOT the Hub (important) - then check Router - and back to same IP.

But NOW - NO ISSUES with packet loss - 0% PACKET LOSS. 

Then I checked HUB Firmware since the "light" was back to white and not green:

Hub 4: Software Version - 01.02.065.21.EURO.PC20

which is a far cry from the version it updated to in 3 days ago when the issues started.

So my best guess now - and seemingly confirmed - is that the new firmware sent out to my device 3 days ago is the culprit of the massive packet loss that forced my routers interface to drop the connection and switch to backup every 2 minutes.

Now I'm just REALLY nervous that Virgin will ship the same firmware out again overnight - infusing massive packet loss again. But since it is IMPOSSIBLE to talk to them - I have NO idea on how to warn them about the issue.  Worst case I'll do a "big reset" again tomorrow....