Network flapping post VM “area downtime”

[Kaz911]

Yesterday - our area (Brighton BN2) was impacted by a VM repair. It lasted from 12 to 5pm - and during that time - our internet flapped in/out every 2 minutes. 

At 5pm - internet stabilised and was fine - and repair notice on VM website went away.

Today at 3:15pm it started again. There is NO repair notice for our area but our internet is flapping in/out every 2 minutes.

I have been waiting on chat since 16:30 - and so far not a single response.

I have previously run my own ISP - and am certified in multiple router and firewall systems. There is nothing wrong in ‘our end’  

HUB shows green and connected - but IP transport is being “disturbed on Virgins side leading to 100% packet loss post our router. (HUB in Bridge mode - router UBNT UDM SE.)

Hub has been rebooted - router have been rebooted - no change. Nothing have changed in configuration apart from HUB downloaded and installed a firmware update on the 30/1/2023. All problems started the next day.

So who do we contact to get the thing fixed yesterday “fixed” again? 

[Sephiroth]

My hub 4 updated to the green light version inn modem mode.  And no problems.

A pinhole reset, as you’ve hinted, should call down the latest firmware.  If that happens and your problems restart thereafter, then the VM bods will have a challenge on their hands because none of their scripts cover this case other than try router mode.

Then, of course, the buggeration factor will be line conditions that can vary at any time.

Be interesting to see what the FT do when they get to this thread.

[Kaz911]

You can see when I reset the hub - and got my firewall rule working (just before 6am) - reset and test of hub - 10:30'ish.. All good at 11:30'ish.

If you do the "big" reset - firmware seems to 'revert' to "original/old" firmware - which works on my Hub without packet loss. 

But it then downloads a new firmware - and install it at some point - and I guess when that happens my packet loss will be back again. 

[edit] so unless I was extremely "unlucky" and reset my hub the exact time VM "fixed" the issue here - I think my diagnosis is correct. Latest firmware for Hub 4 delivered on the 30th and installed at some point there-after - introduced a big bug that cause excessive packet loss. 

But I'll keep an eye on it and keep "thinkbroadband" running to monitor. If I get packet loss back tomorrow we know what happened.. 🙂 

[Kaz911]

and last but not least:

"Put it in Router Mode" is very unhelpful for debug. As you get a completely different IP range with a different source MAC address - you results will only reflect on physical line issues - and not routing issues. I have seen plenty of routing issues on different ISP's where BGP or manual updates went slightly wrong causing all kinds of issues.

So asking a client to switch from Modem to Router mode to debug will only work "correctly" to identify issues if you clone the mac from your Router to your "test pc"

[Sephiroth]

Yes - you did say early on that you saw this as an IP problem. 

That’ll be very difficult for VM to sort out at this level of FT knowledge. 

Are you saying that with the older firmware, routing was fine and firewall rules were operating (to do what?)?

Turning to the BQM, is there anything artificial going on?  In other words, is it a plain operation affected only by the router’s ICMP rule that allows it?

On routing, are you suggesting that the Hub is misaddressing what’s handed off from the router?

Just trying to assemble facts to break down sequencing etc. 

[legacy1]

Any Docsis problems in modem mode should happen in router mode which is why VM say test in router mode even if there are more other problems in router mode it would be rare to have a problem in modem mode thats not in router mode but it has happened.

Suggest testing in modem mode with a PC to the hub with BQM and check ARP/DHCP is fine in Wireshark.

If the hub is port flapping it can trigger a DHCP request.

[Kaz911]

yes - after 60 second+ "reset" it seems like HUB 4 reverts to original firmware from delivery date. 

So plenty of problems with 30/1/2023 firmware - packet loss > 40% - when it cross 50% Unifi router will switch to backup until packet loss is below 50% (I have a LTE Backup - kind of a requirement on VM) 

Packet loss if you look at my graph change over time - so I guess that might be "overall usage" based - so probably collisions at the "destination switch". 

After firmware was downgraded by the reset - all packet loss went away. But HUB 4 have ALREADY downloaded SW update - so I guess it might try to update overnight to the January 30th firmware. And then I think my line will show the same issue. 

If it happens again I'll

1. Switch HUB 4 to Router Mode without reset - and test "direct connection"

2. Reset HUB 4 again to get "old firmware" - test with PC again

3. Re-test with HUB 4 and Router. 

Then results will be conclusive. 

The only reason I talked firewall was I had not setup "thinkbroadband" before - so had to open up for ICMP. But their published Address for their *.*.*.*/28 network is wrong so router would not accept it and I was not awake enough at 5:30am to catch it right away *G*.  (their published address is a *.*.*.64/28 - should be .60/28 in case anyone else runs into FW not opening up)

[Kaz911]

So a night has passed and HUB 4 have NOT updated its firmware and still no packet loss. (And no SMS updates on the "ticket" in VM system - but maybe VM have not made any progress in fixing the "issue")

The latency at 4pm was a download of a backup from a remote site. One of the reasons you get huge latency increases on high speed downloads is the asymmetric upload/download ratios of Cable connections. Normal ISP rule of thumb for ISP's is you should keep your outgoing capacity to at least 10% of incoming to fully utilise your bandwidth due to the way IP4 traffic works. If you like Virgin have a 1200/50 - there is not enough outbound "airtime" to balance traffic with "receipts" sent. So a 1200 download should really be paired with a 120 upload capacity as a minimum to not impact other users on your network.

I really look forward to full symmetric real fibre (counting down..) 

In the other end is a similar router setup but on full 1000/1000 symmetric real fibre. That line has 1/3 of the latency increase during the same operation. 

[legacy1]

---

@Kaz911 wrote:

The latency at 4pm was a download of a backup from a remote site. One of the reasons you get huge latency increases on high speed downloads is the asymmetric upload/download ratios of Cable connections.

---

No the QoS/BWM is not that good in the hub if it has one at all so if you do your own by router you limit -10Mb of download and -1Mb of upload to stay under the hubs rate limit.