PO7 area - high latency and packet loss

[andybundy]

Good morning VM,

I've been experiencing some awful network responses for the past three or more weeks, seemingly starting not long after the migration to 21CV (VOIP) telephone service, although this could be a coincidence.  In the past 12 hours or so I've re-vamped my Smokeping incidence and revised my own local network graphing to best show the problem objectively.  The following graphs are derived from latency readings using ICMP Echo 'pings' in groups of 20, every 5 minutes.  The interface graphs are drawn from my pfsense firewall's WAN and LAN interfaces.

I am running my Superhub 3 in modem mode (the only way to go!).  The purpose of the two distinct measures is to eliminate suggestions that the problem is on 'my side' of the Superhub modem.

For the external latency measures, I've selected points from a traceroute out to Cloudflare (1.1.1.1):

root@Tower:~# traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
1 192.168.0.1 (192.168.0.1) 0.161 ms 0.152 ms 0.139 ms
2 * * *
3 cosh-core-2b-ae50-0.network.virginmedia.net (80.0.142.213) 97.853 ms 103.066 ms 109.307 ms
4 cosh-core-2a-ae1-0.network.virginmedia.net (80.3.160.253) 128.970 ms 115.626 ms 121.710 ms
5 * * *
6 * * *
7 tcma-ic-2-ae9-0.network.virginmedia.net (62.253.174.178) 282.605 ms 205.774 ms 199.425 ms
8 162.158.32.254 (162.158.32.254) 236.945 ms 230.882 ms 224.249 ms
9 162.158.32.11 (162.158.32.11) 255.533 ms 162.158.32.9 (162.158.32.9) 243.045 ms 248.996 ms
10 one.one.one.one (1.1.1.1) 211.733 ms 194.144 ms 180.124 ms

The two hops highlighted in red are shown in the first two graphs below, followed by the first hop (192.168.0.1) which is my router/firewall machine;

It is quite clear that the first hop in the traceroute is very consistent and sub 1ms throughout, whereas the other two graphs show much higher responses which seem to be caused in the 'local' network in or around the VM exchange (I'm presuming this based on the 'cosham' elements on the names.

Over a  10hr period these seem to be acceptable (though 2-300ms pings to the exchange really ARE NOT good.  Zooming into the past three hours shows the true story:

Here you can see the actual problem which is not averaged or smoothed, showing considerable packet loss which explains the horrendous problems, including dropped connections, frequent caching and updated in streaming media such as Netflix.

Just for clarity, here are stats for the same three hour period from my firewall graphed using SNMP:

To summarise the above.

1. The quality of Virgin Media's network has reached unusable levels, with levels of packet loss which render streaming video, for example, next to impossible at peak hours.
2. The graphs and datapoints above are derived using a standard 'Smokeping' installation running on an HP Proliant server machine.  I have a dedicated pc running PFSense, all local networking runs on a Dell Gigabit switch, with 10Gbit connections.
3. I've included data from server <> firewall, and SNMP graphs of the firewall interfaces to highlight that the problems are not caused by my usage.

Frankly I would hope that VM are already aware of this and resolving the problem - I should state now that I'm unlikely to remain with VM if this takes six months to fix as it did some years back.  I'm not trapped by contract, and following the work of a number of Openreach chaps last week in my road I'm aware that Fibre has arrived....

Swift responses would be good.

[David_Bn]

Good Afternoon @andybundy, thanks for your post on our Community Forums, and I'm sorry to hear of the difficulties you've faced on our broadband services, that have come into play since the switch over to our 21CV services.

I understand that since your post you've been able to speak to our team further in relation to this issue.

Please provide us with an update on this matter when you have some free time.

Kindest regards,

David_Bn

[andybundy]

Sorry for the delay - I guess I could blame the network problems to same degree 🙂

Engineer came in Saturday morning (of course, the one morning my carers were late) - he checked and adjusted signal levels and replaced my somewhat aging Superhub with a shiny new model. 

There have still been a couple of short periods of packet loss as seen on the chart above.  Note that while these might suggest periods of higher network usage (though at 0200?) this should result in higher ping times, never packet loss.  There is a presumption, I believe, amongst VM staff that a small amount of packet loss is acceptable - but this is never the case.  Packet loss will cripple streaming video (eg Netflix or similar) because this uses UDP which cannot correct for it.

I would welcome the news that your network teams continue to monitor for errors and capacity issues.

[Paulina_Z]

Hi @andybundy

Thank you for coming back to us and and thank you for your update on your technical appointment. As you stated, you're using the Hub 3 in Modem Mode at the moment. 

Because of this, we're very limited in what power level issues we can see. Would it be possible for you to put your Hub 3 in Router mode for at least 24 hours so we can run some further checks on our systems?

This way we can see what we can do to help and if your Hub is running as it should be. 

What did the engineer advise about this situation?

Please let us know so we can help further if needed.

Thank you.

[orangeloops]

Hi Andy

Just thought i'd get in touch as I am in the PO7 area (Waterlooville) and had an engineer today. My upload speed will not go above 15Mbps. I am on the M500 package and they weren't able to resolve it.

I've had a new hub fitted and he spent a couple hours here but left unable to do anything. Did you manage to get it resolved in the end?

[andybundy]

I find myself despairing the fact that VM support staff remain unable to access the cabinet or exchange level equipment in order to discern the same information - those of us who opt to place their devices in modem-mode often have good reasons for doing so, and do not find it easy to simply 'turn it back on' based on the amount of specific configuration steps which could not be replicated on the supplied router.

I should not need to point out that they should also be aware of the disability flag set on my account which should at the very least change the request to 'can you' rather than 'will you'. 

Regardless, both I and my family have found that the changes made have improved the quality of the connection considerably so we are happy to consider this matter solved.  That said - why do we as customers have to consistently be raising these issues when the kind of issues we were having should be visible to the VM management platform?

[Matthew_ML]

Hey andybundy, thank you reaching back out and letting us know that your issues has been resolved. 

However I am sorry to hear about the experience you have received.

I would like to take a look into this and raise a complaint for you if you wish.

I will send you a PM and we can take it from there.

Please look out for the purple envelope. Thanks