Hi,
So, I have been experiencing the same behavior, which appeared to be 'random' high latency spikes. I tried all sorts of things, like tweaking settings on my router (a linux machine), moving the routing container to a different physical host, adding an attenuator to the cable line, and of cause, resetting the router and all that jaz. None of it worked. So I started graphing the number of tracked connections on my router, the latency to the modem (192.68.100.1) and the first hop on the cable line ([DELETED] The UBR?) from my router and started to see that the ping spikes lined up with a spike in the number of tracked connections on my router.
After seeing this I routed all of the traffic out of an ADSL line, that I have for a backup, and although I was seeing the same spike in the number of tracked connections, this time on the ADSL router, I wasn't seeing a similar spike in the latency over this ADSL line. humm ...
So, I started poking about to try and figure out why the spike in connections is happening and it turned out to be server I was running for a bit of software called synapse for a network, called matrix, which is basically a federated chat network. Some of the rooms I have joined have excess of 10k+ members and these members are spread across a large number of servers and what appeared to be happening was when I clicked on a room that I haven't looked at in a while or a after a period of time my server would connect out to hundred of remote servers to update and synchronize state. This would happen very rapidly (within a few seconds).
This seemed to be related but I wasn't sure why, as I said before, when I routed all my traffic out of the ADSL line I didn't get the ping spikes, even know I was seeing the same number of new tracked connections on my ADSL router. Also, during all of this the latency to the modem (192.168.100.1) was low, if it was an issue with something locally I would expect the latency to the modem to spike as well. (Also, the throughput of the VM line never went over 100Mbps down 10Mbps up for this system, so it wasn't capacity related).
Anyway, So I decided to route all the traffic for this chat service out of the ADSL line and leave everything else going out of the VM cable line and ..... nothing changed!, the ping spikes on the cable line were still happening even know the number of tracked connections on the cable line was now lower than before, still high (spiked up to ~4k) but not like before so this stumped me a bit, while looking at all of this I kept on thinking why is this type of traffic having this effect, from VM's point of view they are just routing traffic, at the IP layer, it shouldn't matter what the traffic is, they aren't doing any NAT, I'm doing the NATing.
Looking at the connection tracking table when the ping spikes are happening now shows lots of tracked DNS connections (I run my own recursive nameserver, I have for years, not that this should be a problem, and I think these days a lot of other people will be as well (with things like Pi-Hole appearing) even know they might not know about it), Looking at the logs on the DNS server it looks like all of the DNS lockups are in quick succession and being caused by the matrix server so I tried routing all of the traffic for the DNS servers out of the ADSL line and everything else out of the cable line and the problem went away!. There are still high spikes for tracked connections on the router for the cable line but no ping spikes.
So, at the moment this looks like VM have some sort of setup in their network that is responding to what it believes to be too many DNS queries which, is probably fine if they get it right! and in this case they haven't, even know this might seem to 'abnormal' traffic, its not, but that's not my biggest concern, my biggest concern is that when they detect this traffic their QOSing policies appear to be throttling the _entire_ connection and not just the traffic that's triggering them.
I'm not saying this is the reason, they might have some bad kit in their network, or something that isn't configured right and this is an unwanted side effect but if they are attempting to stop DNS DDoS's or something their setup isn't working and its effecting legitimate traffic, in my case anyway. And I'm not saying this is the same issues everybody else is having, but as more network systems are going towards high levels of rapid UDP traffic, or similar (game streaming, Games in general, QUIC (HTTP/3), etc. I think VM will need to be careful about any "protective" systems they have in place and look into if they are helping or just causing them to loose customers.
Its a bit of a shame, what is suppose to be a backup ADSL line is now a vital part of my internet connectivity but, hopefully, my geforce now sessions will be playable again.
I'm not expecting this issue to ever be fixed, unfortunately, when you call up the support line the person you normally speak to is trained to cut the dross and the best you will get is "we will send out an engineer" its like the assumption of the problem is that the user / CPE is at fault and I'm not saying this isn't always the case, I'm sure turning it off and on again can solve problems, but unfortunately it make it very difficult to get a message to somebody that manages the core routeing infrastructure and services of the network, to somebody that either knows whats going on or has the ability to fix it.
I think I have done enough testing / checks to know that this problem isn't local, unless the modem (in modem mode) is playing with traffic it shouldn't be touching, but that would still be a VM issue and no I don't believe my traffic is abnormal, especially by today's standards.
There also might be times when some computer has got some virus or whatnot on it and is spamming out DNS requests but VM should be stopping source address spoofing, and remote DNS servers should have rate limits if incoming request rates are a problem. It would be nice tho to be asked if something is wrong before causing problems all over the place, to point out what is happening and allow somebody to fix the problem or declare that its normal (like with the letters about SNMP you send out).
Ezryder, your statement about if you VPN the problem goes away, this says to me that its something in VM's network watching traffic rates that isn't happy as you are basically masking the traffic the sensors are looking for which is basically the same test I was doing using an ADSL line.
Can we please try updating the QoS / DDoS detection policies to only limit the traffic they have matched against? Or if this really is an unknown problem can it please be investigated, with an open mind?
And if VM thinks this isn't correct can somebody please explain to me how a cheap ADSL line can out preform a 350M VM line in stability?
Thanks,
Matt.
[MOD EDIT: Personal and private information has been removed from this post. Please do not post personal or private information in your public posts. Please review the Forum Guidelines]
