Packet Loss (main point) and New R36 Firmware Issues (side point)

[plazma]

Hi,

Im currently having an issue which started on the 3rd of this month (july 2012) the modem dropped out and came back shorltly after, although i was unable to connected to anything i tried due to 100% packet loss, so i phone support and i was told there was currently a week signal problem in the area and that i would have to wait until the 12th before it was fixed (my area ref is 31, sorry i was never given a fault number for it).

The issue happened early on in the day, i went out and when i came back around 5:30pm i noticed the connection was working again, joy of joys.

Since then ive had speradic packet loss issues and flapping pings which is causing me a number of problems with online gaming, voip etc, vpn connections to office networks becoming stale, remote backups falling over etc.

I never had a problem before the 3rd, nothing has changed to my internal network. So i waited patiently until the 12th hoping this was part of the same parcel as it were and the issue would be rectified by then.

Alas its the 13th and it doesnt look like its been fixed, so i called support this evening to report the fault, he couldnt find anything wrong with the connection and noteably the connection was stable whilst he was testing it (typical).

During looking at the problem he check over all my signal to noise/power level etc and all were fine:

Modem Initialisation
Stage Status
Downstream Acquisition Locked
Primary Frequency 411000000 Hz
DHCP Complete
TFTP Complete
Time Of Day 20:10:30
Security Enabled
Counters T1,T2,T3,T4,Sync,Resets 0, 0, 0, 0, 0, 0

Downstream Channels
Lock Status Channel ID Frequency Modulation Rx Power SNR Pre RS Errors Post RS Errors
Locked 81 411000000 Hz QAM256 7.9 dBmV 38.5 dB 39942 0
Locked 82 419000000 Hz QAM256 7.7 dBmV 38.2 dB 28994 0
Locked 83 427000000 Hz QAM256 7.3 dBmV 37.9 dB 23885 0
Locked 84 435000000 Hz QAM256 7.0 dBmV 37.5 dB 18260 0
Locked 85 443000000 Hz QAM256 7.2 dBmV 37.8 dB 12571 0
Locked 86 451000000 Hz QAM256 7.4 dBmV 36.9 dB 15094 0
Locked 87 459000000 Hz QAM256 7.1 dBmV 37.3 dB 41118 0
Locked 88 467000000 Hz QAM256 6.7 dBmV 36.8 dB 14967 0
 

Upstream Channels
Lock Status Channel ID Frequency Modulation Tx Power Mode Channel Bandwidth Symbol Rate
Locked 33 45800000 Hz ATDMA 44.0 dBmV 16QAM 6400000 20480 Kbits/sec
Unlocked 0 0 Hz Unknown 0.0 dBmV Unknown Unknown 0 Kbits/sec
Unlocked 0 0 Hz Unknown 0.0 dBmV Unknown Unknown 0 Kbits/sec
Unlocked 0 0 Hz Unknown 0.0 dBmV Unknown Unknown 0 Kbits/sec

The support guy then proceeded to upgrade the firmware (to R36) of the router (ahead of the national roll out), this is a side point and not really part of the issue but something second line needs to be aware of, After the upgrade i noticed a few things wrong with the firmware update.

1. The firewall on the router had been turned OFF, it was previous on (no tick in the enable firewall box, top left), with:

portscan

ip flood

vpn pass throughs (all on)

all turned were ON, the were now turned OFF (due to the main firewall option being off) ?? this surely is a bit of an issue.

2. Due to the weakness of the WPS system (PIN Brute force attacks) i had turned off the WPS (i might as well be running a wep network if im going to have WPS on) option the previous firmware, however after the update the Enable WPS was turned on. Although the superhubs WPS was still turned off, not sure if this would have made it suspetible to still being attacked (not tested). I cant remeber if i had both options before, or just the "disable the super hubs wps", surely its better to just turn off WPS completely i can see you have been aware of the WPS exploitability of the routers since at least january this year.

3. The wireless option had crippled the speed down to 144Mbps and 2.4GHz to give better range in favour of less range and running at 300Mbps, i know some users on here have complained about the range of the router, doesnt really bother me as i use an upstream AP for wirless (with nice big external antennas) and whilst i can appreciate its going to make some people happy they probably get slightly better range post firmware upgrade, i think your going to upset a few people with the fact they now have lost half their wireless bandwidth. I dont remember it having the option for 5GHz before the firmware update, surely when set for 5GHz, like many routers it will support 2.4GHz at the same time, or is it the case its either one or the other, if so that sux a bit...!

4. The remote management option, i thought had gone, although its now moved to super hub settings off the main (3 button) page. Surely it would make sense to be able to also administrate the option from user internetface management under the advanced options (just a thought).

Ok now back to my problem, post upgrade this an extract from a ping against www.bbc.co.uk:

Reply from 212.58.244.69: bytes=1024 time=210ms TTL=51

Reply from 212.58.244.69: bytes=1024 time=176ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=99ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=65ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=130ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=102ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=118ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=137ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=171ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=133ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=142ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=169ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=201ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=117ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=163ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=194ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=184ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=232ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=191ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=135ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=205ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=188ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=148ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=137ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=184ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=171ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=131ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=203ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=184ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=148ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=241ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=145ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=90ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=18ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=20ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=19ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=19ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=19ms TTL=51
Reply from 212.58.244.69: bytes=1024 time=19ms TTL=51

[plazma]

Ok here im testing around 7pm so maybe its the BBCS fault for being to popular, so i run the a test against 162-14-250-212.static.virginmedia.com [212.250.14.162] :

Reply from 212.250.14.162: bytes=1024 time=68ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=71ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=72ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=66ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=68ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=61ms TTL=58
Request timed out.
Reply from 212.250.14.162: bytes=1024 time=70ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=105ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=66ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=68ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=68ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=67ms TTL=58
Reply from 212.250.14.162: bytes=1024 time=67ms TTL=58

Shazam (packet loss) and as you can see it also has a flapping ping value. 

Ive setup a think broadband monitor against my connection, this is the result for the last 24 hours:

Think Broadband Monitor (snap shot 12-13th July 2012) 
This is my Realtime Stats

The RED line is when the routers firmware was updated, look at 1:30 ish, lots of lost packets.. what was going on with the network there in the middle of the night ??

Here is an extract again inside the virgin network to 212.43.163.134:

Reply from 212.43.163.134: bytes=1024 time=19ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=28ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=19ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=20ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=21ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=104ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=18ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=188ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=44ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=18ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=20ms TTL=249

Reply from 212.43.163.134: bytes=1024 time=20ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=18ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=16ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=19ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=203ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=22ms TTL=249
Reply from 212.43.163.134: bytes=1024 time=18ms TTL=249

Lets test against manc-bb-1b-ae5-0.network.virginmedia.net [213.105.174.186]:

Reply from 213.105.174.186: bytes=1024 time=12ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=15ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=14ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=101ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=16ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=15ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=12ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=10ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=22ms TTL=60

Reply from 213.105.174.186: bytes=1024 time=186ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=136ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=16ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=30ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=40ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=71ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=28ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=23ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=28ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=23ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=22ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=22ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=83ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=24ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=32ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=31ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=35ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=46ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=231ms TTL=60
Reply from 213.105.174.186: bytes=1024 time=52ms TTL=60

Seamingly during my testing to various places i get maybe 1 - 10 minutes of no lost and only slightly wandering/flapping ping times and then blam 1 - 5 seconds of really high pings and depending on where in the world im trying to connect to packet loss.

Now im not saying it is, but it seams very coincidential that since you started getting really dug into thelondon underground roll outs the problem has started to manifest its self!?!

There is one thing that also seams apparent, i dont appear to be the only person to be experiencing the issue i found:

This at the start of the month

This again post of the 5th of the month
From the end of last month

From the 11th of this month

There were other lots of other reports from other forums but i think the four above from your own forum helps to illustrate the issue.

ALL of us are seeing and reporting the same thing none of us appear to have had a fix, the problem for me and others doesnt appear to be related to peak times, see my stats above for 1am it was pretty rough in the middle of the night, its now 8pm and the connections holding down better than it has all day (although im still getting high pings and occasional packet loss to some places in the world).

For the majority of users who only use their connection for only friendsface and such like would probably never notice the issue, however like myself a few of us who appeared to be a bit more clued up, are noticing and would love to have our connection back to being stable again.

Prior to the start of this month ive never had ANY issue like this with virgin, maybe the occasional flapping ping, but no packet loss in all the time ive lived at my current address or the places ive lived and had virgin cable before.

Just for good measure here is a tracert to google name servers 8.8.8.8 (ive omitted the part with my IP and internal network form all the below results, being hop 1 and 2):

3 15 ms 67 ms 13 ms brhm-core-2a-ae3-2192.network.virginmedia.net [213.105.103.217]
4 13 ms 7 ms 26 ms brhm-bb-1a-ae8-0.network.virginmedia.net [62.253.174.73]
5 17 ms 13 ms 17 ms manc-bb-1b-ae5-0.network.virginmedia.net [213.105.174.186]
6 26 ms 48 ms 53 ms tele-ic-3-ae0-0.network.virginmedia.net [212.43.163.70]
7 84 ms 107 ms 73 ms 162-14-250-212.static.virginmedia.com [212.250.14.162]
8 71 ms 48 ms 24 ms 209.85.240.61
9 19 ms 34 ms 23 ms 209.85.253.90
10 22 ms 27 ms 28 ms 209.85.243.33
11 29 ms 45 ms 53 ms 216.239.49.36
12 50 ms 35 ms 35 ms 209.85.255.118
13 31 ms 28 ms 35 ms google-public-dns-a.google.com [8.8.8.8]
Trace complete. 

And again:

3 20 ms 30 ms 18 ms brhm-core-2a-ae3-2192.network.virginmedia.net [213.105.103.217]
4 9 ms 9 ms 12 ms brhm-bb-1a-ae8-0.network.virginmedia.net [62.253.174.73]
5 20 ms 11 ms 11 ms manc-bb-1b-ae5-0.network.virginmedia.net [213.105.174.186]
6 17 ms 23 ms 21 ms tele-ic-3-ae0-0.network.virginmedia.net [212.43.163.70]
7 69 ms 69 ms 73 ms 162-14-250-212.static.virginmedia.com [212.250.14.162]
8 19 ms 25 ms 29 ms 209.85.240.61
9 21 ms 32 ms 17 ms 209.85.253.90
10 26 ms 44 ms 25 ms 209.85.240.28
11 29 ms 28 ms 30 ms 216.239.49.36
12 * 37 ms 35 ms 209.85.255.118
13 31 ms 27 ms 27 ms google-public-dns-a.google.com [8.8.8.8]
Trace complete. 

Ok lets try tracing to some place else, say www.number10.gov.uk:

3 18 ms 12 ms 13 ms brhm-core-2a-ae3-2192.network.virginmedia.net [213.105.103.217]

4 12 ms 43 ms 11 ms brhm-bb-1a-ae8-0.network.virginmedia.net [62.253.174.73]
5 17 ms 86 ms 73 ms manc-bb-1b-ae5-0.network.virginmedia.net [213.105.174.186]
6 66 ms 27 ms 33 ms ae0.man11.ip4.tinet.net [77.67.65.141]
7 30 ms 28 ms 43 ms xe-2-0-0.ams21.ip4.tinet.net [89.149.186.122]
8 30 ms 29 ms 28 ms 77.67.4.40
Trace complete. 

and lets try to www.ispa.org.uk : 

3 17 ms 12 ms 12 ms brhm-core-2a-ae3-2192.network.virginmedia.net [213.105.103.217]
4 9 ms 178 ms 22 ms brhm-bb-1a-ae8-0.network.virginmedia.net [62.253.174.73]
5 17 ms 16 ms 26 ms nrth-bb-1b-ae2-0.network.virginmedia.net [62.253.185.85]
6 37 ms 19 ms 47 ms tele-ic-4-ae0-0.network.virginmedia.net [62.253.174.18]
7 20 ms 27 ms 20 ms 10G-2-2-core04.thd.as8586.net [195.66.224.73]
8 46 ms 200 ms 206 ms Te5-1.Core01.thd.uk.as8586.net [212.58.37.118]
9 47 ms 19 ms 19 ms g0-0-1011.core01.ptl.uk.as8586.net [213.246.145.94]
10 24 ms 20 ms 20 ms v150.sw04.ptl.uk.as8586.net [62.8.96.4]
11 27 ms 20 ms 24 ms vweb20.ptl.altohiway.com [195.12.18.226]
Trace complete. 

or www.ofcom.org.uk:

3 10 ms 11 ms 12 ms brhm-core-2b-ae3-2192.network.virginmedia.net [213.105.103.249]
4 28 ms 18 ms 69 ms brhm-bb-1b-ae8-0.network.virginmedia.net [62.253.174.77]
5 52 ms 125 ms 27 ms nrth-bb-1a-as4-0.network.virginmedia.net [62.253.185.105]
6 17 ms 16 ms 21 ms popl-bb-1b-as4-0.network.virginmedia.net [213.105.64.18]
7 29 ms 23 ms 18 ms tele-ic-5-ae0-0.network.virginmedia.net [213.105.159.117]
8 23 ms 51 ms 34 ms 195.99.125.113
9 24 ms 31 ms 31 ms core1-te0-8-0-11.faraday.ukcore.bt.net [109.159.254.200]
10 17 ms 17 ms 16 ms vhsaccess1-pos2-1.faraday.fixed.bt.net [194.72.4.56]
11 31 ms 22 ms 23 ms ftip003169704.vhsaccess1.faraday.fixed.bt.net [81.134.228.66]
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

In all of the above examples you can see high pings whilst im still inside the virgin network and normally once i get past brhm-core.

And nope im not trying to be snarky picking these sites, its just i figured they wouldnt get much traffic this time of the day and the first that came to mind, im sure old sigmund would have something to say about my choice of sites.. (see despite the hair pulling at my internet, my sense of humour hasnt completely failed yet.)

Whats going on with your network, can i have a stable connection back please ??

[plazma]

Bump a di do dhar .. 

[Lykaios]

If you require an instant reply then call 150 from your virgin phone, otherwise the reply time for the forums is normally 7-10 working days.

[plazma]

Just an update, ive combined the thinkbroadband log so far and uploaded it as a single image here:

http://postimage.org/image/teqio5fwn/full/

Note: yes at peak times the ping is worst, but no matter what time of the day or night im getting random packet loss regardless..!

[Lindsey_P]

Hi Plazma, 

Sorry that you've encountered problems with your service, I have checked the network and can see that you're being affected by high utilisation. I have now raised this for further investigation to our Networks Team, if you poart again in 24hrs we should be able to provide you with further information. 

Sorry for the inconvenience.

[plazma]

Thanks Lindsey,

The issue currently (i know its not probably been looked at yet) is still persisting and the packet loss is still causing me issues.

PS i assume poart is actually post (glad to see its not just me who fingers sometimes get carried away lol)

[Darren_G]

Hi plazma, 

The UBR which handles your connection is currently suffering from very high utilisation, particularly at peak times. This has been raised to our network teams under reference F002083591 Work to resolve this issue is esimated to be fixed  at the end of November  (although this may be subject to change).

We're very sorry for the inconvenience this has caused.

[plazma]

Hi Darren,

November, you cannot be serious that it’s going to take 4 months to address this issue, whilst i appreciate that under busy times it UBR is suffering, however im getting dropped packets through the day regardless of time.

I accept im on a contract i don’t feel that it’s fair to expect me to pay the full price for a connection that is unfit for my purposes and can’t hold a stable connection/pipe to anything for more than 10 minutes most of the time, sometimes i get lucky and pipes won’t drop out for 1 hour, but that’s about the best I can get.

The other thing is my connection was fine until the morning of the 3rd, never had lost packets, high pings at times, but im not so worried about the latency issues as much as the packet loss. Something changed on the 3rd and its been bad since them. By the time the connection came back in the afternoon it all started to go wrong.

I can only conclude that either you signed up and activated a massive number of heavy new user/subscriber during the 3rd (which i think is highly unlikely) or some change was made to the UBR on the 3rd to work around the week signal issue I was informed was causing the disconnection on the phone in the morning when the router lost sync.

All my testing appears to indicate the loss is happening at least one/two hops after router and possibly further into the network (although im not able to do the level of diagnosis you are with your own network), which leads me to conclude that the issue is going to be the same for number of users/subscribers.

It appears im also not alone with users up and down the country have had similar issues since the start of the month, ive found some chap in London who experienced an initial fault on what appears to be the same date as me and he is being told it’s a local UBR issue and will also have to wait until November.

http://community.virginmedia.com/t5/Up-to-100Mb-broadband/Extreme-latency-issues-and-packet-loss/m-p...

Either way I feel pretty let down to know that you have just told its going to likely be this bad until November and i assume it’s going to get worse as you sign up/upgrade additional users to the same UBR between now and then.

I would like to allow you this opportunity to come back to me with an acceptable level of bill reduction for the period that im going to be effected and also taking into consideration that I’ve been effected since the 3rd of this month and that will potentially mean 5 months until the fix arrives to address the UBR issues.

[Sythen]

Im not the only person thats had issues with sporadic packet loss with r36.

I reported this issue while r36 was in closed beta, and it seems that i was ignored.

The only fix outside of virgins firmware team fixing the problem, is to switch the SH into modem mode.

that fixed it for me.

I complained to Customer services and they said its an issue with my SH, ive got a new one on the way

So i guess ill see if that fixes it