Menu
Reply
  • 214
  • 8
  • 78
PaulMoore
Superfast
1,010 Views
Message 1 of 58
Flag for a moderator

Debugging the buffering by going back to basics...

Hi folks.

 

Not sure where to start with all this, but I'll try to keep it short.

 

Trying to watch ITV4 (Roland Garros) over the last few days via a PC (8 core, 8GB ram, Windows 8.1, Firefox & flash) on my home VM 100Mbps connection.  Basically, it was hopeless.  You'd see 3/4 seconds of a point, then it'd buffer for 30 seconds or crash completely.

 

I'll spare you the 1st line nonsense, apart from mentioning the fact he changed the WiFi channel to a heavily congested one, despite telling him it's a wired connection!  After trying various things, he proceeded to tell me it's my PC at fault.  I refused to allow remote access and asked to speak with 2nd line.  He initially refused, saying they'd only repeat what he'd just said.

 

10 minutes later, I'm through to 2nd line.  The call landed with "Steve" who I must say, was very helpful and patient Smiley Wink

 

The next 1hr 20 minutes were spent arguing (albeit respectfully) that the other party was at fault.  Steve ran a variety of checks and categorically stated "it's not a network or server fault, I need remote access to your PC".  Again, I refused.  I've nothing against Steve, but this machine contains sensitive corporate information, the majority of which held under NDA.  Although it's encrypted, it's not a risk I'm prepared to take for something as trivial as watching TV.

 

Long story short, Steve couldn't continue without remote access.  Having already investigated the issue last night, I knew the PC wasn't at fault.  Although Steve obviously couldn't agree to it, I made it clear that if I had to debug this issue too (won't go into detail about that backstory), I intended to bill VM for my time (£45p/h).  I agreed to throw in a spare SSD and install a vanilla build of Windows 7x64.  Steve said he'd call back today at 12PM.  He did... bang on time.  As the phone was ringing, Windows was doing its first reboot... so he literally had to install Firefox & Flash himself.  Every other device in the network (NAS's, phones, tablets etc) were disconnected, leaving just a gigabit NIC.  A speed test showed 107Mb down and 6.5 up, mirroring yesterday's results.

 

So with Windows running *just* flash player & Firefox (absolutely nothing else, not even updates), it proceeds to buffer on multiple channels.  Initially, Steve blamed ITV's stream until channels like Dave, Watch & Sky1 started buffering too.  At this point, Steve starts to raise a ticket.  I then gave him these two videos, showing the issue and the cause.

 

https://www.youtube.com/watch?v=eXn8CjlRJro

https://www.youtube.com/watch?v=kGGZ4JzP60M

 

The first video shows the stream repeating, along with the network traffic associated with each fragment.  At 58 seconds, you'll see two requests highlighted with an arrow.  Despite having different fragment numbers (supplied by the bootstrap), the server returns exactly the same binary data... twice.  The first request takes a few hundred milliseconds (which is perfectly normal given the size of the fragment).  The next request returns almost immediately.  The server runs Varnish cache, which explains why the 2nd response came back so quickly.  However, it should never receive the same fragment twice unless the first request fails.  If it does, the video player will obviously play it twice.

 

Towards the end of the video (when James May is talking), you'll see the same happen again.  The server makes two requests (for argument sake, 5 seconds of the video, followed by the next 5 seconds) but instead of returning the next part of the stream, it returns exactly the same segment again.  Again, the video player repeats that block.

 

The second video shows the stream buffering.  Again, Steve was quick to blame network congestion, bandwidth bottlenecks and/or server issues.  I ran diagnostics here too, which revealed something interesting.  The rate at which each request goes out to "multiscreen.virginmedia.com" is barely sufficient to sustain the bitrate for 720p.  Some of the requests (not shown in the video) returned empty with bizarre HTTP response codes, meaning the video player has just 400/500KB (from the previous request) of video data left in the buffer and nothing else to display.  400/500KB of SD video isn't too bad and is unlikely to buffer unless a significant percentage of future requests fail.  At HD (High) quality, the buffer is exhausted before the "retry" request goes out to the server... which causes it to buffer.  Assuming that response comes back in a reasonable time, it'll play the next segment.  If there's a significant delay between the two segments, the player frantically calls the server to catch up.  Even after it has the data, the flash player needs to catch up too before playback can resume.

 

From all this, I'd make the following observations.

 

1)  The level of fault tolerance is way, way too low.  The requests either need to happen more often at higher bitrates or the server needs to return more data with each request to sustain the player if future requests fail or are delayed.  I appreciate you can only return as much data as you have (particularly in the cases of live streams), but the time-shift at the start of playback needs to be increased... again to increase fault tolerance in the event of network congestion at either end.

2)  It is not handling HTTP response codes correctly, if at all.  If a response type is "plain" and it expects binary, a "200" isn't appropriate.  It's quicker to parse a response code than try to parse the body, decide it's malformed and re-issue the request.

3)  If you must cache (and I see the logic in doing so), the player at least needs to perform a basic check to see if the response it's received resembles others within a particular time period, either with a checksum/digest.

4)  Because of the above, it'd only take the slightest network issue to cause considerable buffering.  It looks like it can sustain 1 failed request (2 at a push, depending on the previous fragment size), but given the demand and various other factors, I'd argue that's not enough.  If you compare it to YouTube, their fragment size is larger even at lower bitrates.  It's still possible to buffer YouTube vids (as we all know!) but it's much more tolerant to OU/congestion issues.

5)  Stop being so quick to blame the customer's equipment!  It took all of 5 mins for 1st line to write off my PC as knackered and in need of remote support.  If you remote connect and it turns out to be the device, fair enough... but it's a bit premature to deny all liability until you've actually looked at it.

6)  1st-line don't listen to customers.  If the customer says "it's wired", it's obvious that changing the WiFi channel won't help.  Likewise, blindly moving it from x to y channel without any method of scanning them is only likely to lead to further support requests with "my wifi is slow" etc.

7)  Running a tracert to "anywhere.virginmedia.com" is pointless!  This is probably the only issue I would mention with Steve's support.  I ran tracert from the device, he ran it from the SH2AC.  "See, all the requests come back in under 30ms".  While true, it's irrelevant.  The SWF is delivered via that endpoint, but the data comes from an entirely different endpoint (multiscreen.virginmedia.com).  Had I taken this explanation at face value, I'd be no further forward and leaning towards my PC being at fault.

 

I put the bill to VM this evening and they credited the 2hrs (£90) to the account immediately Smiley Happy

 

As an FYI - I was given a 7-10 day lead time to investigate... but I doubt it'll be fixed this month.  Other than #7, I have no complaints with Steve at all.

 

So if your TV Anywhere buffers/repeats and VM swear blind it's your PC, it may not be.

  • 131
  • 0
  • 17
Mobes
Dialled in
982 Views
Message 2 of 58
Flag for a moderator

Re: Debugging the buffering by going back to basics...

Virgin Media don't seem to be taking this seriously at all.

 

Like you I was trying to watch Roland Garros on my iPhone 5c this week.

 

I gave up! Stuttering, repeating, skipping, breakup. It was appalling.

 

EVERY channel I looked at on TV Anywhere was the same.

 

This week is the first time I've had major problems like this.

 

LOADS of people on twitter, here etc etc have logged the same problems they can't and aren't all down to our products or broadband connection.

 

Sort it out VM!

 

0 Kudos
  • 214
  • 8
  • 78
PaulMoore
Superfast
975 Views
Message 3 of 58
Flag for a moderator

Re: Debugging the buffering by going back to basics...

To be fair, Steve couldn't have been more helpful.

 

There's quite a few things going on; with issues spanning various departments/teams.  Trouble is, this is only the 4th time I've tried to use TVA.  The first time was 18 months ago and it wasn't really usable back then... and little appears to have changed since to be honest.  Unless it's an obvious fault, both 1st and 2nd line didn't appear to know how to handle it (MTR'ing the wrong address is a prime example)

 

I'm away during Wimbledon week, so I'm hoping it's all resolved by then.

0 Kudos
  • 8.3K
  • 518
  • 2.3K
Superuser
Superuser
972 Views
Message 4 of 58
Flag for a moderator

Re: Debugging the buffering by going back to basics...

Whilst your story is interesting reading for those of us with a technical bent (but holds no surprises), you are actually likely to ADD to the total sum of misery.

 

Joe Blogs will read the "support didn't understand" "my pc wasn't at fault" and conclude that in EVERY case, that is true. And insist their malware infested , poorly configured system with drivers from the year dot, is equivalent to your fresh test rig.

 

The reason I point it out is not in any way to attack, or belittle what you have taken the time and trouble to point out.

 

I think the post and subsequent discussion probably belong in a more technical part of the forums so it can get some actual traction (hopefully) instead of getting lost in the general miasma of cat calls about TV anywhere.

 

Nice work BTW

 

 

 

 


0 Kudos
  • 214
  • 8
  • 78
PaulMoore
Superfast
936 Views
Message 5 of 58
Flag for a moderator

Re: Debugging the buffering by going back to basics...

Thanks.

 

Any ideas where the thread should be, or who to ask about moving it?

0 Kudos
  • 887
  • 69
  • 274
Crockret
Well-informed
911 Views
Message 6 of 58
Flag for a moderator

Re: Debugging the buffering by going back to basics...

The topic relates to TV Anywhere, of course it should be here. I'm really grateful that someone who knows about this sort of stuff has managed to identify issues that VM techs haven't got anywhere near to solving in literally years. Thanks for taking (even more) time to post all the details & evidence.

I'd like to think this is a breakthrough of sorts, but even armed with clear evidence of how and why the TVA system is failing, I've little condidence in VM ever getting this right.

Hats off to you for not accepting the blame game that support always try to play.
  • 50
  • 0
  • 11
jhactually
Dialled in
885 Views
Message 7 of 58
Flag for a moderator

Re: Debugging the buffering by going back to basics...

Thanks for this.

 

I've been reporting problems with this services for some time.  The last time I spoke to them I was speaking to 1st line, but they kept talking to someone in 2nd.  I also refused them connecting to my computer and was also asked to do some pointless stuff which I had to point out did not require (in my case anyway) doing.

 

I was also told that 'this problem has not been reported before' which obviously, as anyone looking at these forums would know, is not the case.

 

They also tried to fob me off saying 'The service is free anyway so it's not like you're paying for it'.  This is a strange way to keep hold of your customers!

 

I have also offered to debug their code, but hadn't got right into it yet - I'm glad I found this an that I don't need to do it!

 

I think it's a shame that Virgin don't see TVA as we do - part of our service.

 

I was also told that if anyone else in my house was streaming at the same time as me, they couldn't guarantee TVA would work - I had to explain that a 100Mb connection is WAY more than enough to play quite a few streams....well, it should be, anyway.

 

Please Virgin - sort this problem out and see your customer satisfaction grow.  I can stream from many other services quite easily.  In fact, I found one night when TVA was working, that I could open lots of TVA streams in different tabs - I think I got to 6 - all playing properly.

 

Considering I can't play on my unmodified Nexus 9 (TVA says it;'s modified) due to (possible) license/security violations (and i'd like to know what they might be), and the fact you can only have 2 devices for similar reasons, it seems odd to me that I can open as many streams as I want in the browser - when TVA works.

 

Please, as above, can we do away with Flash technology, and please, make good use of the info you've been provided with, info your software dev team should have been able to give you in the first place.

0 Kudos
  • 214
  • 8
  • 78
PaulMoore
Superfast
872 Views
Message 8 of 58
Flag for a moderator

Re: Debugging the buffering by going back to basics...

"The service is free anyway so it's not like you're paying for it'.  This is a strange way to keep hold of your customers!"

 

I had exactly that line too, when I called with the invoice.  Bit of a loaded question, but I asked if I could cancel the TV services and watch online instead.  Unsurprisingly, he said you have to pay to use it... and his argument promptly fell apart.  I did explain though, the bill was for services rendered and if they wanted to compensate for a loss of service too, that's welcomed but not expected.  To be fair to him too, there were no arguments (and I expected it, to be honest), he just credited my invoice to the account.

 

I agree with Kippies though.  This isn't intended to be used as an excuse for a tired, malware-riddled machine.  However, it does demonstrate that even if the cleanest of environments under damn-near perfect circumstances (ie - techies at either end of the phone), it's still a mess.  I'm not kvetching here either... it's virtually unusable unless you're happy to halt all other network traffic.

 

I'm not sure how receptive VM are to allowing 3rd-party plugins, but it's probably quicker to write my own video player to bolt into Chrome/Firefox.

  • 214
  • 8
  • 78
PaulMoore
Superfast
861 Views
Message 9 of 58
Flag for a moderator

Re: Debugging the buffering by going back to basics...

Bit more debugging...

 

There appear to be 2 servers running the multiscreen.virginmedia.com address, labelled 01 and 02 in the cache names.  Presumably there's a load balancer controlling traffic, so where you end up depends on current demand/resource constraints etc.

 

Interestingly, if you hit the same server twice with the same fragment number, the server returns a plain/empty response.  That'd be fine, but the video player isn't handling the empty response correctly.  However, if the video player makes the same request twice with each one landing at either server, they both receive binary data... so that section of video plays twice.

 

I've just sat and watched 5 minutes of "2 Fast 2 Furious" and watched the live network traffic.  You can predict exactly (and with 100% accuracy) when it's going to buffer or repeat before it's happened.  So far, every single time it happens (which is 20+ times in 3 minutes), each request lands at differing servers.  One request ended up at the same server which returned an empty response.  The video player handled it and had sufficient data in the buffer to *just* avoid buffering.  The next request doesn't respond for nearly 4 seconds... at which point, the player is ready & waiting to play *not* the next section, but the section after next.  It makes 2 bootstrap requests which respond immediately.  The timestamp is identical.  The first response comes in, the player is fine.  Once the second response comes in (with the same timestamp), the player crashes and the browser needs a refresh to bring it back.

 

I've saved all the network traffic and isolated the instances which cause a complete loss of video stream.  If anyone at VM wants the Fiddler traffic, please PM me.  I've also recorded the entire session, which I'll render out and upload to YouTube tomorrow if I get chance.  If you're unfortunate enough to have YouTube buffer while watching a video about TVA buffering, I'll upload the MP4 in full Smiley Wink

  • 887
  • 69
  • 274
Crockret
Well-informed
821 Views
Message 10 of 58
Flag for a moderator

Re: Debugging the buffering by going back to basics...

Loving your work Paul.
0 Kudos