Hi @snu_mk1,
You will find your friend on the EE connection will probably be using Openreach's network, and for many years with everything Openreach did wrong, the main thing they got right was their latency and response times on their network.
Virgins network has never been great for latency, and you can expect anywhere from 20-40ms difference between them both in peak times when the VM network becomes busy.
With the server already being 80ms at a lowest, this is a target server that is probably far away and not in the UK. If it is in the UK, then it must be very congested.
There is unfortunately no resolution to this problem, you will need to wait for the VM upgrade to hit your area to get XGSPON / Full Fibre. - If you are in doubt, you can always login to the router and under Tools you can do a traceroute, do one to facebook.com as that seems to give me the lowest results.
Hope this helps.
Jaye