To try and see what's happening, we'd need other information as well.
From the router status page, copy and paste here (as formatted text) the contents of the tabs marked Downstream, Upstream and Network log. Also, when having problems with latency or packet loss it makes sense to visit Thinkbroadband and set up a Broadband Quality Monitor. That then takes 24 hours to produce a full trace of latency and packet loss. Then, generate a "shared snapshot" chart and post the link (not an image) here.
I'm a Very Insightful Person, I'm here to share knowledge, I don't work for Virgin Media. Learn more
Have I helped? Click Mark as Helpful Answer or use Kudos to say thanks