If you go to http://192.168.0.1/ (or http://192.168.100.1/ if you are in modem mode) to get the VM hub GUI. Don't login (unless this is your first visit to the hub GUI, which does require you to login) but instead click on Router Status which is either a button in the top right of the screen or a text link at the bottom of the screen depending on which hub you have. Click on the Downstream, Upstream and Network Log sections and copy & paste the figures from those screens and post them here, don't worry about the formatting or you can post screenshots if that is easier. Be careful to edit out or obscure any reference to your WAN IP or MAC address in the log to protect your privacy.
The community can have a look for you and see if anything in the figures looks wrong and advise accordingly. Or you can wait for the VM staff to pick up this thread, which can take a few days.
Disclaimer - I don't work for Virgin Media. I'm just another VM user trying to help out. My setup: V6 TV box, Vivid 200 Optical fibre with Hub 3 in modem mode connected to a Netgear R7000 router. Telewest/VM user since 2001.
The downstream power levels are right on the recommended lower limit or just above, they should be between -6 and +10. If they were to fluctuate below -6 on occasion then this could be the reason for your fluctuating speeds.
You will need to get VM to send out a technician to get those downstream power levels adjusted so that they are well inside the recommended range.
You can call VM and try to book a visit or wait for the VM staff to get to this thread and they can help you book a visit, but it will probably take them a few days.