Here's my take on it (which is nothing more than a non-expert, personal opinion, so could well be quite incorrect!). Make of it what you will. I reserve the right to be wrong!
When a residential customer signs up for a VM service, they allow VM to install the cable on their property and grant access right to inspect/maintain/service the equipment while the homeowner is a customer. There seems to be a separate process where VM want to access land in order to take their services into a new geographical area or for serving flats etc. in a shared building
https://www.virginmedia.com/lightning/network-expansion/wayleave
There is a download document above which infers that kind of agreement is perpetual (which makes sense to allow VM to maintain/service any buildings/tenants who want to be customers, which may change over time)
My take on the normal residential agreement is that it is in place for as long as the customer remains with VM. I haven't seen anything in there which infers it is a perpetual agreement. It actually mentions the agreement lasts a minimum of 12 months after which it ends.
https://www.virginmedia.com/shop/the-legal-stuff/terms-and-conditions-for-fibre-optic-services
'Section Q. Visiting your home' is probably the most relevant bit
So, in your case, you are the freeholder and the tenant (who originally signed up) had no right to grant permission for the installation (presumably under whatever tenancy agreement you had in place). Since the tenant's agreement with VM has now ended, I can't see that VM has any subsequent right to access your property and certainly not lay additional cables across your property to serve other users.
The matter of the supply cable itself and the outside box may possibly be different as there may be telecoms legislation in place which covers that particular hardware. Section Q above does mention that VM won't remove cables at the end of an agreement. I certainly would not recommend interfering with it yourself (not least because your neighbours are using it) but working with VM to try to resolve the issue.
The quality of VM cable installations is often not great, with green plastic conduit trailing over gardens being quite common. If a subsequent property owner is not a VM customer, why would they want to see that in their garden? Unfortunately, VM does not really seem to have any clear processes and methods to decommission services when they are not in use any more and someone else does not want the VM installation in place.
Wait until the VM forum team get here. Hopefully they will arrange an on-site inspection for you to deal with the issue.