When it works, Vermin Media is actually quite good - but as you've found the company is useless at communicating with customers, and it is often difficult to speak with UK based staff able to resolve problems. If the speed or package is important, I suggest you hang on in there, give VM a chance to get things working.
If you have the option of a decent Openreach connection, and the VM package isn't anything special, then you may be better off leaving, but if you want good customer service don't make the mistake of going with another big, faceless corporation like Sky, BT/EE/Plusnet, Talktalk or their ilk, as they all have much the same poor customer service - choose a smaller ISP like Zen Internet, Aquiss, or Uno. As a rule these smaller ISPs don't offer the attractive initial discounts that the big companies lure customers with, but don't then stab you with a huge price rise after twelve months, so expect smaller ISPs to look more expensive (you get what you pay for).
Also, if you decide you do want to leave, you need to notify VM that you're leaving within the two week cancellation period that (if memory serves me correctly) all contracts start with. If you wait until you are outside that period, then VM will demand an "exit penalty" near enough the same as the total cost of the contract, so you need to decide immediately whether VM get to fix things, or whether you've seen enough and you're leaving immediately - check your T&Cs. It used to be the case that contacting UK staff was as easy as dialling 150 from a VM landline, selecting options 1, 1, 4, 5 and that got you through to the UK based "retention" team during UK call centre hours, and they are generally good at resolving problems, but if not they would be the people to cancel with. A recent call of mine wouldn't go through to that team directly until I'd reset my telephone password via the offshore staff, so be aware of that niggle.
Failing that, there's an address to cancel in writing here.