Yes you can, call in to 150 from a VM phone / 0345 454 1111 from any phone -> Option 1 -> Confirm your details when prompted for them -> Option 4 -> Then either Option 2 or Option 5 (if between 8AM to 8PM M-F & 9AM to 5PM Sa.)
The phone+BB costs can be found here: http://www.virginmedia.com/shop/broadband/compare.html when using this page ignore the big font price and focus in on the 'Then £XX a month." The £xx value listed is what you'd pay as an existing customer (again, ignore the activation fee.)
Note: You can save a little extra on top of that downgrade change by requesting a 'loyalty discount' when you're calling in to do the downgrade.
______ Any opinions expressed by myself are entirely my own and do not represent Virgin Media in any way.
I don't think it is possible, at least it wasn't for me.
I ported my number to BT so that was cancelled (causing my existing package price to go up by £30 pm) but when I rang in to cancel my TV service and leave just the BB (as Bt have really messed up my installation and I didn't want to lose Internet access all together, even though it is the reason I was switching) I was told I couldn't just cancel the TV as it is linked to the Broadband.
The only thing they could do to was offer me the £45 bundle deal for another Phone, TiVo and 50Mb BB and I would have to wait at least 15 days for that.