I've got a TiVo box and yes, the iPlayer app is slow to load and probably takes a couple of minutes to startup and get a stream going. But once it's running, I have no complaints with what is a perfectly smooth playback with no issues. Many people say the V6 box is a lot quicker at opening the app, eliminating that issue too.
Although both TiVo & V6 are primarily recording boxes, and I'd naturally have to ask whether you're choosing to not record programs where you can? VOD is a great insurance policy with the terrestrial channels, but isn't meant to replace personal recordings.