I have dolby set up follow content on virgin box. Optical lead from box direct to amp. Hdmi from box to TV. Delay set at default of 0. All my problems which were same as yours vanished when I did this
I find if I reconnect the hdmi lead from tv to amp (hdmi arc) the sync does drift out again so I leave this lead disconnected now