It's a Microsoft issue.
I have the same issue, but with a completely set-up.
We have 2 Xbox Series X and both are doing it, wired or wireless. I also have a different ISP I can use as well as Virgin, and it's also on that ISP.
Loads of posts on reddit and Xbox forums.
Hopefully will be resolved today in most areas, as I have seen a few posts saying some user have gone back to 0% packet loss, but other have gone down from 100% to 5% packet loss.