Of course; because what does the term ‘latency’ mean? Crudely it’s a measure of the round-trip time between two hosts. So yes, naturally where the ‘target’ system is absolutely does matter. There is no such thing as ‘my latency is...’, it’s more a case of ‘latency to where?'
However, for the OP’s situation is this even relevant? They are on a wired connection, playing in the early hours of the morning; there’s nothing more they can do. Anything else is down to the underlaying ISP infrastructure - not something that a mere customer will have any influence over.