Menu
Reply
  • 5
  • 0
  • 0
kevingc666
Joining in
467 Views
Message 1 of 14
Flag for a moderator

NTP issues

I have a security camera which synchronises its time to an NTP server. Recently I've noticed that the log is full of no reply received in time errors. I've tried various NTP servers, including Virgin's own, all with the same result. Is Virgin somehow throttling NTP?

0 Kudos
Reply
  • 1.79K
  • 101
  • 285
chenks
Super solver
462 Views
Message 2 of 14
Flag for a moderator

Re: NTP issues

highly unlikely.
easy to check though, try pinging or connecting to any NTP from your desktop/laptop.
0 Kudos
Reply
  • 5
  • 0
  • 0
kevingc666
Joining in
450 Views
Message 3 of 14
Flag for a moderator

Re: NTP issues

Some servers respond to ping, some don't. Probably down to how it is configured.

0 Kudos
Reply
  • 1.79K
  • 101
  • 285
chenks
Super solver
446 Views
Message 4 of 14
Flag for a moderator

Re: NTP issues

regardless of ping you can still attempt to connect and get a time sync, so are you able to?
specifically the one that your cameras are trying to use (and the other ones you tried) ?
0 Kudos
Reply
  • 16.73K
  • 922
  • 6.57K
Superuser
Superuser
396 Views
Message 5 of 14
Flag for a moderator

Re: NTP issues

It would be interesting to see the log itself.

For my part I've installed an NTP daemon on my Windows machine and turned off the Windows time service (because I'm nosy)

This page discusses getting it up and running: https://www.satsignal.eu/ntp/setup.html

Here's what I get when I run ntpq on it after a few hours.

PS C:\tools\ntp\bin> ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+185.121.25.166  85.199.214.99    2 u  104  128  377   19.857   -2.635   1.459
+ntp.teaparty.ne 85.199.214.98    2 u   58  128  377   17.855   -3.168   1.335
*188-39-213-7.st .GPS.            1 u   61  128  377   24.405   -3.756   2.694
-95.211.120.4 (a 193.79.237.14    2 u  108  128  377   23.140   -1.410   0.824
-thuis.bentware. 46.243.26.34     2 u  119  128  377   27.444   -2.253   0.439

The daemon is polling several ntp servers from uk.pool.ntp.org once every 128 seconds.

A reachability statistic of 377 means there have been no packets dropped from these servers in the last 8 polls of the NTP servers.

Note that I did drop one packet later on as can be seen here

Checking current status of NTP service with ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+185.121.25.166  85.199.214.99    2 u   45   64  377   19.665   -2.454   1.371
+ntp.teaparty.ne 85.199.214.98    2 u    -   64  377   17.492   -2.785   2.213
*188-39-213-7.st .GPS.            1 u   65   64  357   24.504   -3.035   0.367
-95.211.120.4 (a 193.79.237.14    2 u   42   64  377   23.077   -1.405   0.622
-thuis.bentware. 46.243.26.34     2 u   60   64  377   27.394   -1.431   0.333
(Auto-Refresh every 10s  --- CTRL+C to Cancel)

However this dropped off and the reachability statistic returned to 377 later on.

Tim

________________________________________


Only use Helpful answer if your problems been solved.

0 Kudos
Reply
  • 84
  • 3
  • 16
ToffeeSurprise
Up to speed
352 Views
Message 6 of 14
Flag for a moderator

Re: NTP issues

I use Cisco VIRL for my job which needs to contact NTP servers as part of the checks. Unless I run a VPN these requests are blocked by Virgin Media's network.

I think this change was made about 12 months ago because it previously used to work fine. Kind of irritating.

0 Kudos
Reply
  • 16.73K
  • 922
  • 6.57K
Superuser
Superuser
343 Views
Message 7 of 14
Flag for a moderator

Re: NTP issues


@ToffeeSurprise wrote:

I use Cisco VIRL for my job which needs to contact NTP servers as part of the checks. Unless I run a VPN these requests are blocked by Virgin Media's network.

I think this change was made about 12 months ago because it previously used to work fine. Kind of irritating.


I always have a problem with the assumptions inherent in statements such as these.

There are actually a number of reasons why the connection might fail, and they're not all down to deliberate action by Virgin Media.

That said, I'd like to get to help get to the bottom of this, and if it is an issue on VM's side - try to get it resolved.

Are you able to run a tracert to the NTP servers?

Once without the VPN
Once with the VPN

So we can have a look and see if there are any issues with the route.

Tim

________________________________________


Only use Helpful answer if your problems been solved.

0 Kudos
Reply
  • 84
  • 3
  • 16
ToffeeSurprise
Up to speed
313 Views
Message 8 of 14
Flag for a moderator

Re: NTP issues


@ravenstar68 wrote:

@ToffeeSurprise wrote:

I use Cisco VIRL for my job which needs to contact NTP servers as part of the checks. Unless I run a VPN these requests are blocked by Virgin Media's network.

I think this change was made about 12 months ago because it previously used to work fine. Kind of irritating.


I always have a problem with the assumptions inherent in statements such as these.

There are actually a number of reasons why the connection might fail, and they're not all down to deliberate action by Virgin Media.

That said, I'd like to get to help get to the bottom of this, and if it is an issue on VM's side - try to get it resolved.

Are you able to run a tracert to the NTP servers?

Once without the VPN
Once with the VPN

So we can have a look and see if there are any issues with the route.

Tim


Tim,

While I appreciate the reply, not everyone out there who posts on these forums is a know-nothing know-it-all who always blames Virgin Media, and I did a bunch of troubleshooting before coming to my conclusion. I work as a network engineer at an ISP/MSP that is larger than Virgin Media (hence why I use VIRL), and while I've worked in this industry long enough to never rule anything out, I am very confident that Virgin Media are blocking the port, or otherwise interfering. Given that this started happening around the time there were a bunch of NTP attacks in the news, I was not too surprised that NTP started (seemingly) being blocked.

I do not believe it is a routing issue in the VM/LG network, as it is perfectly possible to reach the NTP servers via ping, and tracert shows no issues either (see example below to one of the main pool.ntp.org servers). It seems very apparent that at least this part of the Virgin Media network I am on is interfering with port 123, and I believe this issue first started around October/November 2017, though I may be mistaken.

C:\Windows\System32>tracert 185.121.25.166

Tracing route to 185.121.25.166 over a maximum of 30 hops

1 1 ms 1 ms 1 ms 192.168.0.1
2 9 ms 9 ms 7 ms 10.48.68.1
3 13 ms 10 ms 10 ms perr-core-2b-xe-1110-0.network.virginmedia.net [62.255.32.173]
4 * * * Request timed out.
5 * * * Request timed out.
6 16 ms 17 ms 16 ms m686-mp2.cvx1-b.lis.dial.ntli.net [62.254.42.174]
7 17 ms 15 ms 15 ms 213.46.174.38
8 18 ms 17 ms 15 ms h185-145-200-80.reverse.clouvider.net [185.145.200.80]
9 19 ms 26 ms 20 ms h185-145-200-81.reverse.clouvider.net [185.145.200.81]
10 22 ms 17 ms 19 ms 185.121.25.166


C:\Windows\System32>ping 185.121.25.166

Pinging 185.121.25.166 with 32 bytes of data:
Reply from 185.121.25.166: bytes=32 time=17ms TTL=55
Reply from 185.121.25.166: bytes=32 time=22ms TTL=55
Reply from 185.121.25.166: bytes=32 time=25ms TTL=55
Reply from 185.121.25.166: bytes=32 time=20ms TTL=55

Ping statistics for 185.121.25.166:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 17ms, Maximum = 25ms, Average = 21ms

 

The following resolves the issue instantly with no configuration changes:

Using another ISP such as BT.
Running over a VPN on the VM network.
Tethering using Vodafone/Three.


The following has no impact:

Disabling all firewalls.
Changing NTP servers (except to ntp.virginmedia.com which resolves the issue. time.windows.com also seems to work, so possible whitelisting). pool.ntp.org and nist.time.gov are blocked.


The moment I move off from the Virgin Media network, the issue magically resolves itself.

When using a non-VM NTP server aside from the Windows one, it will not sync and gets stuck in the INIT state.

Thanks.

 

Edit: I also tested uk.pool.ntp.org which works. The main pool.ntp.org does not. It would be interesting to see if your daemon stops working if you point it elsewhere.

0 Kudos
Reply
  • 465
  • 42
  • 133
DreamOfCheese
Fibre optic
308 Views
Message 9 of 14
Flag for a moderator

Re: NTP issues

You may believe that you've narrowed it down to being a network level block by Virgin, however pool.ntp.org works perfectly over Virgin for me, as does the IP address you've pinged in your response, 185.121.25.166.

Following is a paste from my local development server (over VIVID350):

Spoiler

[root@dev ~]# ntpdate pool.ntp.org

 3 Feb 03:00:34 ntpdate[13338]: adjust time server 217.114.59.3 offset 0.000553 sec

[root@dev ~]# ntpdate 185.121.25.166

 3 Feb 03:00:50 ntpdate[13351]: adjust time server 185.121.25.166 offset 0.000293 sec

[root@dev ~]# traceroute 185.121.25.166

traceroute to 185.121.25.166 (185.121.25.166), 30 hops max, 60 byte packets

 1   [Redacted]

 2  * * *

 3  [Redacted]

 4  * * *

 5  * * *

 6  86.85-254-62.static.virginmediabusiness.co.uk (62.254.85.86)  32.314 ms  31.563 ms  30.643 ms

 7  * * *

 8  uk-lon03a-ri1-ae2-0.aorta.net (84.116.135.46)  24.133 ms  19.974 ms uk-lon03a-ri1-ae5-0.aorta.net (84.116.135.42)  24.840 ms

 9  213.46.174.38 (213.46.174.38)  24.017 ms  20.921 ms  18.915 ms

10  * * *

11  h185-145-200-81.reverse.clouvider.net (185.145.200.81)  21.971 ms  22.973 ms  21.954 ms

12  185.121.25.166 (185.121.25.166)  21.881 ms  23.985 ms  20.918 ms

As Tim mentioned, there's a problem with jumping to conclusions or assumptions even if you believe it to be accurate.

0 Kudos
Reply
  • 84
  • 3
  • 16
ToffeeSurprise
Up to speed
297 Views
Message 10 of 14
Flag for a moderator

Re: NTP issues

Given that this topic was created due to another customer experiencing NTP issues, seemingly the same very specific and somewhat obscure error, one should at least be willing to entertain the idea that something has been configured either by accident or design somewhere that is causing issues with NTP for at least parts of the Virgin Media network. Your post comes dangerously close to "it's working for me so therefore it must be working everywhere", which ironically enough would be a far bigger assumption than anything I have suggested.

I submit that doing a large amount of troubleshooting to eliminate essentially every other possible cause is not jumping to conclusions. If you want to be really pedantic, then fine. "Based on my troubleshooting listed below, it looks like Virgin Media may be interfering with NTP on my segment of the network, as I seem to have checked all the other likely causes". Is that sufficiently non-absolute wording?

Tim did not have the luxury of knowing what troubleshooting I had performed when making the original response, but given the large body of checks I have done, I am confident there is a lot of evidence to make my claim a very strong hypothesis. At the end of the day, I do get paid well to troubleshoot a very large network as quickly and accurately as possible. I may very well be wrong on this occasion (and others), but I do make my living by being right, and having some idea about what I'm talking about.
0 Kudos
Reply