NTP server is giving a time far in the future

[AlexisMontagne]

Description

Twice in the last week, we observed that one machine in particular (out of a fleet of 100s) has had their clock drifting far in the future (more than 10hrs).

After restarting timesyncd the correct time was updated.

When i queried the ntp servers after noticing the issue the correct time was returned.

Do you have any idea what could cause this issue? Who is running 2.flatcar.pool.ntp.org?

Impact

The wrong time was set in the kernel, created confusion for the applications using the kernel time.

Environment and steps to reproduce

core@xyz ~ $ date
Wed Mar 23 02:39:07 UTC 2022
core@xyz ~ $ timedatectl timesync-status
       Server: 86.108.190.23 (2.flatcar.pool.ntp.org)
Poll interval: 34min 8s (min: 32s; max 34min 8s)
         Leap: normal
      Version: 4
      Stratum: 1
    Reference: GPS
    Precision: 1us (-23)
Root distance: 15us (max: 5s)
       Offset: +8.783ms
        Delay: 135.816ms
       Jitter: 8.976ms
 Packet count: 1962
    Frequency: +4.611ppm
core@core-02 ~ $ date
Wed Mar 23 02:40:32 UTC 2022
core@xyz ~ $ timedatectl
               Local time: Wed 2022-03-23 02:41:03 UTC
           Universal time: Wed 2022-03-23 02:41:03 UTC
                 RTC time: Wed 2022-03-23 02:41:04
                Time zone: UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

core@xyz ~ $ sudo systemctl restart systemd-timesyncd.service
core@xyz ~ $ systemctl status systemd-timesyncd.service
● systemd-timesyncd.service - Network Time Synchronization
     Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; disabled; vendor preset: disabled)
     Active: active (running) since Wed 2022-03-23 02:43:07 UTC; 11h left
       Docs: man:systemd-timesyncd.service(8)
   Main PID: 3970539 (systemd-timesyn)
     Status: "Initial synchronization to time server 138.236.128.36:123 (0.flatcar.pool.ntp.org)."
      Tasks: 2 (limit: 61283)
     Memory: 860.0K
     CGroup: /system.slice/systemd-timesyncd.service
             └─3970539 /usr/lib/systemd/systemd-timesyncd

Mar 23 02:43:06 xyz systemd[1]: Starting Network Time Synchronization...
Mar 23 02:43:07 xyz systemd[1]: Started Network Time Synchronization.
Mar 22 15:39:16 xyz systemd-timesyncd[3970539]: Initial synchronization to time server 138.236.128.36:123 (0.flatcar.pool.ntp.org).
core@core-02 ~ $ date
Tue Mar 22 15:39:24 UTC 2022

Expected behavior

[ describe what you expected to happen at 4. above but instead got an error ]

Additional information

Please add any information here that does not fit the above format.

[kh34]

@AlexisMontagne I'm seeing the same issue using Flatcar stable 3033.2.0 and with:

ServerName=0.flatcar.pool.ntp.org
ServerAddress=86.108.190.23

In my case, ~11 hours in the future.

[jepio]

These nodes come from https://www.ntppool.org/en/, the Flatcar project doesn't control them. The 'X.flatcar.pool.ntp.org' is an alias assigned to a vendor, but refers to the public pool of machines.

For 86.108.190.23 I found a reference to something like this happening last year: https://forum.proxmox.com/threads/ntp-node-has-backed-in-time-for-5h.82767/.

There is nothing we can do here.

[jepio]

You may consider overriding the server config to use a different pool that you trust or following the advice from ntpool.org:

If you are synchronising a network to pool.ntp.org, please set up one of your computers as a time server and synchronize the other computers to that one.

[tormath1]

Just checked, it seems to work now:

core@localhost ~ $ date
Wed Dec  7 21:23:40 CET 2022
core@localhost ~ $ timedatectl
               Local time: Wed 2022-12-07 21:23:43 CET
           Universal time: Wed 2022-12-07 20:23:43 UTC
                 RTC time: Fri 2023-09-08 14:11:40
                Time zone: Europe/Paris (CET, +0100)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no
core@localhost ~ $ sudo systemctl restart systemd-timesyncd.service
core@localhost ~ $ date
Fri Sep  8 16:11:54 CEST 2023
core@localhost ~ $ timedatectl
               Local time: Fri 2023-09-08 16:11:56 CEST
           Universal time: Fri 2023-09-08 14:11:56 UTC
                 RTC time: Fri 2023-09-08 14:11:56
                Time zone: Europe/Paris (CEST, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Anyway, as mentioned by @jepio :

These nodes come from https://www.ntppool.org/en/, the Flatcar project doesn't control them. [...] There is nothing we can do here.

I'm going ahead and closing this.