Skip to content

Dealing with time

Time is a complicated concept and it's hard to implement properly in computers.

This page aims at documenting some bits that we have to deal with in TPA.

Host clock desynchronized

If a host's clock has drifted and is no longer in sync with the rest of the internet, some really strange things can start happening, like TLS connections failing even though the certificate is still valid.

First, check the clock against a known good source (e.g. your own computer):

TZ=UTC date --iso-8601=seconds

This will display the current UTC time in seconds, unambiguously. Make sure you run the same command on both sides to have the same timezone. If you are missing such a reliable source for some reason, you can check the mirror system (assuming its time is correct) or any other reliable webserver by checking the headers:

> curl -sI test.torproject.org | grep -i date
Date: Tue, 14 Apr 2026 19:31:54 GMT

The NIST time.gov site is also a good source:

> curl -sI https://www.time.gov | grep -i ^date
Date: Tue, 14 Apr 2026 19:31:54 GMT

You can also look at the state of the fleet time synchronization in the Grafana Time dashboard.

If a host has time synchronization issues, check that the ntpd service is still running:

systemctl status ntpd.service

On some hosts, this will be a different service, ntpd-rs or timesyncd:

systemctl status ntpd-rs

or:

systemctl status systemd-timesyncd

You can gather information about which peer servers are drifting:

ntpq -pun

With ntpd-rs you can check the status with ntp-ctl status:

$ ntp-ctl status
Synchronization status:
Dispersion: 0.000713s, Delay: 0.029473s
Stratum: 5

Sources:
ntpd-rs.pool.ntp.org:123/51.161.47.242:123 (1): +0.001606±0.001056(±0.046275)s
    poll interval: 256s, missing polls: 0
    root dispersion: 0.000015s, root delay:0.000015s
ntpd-rs.pool.ntp.org:123/216.128.178.20:123 (2): +0.004368±0.002472(±0.030223)s
    poll interval: 1024s, missing polls: 0
    root dispersion: 0.007965s, root delay:0.019409s
ntpd-rs.pool.ntp.org:123/208.81.1.244:123 (3): +0.008850±0.002244(±0.050996)s
    poll interval: 1024s, missing polls: 0
    root dispersion: 0.021805s, root delay:0.034073s
ntpd-rs.pool.ntp.org:123/167.160.187.12:123 (4): +0.000765±0.001639(±0.027886)s
    poll interval: 1024s, missing polls: 0
    root dispersion: 0.000473s, root delay:0.001587s

Servers:

There you can force a synchronization with force-sync command:

anarcat@marcos:~$ ntp-ctl force-sync
Determining current time...
Your clock is already within 1s of the correct time

With timesyncd, check the status with the timedatectl command:

root@fsn-libvirt-01 ~ # timedatectl timesync-status
       Server: 195.201.173.232 (0.pool.ntp.org)
Poll interval: 34min 8s (min: 32s; max 34min 8s)
         Leap: normal
      Version: 4
      Stratum: 2
    Reference: 83BC03DF
    Precision: 1us (-25)
Root distance: 5.653ms (max: 5s)
       Offset: -364us
        Delay: 949us
       Jitter: 156us
 Packet count: 49

Logs for this service are sent to the systemd, so you can take a look there to see if some errors were mentioned.

If restarting the service does not work, verify that a firewall is not blocking port 123 UDP.

Daylight saving handling

Sending an announcement

For some reason, anarcat ended up with the role of "time lord" which consists of sending hilarious reminders describing the chaos that, bi-yearly, we inflict upon ourselves changing time backwards and forward.

This used to be done as a one-off, but people seem to like those announcements, so we're trying to make those systematic.

We consult the tzdata announcements since the last mailing and, if time permits, the full discussion list which helps resolve sticky questions and rumours, which often end up there.

We also go through the daylight saving time Wikipedia page which has good overviews.

Overall, the procedure is generally to look at the last similar (say, the last spring) announcement and treat it as a template, changing the dates here and there, but also changing phrasing slightly so that people want to read it. It's also good to look at the previous mailing (say, last fall if we're in spring) to get the last example.

Calendar

To get a reminder to do that work, a calendar was created in Nextcloud. First attempts at creating the calendar by hand through the web interface failed. It's unclear why: events would disappear, the end date would shift by a day. I suspect Nextcloud has lots of issues dealing with the uncertain time during which daylight savings occur, particularly when managing events.

So a calendar was crafted, by hand, using a text editor, and stored in time/dst.ics. It was imported in Nextcloud under the Daylight saving times calendar, but perhaps it would be best to add it as a web calendar. Unfortunately, that might not make it visible (does it?) to everyone, so it seems better that way. The calendar was shared with TPI.

Future changes to the calendar will be problematic: perhaps NC will deal with duplicate events and a new ICS can just be imported as is?

The following documentation was consulted to figure things out:

Curses found

Doing this research showed a number of cursed knowledge in the iCal specification:

  • if you specify a timezone to an event, you need to ship the entire timezone data inside the .ICS file, including offsets, daylight savings, etc. this effectively makes any calendar file created with a local time eventually outdated if time zone rules change for that zone (and they do), see 3.8.3.1. Time Zone Identifier and 3.6.5. Time Zone Component
  • blank lines are not allowed in ICS files, it would make it too readable
  • events can have SUMMARY, DESCRIPTION and COMMENT fields, the latter two which look strikingly similar and 3.8.1.4. Comment
  • alarms are defined using ISO8601 durations which are actually not defined in a IETF standard, making iCal not fully referenced inside IETF documents
  • events MUST have a 3.8.7.2. Date-Time Stamp (DTSTAMP) field that is the "date and time that the instance of the iCalendar object was created" or "last revised" that may differ (or not!) from 3.8.7.1. Date-Time Created (CREATED) and 3.8.7.3. Last Modified (LAST-MODIFIED) depending on the 3.7.2. Method (METHOD), we've elected to use only DTSTAMP since it's mandatory (and the others don't seem to be)
  • the Netherlands used to have the weirdest timezone, UTC+00:20, also known as "Amsterdam Time" or "Dutch Time" which was +0h 19m 32.13s ahead of Greenwich time, until German occupation during World War II, at which point the Netherlands switched to Berlin time