Skip to content

ADR-111: Maintenance windows

Context

Releases of the Linux kernel have accelerated in the past year, so much that we have to upgrade to a new kernel every other week. Every time, the fleet needs to be rebooted.

TPA has historically had a policy of arbitrary reboot times: service admins should expect services to reboot at any time, period. This is causing too much strife for our users who suffer unexpected outages.

Decision

A new maintenance window of 3 hours is now open on every Thursday, from 18:00UTC to 21:00UTC, which is equivalent to:

  • 10:00 US/Pacific
  • 13:00 US/Eastern
  • 15:00 America/Sao_Paulo
  • 19:00 Europe/Amsterdam
  • 21:00 Africa/Nairobi
  • 07:00 Pacific/Auckland

... and, with daylight saving changes for places that observe it:

  • 11:00 US/Pacific
  • 14:00 US/Eastern
  • 20:00 Europe/Amsterdam
  • 06:00 Pacific/Auckland

This is in addition to the already existing (but up to now undocumented) daily maintenance window between 06:00 and 07:00 UTC when unattended upgrades are already scheduled. 06:00 UTC is equivalent to:

  • 22:00 US/Pacific
  • 01:00 US/Eastern
  • 03:00 America/Sao_Paulo
  • 07:00 Europe/Amsterdam
  • 09:00 Africa/Nairobi
  • 19:00 Pacific/Auckland

... or, with daylight saving changes:

  • 23:00 US/Pacific
  • 02:00 US/Eastern
  • 08:00 Europe/Amsterdam
  • 18:00 Pacific/Auckland

Maintenance MAY be scheduled outside of maintenance windows in special cases, including during nights and weekends.

As before, emergency outages are announced on IRC, in #tor-project. Prolonged or planned outages are also be featured on the https://torprojectstatus.org/ site and by email to tor-project@lists.torproject.org.

A new Nextcloud calendar (TPA Maintenance Windows) can be used to track the weekly maintenance windows.

Changes to the maintenance window times will be announced to the tor-project mailing list and made directly in this document and the calendar.

Consequences

During the maintenance windows, TPA may proceed with maintenance work that can impact availability. As a rule of thumb, such operations will take place every other week or so, but might be more or less frequent depending on pending upgrades or the state of the fleet.

We hope maintenance windows should make it easier for users and teams to plan their work to avoid outages.

Unattended upgrades are unaffected: TPA already performs automated upgrades on a daily basis (currently randomly distributed between 6:00UTC and 7:00UTC, defined by Debian's upstream apt-daily.timer systemd unit). During that time, some upgraded services may restart automatically, causing short outages.

More Information

Special cases and examples

Examples of exceptions to the maintenance window include:

  • critical security vulnerabilities (example: remote code execution in GitLab)
  • ongoing attacks (example: card testing attacks on donate website)
  • hardware outages or maintenance requiring immediate intervention (example: server running out of disk space in the next day)
  • Debian major upgrades (example: the Debian 14 "forky" upgrade likely scheduled in 2027)
  • reboots that do not have user-visible changes (example: rebooting a canary server like idle-fsn-01, a server in a pool like web-fsn-01, or fsn-node-08 but only if instances are migrated instead of shutdown)

Those are not exceptions and do need to go through a maintenance window:

  • fleet-wide Linux kernel upgrades
  • outdated libraries alerts requiring manual service reboots
  • Debian point upgrades (which are actually performed automatically during the 6:00UTC maintenance windows, otherwise treated as the above)

Urgent events will not be added to the calendar, but planned maintenance should be added to the calendar.

Survey evaluation

A survey was performed (tpo/tpa/team#42782) to evaluate if a maintenance window would help and the best time it could be. Out of 21 responses so far:

  • 10 wanted a maintenance window
  • 8 didn't need a maintenance window
  • 2 wanted updates to happen as soon as possible
  • 1 didn't answer

It should be noted that some updates are performed as soon as possible on software we know can be reliably restarted, as mentioned above.

Thursday was the most popular (5), followed by Monday (4), Wednesday (3) and Tuesday (2).

8 people answered the question "is 1800 UTC ok?":

  • 4 agreed (half!)
  • one refused to suggest another time
  • another suggested 10:30UTC
  • two suggested after 20:00UTC

Other options considered

We considered doing maintenance on Mondays, but those are too busy: we have our weekly check-in, one-on-one calls and generally deal with emergencies from the weekend.

We did consider doing maintenance work in Europe's morning, but that would have centralized the work on only a part of the team.

We did not consider weekends or Fridays, as we do not want to routinely make our staff work out of hours.

We similarly didn't consider night shifts (which 10:30UTC and 20:00UTC are for some or staff), as we do not have 24/7 coverage, and do not want to disrupt people's sleep schedules.

We considered 19:00UTC as a compromise between 18:00UTC and the suggested 20:00UTC and, in normal time, it works: that is 14:00 America/Eastern and gives us 3 hours to do reboots before the end of the work day (at 17:00), but degrades to 15:00 during summer time which gives us only 2 hours for reboots, which is often not enough.

We also considered targeting reboot notifications better, but feel this was only working around the problem. We can still improve notifications, but we feel more important to set better expectations in the short term.

Linux kernel security advisory review

As of 2026-09-24, there has been 22 Debian security advisories concerning the Linux kernel. This is an all-time high since anarcat has been tracking those announcements, in 2008, and already more than double last year's number (9).

If this trend continues, we will have 29 such advisories in 2026, which is more than one per two week.

The previous record was in 2018, with 14 advisories. Here is a graph of advisories over the past two decades or so:

anarcat@angela:~$ notmuch search  --format=json 'to:debian-security-announce and subject:linux -subject:util-linux' | jq .[].timestamp | while read ts; do date -I -d "@$ts"; done | sed 's/\([^-]*\)-.*/\1/'   | uniq -c | while read count year; do printf "%s %s %s\n" $year $(python -c "print($count * '#')") $count; done
2026 ###################### 22
2025 ######### 9
2024 ########### 11
2023 ######### 9
2022 ########### 11
2021 ### 3
2020 #### 4
2019 ####### 7
2018 ############## 14
2017 ######## 8
2016 ####### 7
2015 ########## 10
2014 ###### 6
2013 ### 3
2012 ### 3
2011 # 1
2010 ##### 5
2009 ######## 8
2008 ######## 8

The mean number of releases over the period is 7.8, standard deviation is 4.8, mode is 3, and median is 8.

This year will be around 4 times average.

Metadata