Skip to content

Tails time-based APT snapshots

Most of the documentation for this service lives in the Tails website: Time-based snapshots of upstream APT repositories.

This page documents some TPA-specific bits.

Systemd service unit failures

Sometimes, the tails-update-time-based-apt-snapshots-* Systemd service units fail with something like:

Apr 06 05:20:56 apt flock[1372750]: File "pool/main/r/rust-profiling-procmacros/rust-profiling-procmacros_1.0.16-1.dsc" is already registered with different checksums!
Apr 06 05:20:56 apt flock[1372750]: md5 expected: ec65e96702dd9b308cc11c8f87cbf8bf, got: d01aa6a5d949a5e56fbcf700bc95e732
Apr 06 05:20:56 apt flock[1372750]: sha256 expected: c0ea5f15e49e363bc39c5349b6aca71962e503c5e6191bd1e86e5e1a93d04c5c, got: 52d05f7aa9616990f6473037ff8272c6243f685a6c9ca26587345dcc7549ef49
Apr 06 05:20:56 apt flock[1372750]: size expected: 2463, got: 2480
Apr 06 05:20:56 apt flock[1372750]: File "pool/main/r/rust-profiling-procmacros/rust-profiling-procmacros_1.0.16-1.debian.tar.xz" is already registered with different checksums!
Apr 06 05:20:56 apt flock[1372750]: md5 expected: 1aa09a5d684b0f233932dd02dca8035d, got: 09c55f50e6827e439db90b2da8e33a90
Apr 06 05:20:56 apt flock[1372750]: sha256 expected: 10abe5b08cf4083d196c00b54fb548c3838a6ab8e745811f898d88ec92dfda83, got: 6bba96adb637fa415d2c2d7f5dbe3f095a3b76d98a6dbad7c20e3672c25c5fa3
Apr 06 05:20:56 apt flock[1372750]: There have been errors!

One example of this issue with links to other instances of the problem can be seen in: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42578

The high-level procedure to fix this problem is:

  1. Check what caused the failure and whether the offending package is used to build Tails
  2. If the package is not used to build Tails, remove every snapshot that contains the package that we want to replace
  3. If the package is used to build Tails, consult the Tails Team about what to do next

Check what caused the failure and whether the offending package is used to build Tails

Usually failures like this happen because of packages that have ben removed from a mirrored repository and then uploaded once again with different tarball or metadata, causing a mismatch between the cache and the remote repository. Check https://tracker.debian.org/ and see if you can pinpoint what might've happened.

Then, list the snapshots containing the offending file. In this case, the failed units were tails-update-time-based-apt-snapshots-debian-*, so we look in the debian repository in apt.mtl-libvirt-01:

$ cd /srv/apt-snapshots/time-based/repositories/debian
$ sudo -u reprepro-time-based-snapshots reprepro dumpreferences  | grep --fixed-strings 'pool/main/r/rust-profiling-procmacros/rust-profiling-procmacros_1.0.16-1.debian.tar.xz'
s=sid=2025111701 pool/main/r/rust-profiling-procmacros/rust-profiling-procmacros_1.0.16-1.debian.tar.xz
s=sid=2025120402 pool/main/r/rust-profiling-procmacros/rust-profiling-procmacros_1.0.16-1.debian.tar.xz
s=sid=2026011201 pool/main/r/rust-profiling-procmacros/rust-profiling-procmacros_1.0.16-1.debian.tar.xz

The first column shows the serial numbers of the snapshots that contain the offending file.

We can delete any of these snapshots safely if it's not currently used by any of the main branches of tails/tails (stable, testing, devel).

To check this, for every main branch:

  • check if the snapshot is used during the build: on that Git branch, the info is encoded in config/APT_snapshots.d/*/serial

    Note: this can also be extracted from the .build-manifest of the latest successful build, but since we anyway need to query Git for the 2nd check below, better do it this way here as well.

  • check if the snapshot is used by the Vagrant box: on that Git branch, the info is encoded in vagrant/definitions/tails-builder/config/APT_snapshots.d/*/serial

If you've determined that the offending snapshots can be deleted, continue with the next section. Otherwise, ask the Tails Team to handle the problem.

Remove every snapshot that contains the package that we want to replace

To delete a given snapshot, use the tools that we already have to:

  1. make the snapshot Valid-Until be in the past: follow https://tails.net/contribute/APT_repository/time-based_snapshots/#bump-expiration-date with a negative $DAYS_FROM_NOW

  2. delete expired snapshots

    Use the same tails-delete-expired-apt-snapshots command as used in 1 of the relevant tails-update-time-based-apt-snapshots-*.service units.

    For example, if the offending archive is the debian one, this would be: flock --verbose -n '/var/lock/tails-delete-expired-apt-snapshots-debian' tails-delete-expired-apt-snapshots '/srv/apt-snapshots/time-based/repositories/debian'