Updating Mirrors¶

Analyzing failures¶

Check the history of the mirror checks in Icinga2.

If the mirror is working now, it might have been a transient error, either on the mirror or on Mirrorbits. Depending on the error, it might make sense to still notify the admin.

If the mirror is still broken, continue.
Has the mirror failed already in the last 6 months?

If this is the second time this mirror failed in the last 6 months, and there's no indication the root cause of the problem will go away once for all, go to the "Removing a mirror completely" section below.
Is the problem a red flag?

If the problem is one of these:

- The mirror regularly delivers data slowly enough for our cronjob to report about it. - We're faster than the mirror operator to notice breakage on their side. - The mirror uses an expired TLS certificate. - The web server does not run under a supervisor that would restart it if it crashes. - Maintenance operations that take the server down are not announced in advance.

Then it is a red flag, that suggests the mirror is operated in a way that will cause recurring trouble. These red flags warrant removing the mirror permanently, using your judgment on a case-by-case basis. In which case, go to the "Removing a mirror completely" section below.

For context, see https://gitlab.tails.boum.org/tails/blueprints/-/wikis/HTTP_mirror_pool#improve-ux-and-lower-maintenance-cost-2021
Else, go to "Disabling a mirror temporarily".

Adding a mirror¶

Check if the mirror satisfies the requirements.

If the error is about an URL or a DNS host, read on to disable it temporarily from the JSON pool. For example: 2. Add the mirror to the tails::mirrors list in data/domains/tails.net/common.yam.

Answer to the mirror administrator. For example:

Hi,

Your mirror seems to be configured correctly so we added it to
our pool of mirrors. You should start serving downloads right
away.

Thanks for setting this up and contributing to Tails!

Disabling a mirror temporarily¶

Update the tails::mirrors list in data/domains/tails.net/common.yam:
- Change the weight of the mirror to 0 to disable it temporarily.
- Add a note to document the failure, for example:
```
 2020-05-21: No route to host
```
  The "notes" fields has no strict format. I find it easier to document the latest failure first in the string.

Notify the admin. For example:

Hi,

Today, your mirror is $ERROR_DESCRIPTION:

    https://$MIRROR/tails/

Could you have a look?

Thanks for operating this mirror!

Keep track of the notification and ping the admin after a few weeks.

It's easy to miss one email, but let's not bother chasing those who don't answer twice.

Updating weights¶

To decrease the impact of unreliable mirrors in the pool, we give different weights to mirrors depending on their last failure:

We give a weight of:
- 10 to a few mirrors that haven't failed in the last 12 months and have a huge capacity.
- 5 to mirrors that haven't failed in the past 12 months.
- 2 to mirrors that haven't failed in the past 6 months.
- 2 to new mirrors.
- 1 to mirrors that have failed in the past 6 months.

We only keep notes of failures that happened less than 12 months ago.

We don't have a strict schedule to update these weights or remove notes on failures older than 12 months.

Removing a mirror completely¶

We remove mirrors and clean the list in data/domains/tails.net/common.yam proactively, from time to time, though we don't have a fixed schedule for that.

The mirrors that can be completely removed are either:

Mirrors that expose red flags documented above.

Mirrors that had problems at least twice in the last 6 months.

Mirrors that have been disabled, notified, and pinged once.

Mirrors that have seen the most failures in the past year or so.

Template message for mirrors that are repeatedly broken:

Hi,

Today your mirror is XXX:

 https://tails.XXX/

So I removed it from the pool for the time being.

I also wanted to let you know that your mirror has been the most
unreliable of the pool in the past year or so:


  - YYYY-MM-DD: XXX
  - etc.

We have a lot of mirrors already and right now we are more worried about
reliability and performance than about the raw number of mirrors.
We have some ideas to make our setup more
resilient to broken mirrors but we're not there yet. So right now, a
broken mirror means a broken download or a broken upgrade for users.

So unless, you think that the recent instability has a very good reason
to go away once and for all, maybe you can also consider retiring your
mirror, until our mirror pool management software can accommodate less
reliable mirrors. How would you feel about that?

It has been anyway very kind of you to host this mirror until now and we
are very grateful to your contribution to Tails!

Cheers,