Skip to content

Status

A "status" dashboard is a simple website that allows service admins to clearly and simply announce down times and recovery.

Note that this be considered part of the documentation system, but is documented separately.

The site is at https://torprojectstatus.org/ and the source at https://gitlab.torproject.org/tpo/tpa/status-site/.

Tutorial

Local development environment

To install the development environment for the status site, you should have a copy of the Hugo static site generator and the git repository:

sudo apt install hugo git-pages-cli
git clone --recursive -b main https://gitlab.torproject.org/tpo/tpa/status-site.git
cd status-site

Note that git-pages-cli is only available in debian starting with the forky release. If you can't install it, there are instructions in the Manual Grebedoc deployment section for sending the site without that tool.

WARNING: the URL of the Git repository changed! It used to be hosted at GitLab, but is now hosted at Gitolite. The repository is mirrored to GitLab, but pushing there will not trigger build jobs.

Then you can start a local development server to preview the site with:

hugo serve --baseURL=http://localhost/
firefox http://localhost:1313/

The content can also be built in the public/ directory with, simply:

hugo

Creating new issues

Issues are stored in content/issues/. You can create a new issue with hugo new, for example:

hugo new issues/2021-02-03-testing-cstate-again.md

This create the file from a pre-filled template (called an archetype in Hugo) and put it in content/issues/2021-02-03-testing-cstate-again.md.

If you do not have hugo installed locally, you can also copy the template directly (from themes/cstate/archetypes/default.md), or copy an existing issue and use it as a template.

Otherwise the upstream guide on how to create issues is fairly thorough and should be followed.

In general, keep in mind that the date field is when the issue started, not when you posted the issue, see this feature request asking for an explicit "update" field.

Also note that you can add draft: true to the front-matter (the block on top) to keep the post from being published on the front page before it is ready.

Maintenance windows should be done with the announcement mechanism which is essentially to use informational: true and (optionally) pin: true.

Uploading site to the static mirror system

Uploading the site is automated by continuous integration. So you simply need to commit and push:

git commit -a -myolo
git push

Note that only the TPA group has access to the repository for now, but other users can request access as needed.

The GitLab repository is actually mirrored to a Codeberg repository where the build and deployment actually happens. A build also happens in GitLab, but merely as a test.

If GitLab is down, you should be able to push directly to the Codeberg repository.

If Codeberg is down, you should still be able to upload the site, which is actually hosted on Grebedoc, see the Manual Grebedoc deployment and other disaster recovery options below.

Keep in mind that this is a public website. You might want to talk with the comms@ people before publishing big or sensitive announcements.

How-to

Changing categories

cState relies on "systems" which live inside a "category" For example, the "v3 onion services" are in the "Tor network" category. Those are defined in the config.yml file, and each issue (in content/issues) refers to one or more "system" that is affected by it.

Theming

The logo lives in static/logo.png. Some colors are defined in config.yml, search for Colors throughout cState.

Pager playbook

No monitoring specific to this service exists.

Disaster recovery

It should be possible to deploy the static website anywhere that supports plain HTML, assuming you have a copy of the git repository.

The instructions in all of the subsections below assume you have a copy of the git repository.

Important: make sure you follow the installation instructions to also clone the submodules!

If the git repository is not available, you can start from scratch using the example repository as well, although we strongly advise against that. There should be at least one mirror available somewhere, in one of those locations, even if out of date:

From here on, it is assumed you have a copy of the git repository (or the example one).

Manual Grebedoc deployment

If Codeberg is down or too slow, deployments will fail.

You should still be able to post updates to the site, assuming you have a copy of the git repository, by using the git-pages endpoint of Grebedoc directly, with:

cd status-site
hugo
git-pages-cli https://torprojectstatus.org/ --upload-dir public/ --password="$(pass tor/dns/grebedoc.dev)" --server grebedoc.dev

This will respond with:

result: replaced
(archive)

This uses a secret stored as a hash of the domain and the secret in the _git-pages-challenge.torprojectstatus.org TXT record, which should already be populated and stored in the password manager.

If that password fails or has been lost, you can set a new one with:

printf "torprojectstatus.org $(pass tor/dns/grebedoc.dev)" | sha256sum

If you do not have the git-pages-cli (available from Debian 14 "forky" and later), you can also perform the equivalent with:

cd status-site
hugo
tar -C public/ -c -f - . | curl -X PUT -i \
  https://torprojectstatus.org/ \
  -H "Authorization: Pages $(pass tor/dns/grebedoc.dev)" \
  -H "Content-Type: application/x-tar" \
  --data-binary @-

In that case, curl should respond with the string:

HTTP/2 200 
alt-svc: h3=":443"; ma=2592000
content-type: text/plain; charset=utf-8
date: Tue, 02 Jun 2026 18:41:23 GMT
server: git-pages (rage4-chi1)
strict-transport-security: max-age=31536000
update-result: replaced
via: 1.1 Caddy
content-length: 10

(archive)

You can also compress the file with zstd with this:

tar --zstd -C public/ -c -f - . | curl -X PUT -i \
  https://torprojectstatus.org/ \
  -H "Authorization: Pages $(pass tor/dns/grebedoc.dev)" \
  -H "Content-Type: application/x-tar+zstd" \
  --data-binary @-

You might also get the header:

update-result: no-change

... if the site is unchanged, or:

update-result: replaced

... when there is a change. If you're starting from scratch on a new git-pages hosting, you will get:

update-result: created

Codeberg pages deployment

The above assumes Grebedoc is still operational. If it fails, you can still publish the website to Codeberg, as it has its own Git pages deployment.

First, you'll need to point the domain at codeberg.page, by replacing the ANAME entry with a codeberg.page entry.

Then replace the server line in the actions/git-pages entry, so it looks like this:

      - uses: actions/git-pages@v2
        with:
          site: https://torprojectstatus.org/
          token: ${{ forge.token }}
          source: public/
          server: codeberg.page

And push the update to Codeberg, which should run this action.

If that fails, follow the Manual Grebedoc deployment guide but replace any occurrence of grebedoc.dev with codeberg.page.

GitLab pages deployment

A site could also be deployed on another GitLab server with "GitLab pages" enabled. For example, if the repository is pushed to https://gitlab.com/, the GitLab CI/CD system there will automatically pick up the configuration and run it.

Unfortunately, due to the heavy customization we used to deploy the site to the static mirror system, the stock .gitlab-ci.yml file will likely not work on another system. An alternate .gitlab-ci-pages.yml file should be available in the Git repository and can be activated in the GitLab project in Settings -> CI/CD -> CI/CD configuration file.

That should give you a "test" GitLab pages site with a URL like:

https://user.gitlab.io/tpa-status/

To transfer the real site there, you need to go into the project's Settings -> Pages section and hit New Domain.

Enter status.torproject.org there, which will ask you to add an TXT record in the torproject.org zone.

Add the TXT record to domains.git/torproject.org, commit and push, then hit the "Retry verification" button in the GitLab interface.

Once the domain is verified, point the status.torproject.org domain to the new backend:

status CNAME user.gitlab.io

For example, in my case, it was:

status CNAME anarcat.gitlab.io

See also the upstream documentation for details.

Manual deployment to the static mirror system

If Grebedoc, Codeberg, GitLab are all down, you can upload the public/ folder anywhere you can host files.

The status site used to be under status.torproject.org in the static site system. You might be able to restore that service by re configuring a static component in Puppet (in modules/staticsync/data/common.yaml) to the server static-gitlab-shim.torproject.org and the directory /srv/static-gitlab-shim/status.torproject.org/. See the Adding a new component guide for details.

Then this rsync command should be enough:

rsync -rtP public/ static-gitlab-shim@static-gitlab-shim.torproject.org:/srv/static-gitlab-shim/status.torproject.org/public/

This might require adding your key to /etc/ssh/userkeys/static-gitlab-shim.more.

Then the new source material needs to be synchronized to the mirrors, with:

sudo -u mirroradm static-update-component status.torproject.org

This requires access to the mirroradm group, although typically the machine is only accessible to TPA anyways.

Don't forget to push the changes to the git repository, once that is available. It's important so that the next people can start from your changes:

git commit -a -myolo
git push

Netlify deployment

Those instructions are untested, but Netlify is a last resort option.

Upstream has instructions to deploy to Netlify, which, in our case, might be as simple as following this link and filling in those settings:

  • Build command: hugo
  • Publish directory: public
  • Add one build environment variable
    • Key: HUGO_VERSION
    • Value: 0.48 (or later)

Then, of course, DNS needs to be updated to point there.

Git pages deployment

The above assumes Grebedoc, Codeberg, GitLab, and our static mirror system are operational. If all of those fails, it should still be possible to host the site elsewhere.

You should be able to install git-pages on any virtual machine, but at that point, it might be easier to just deploy the source as a static website.

Reference

Installation

See the instructions on how to setup a local development environment and the design section for more information on how this is setup.

Codeberg mirror

A mirror of the site was setup on Codeberg. It was first created by "migrating" the repository (which, really, just clones the report and copies the description) under the torproject namespace namespace, as torproject/status-site.

Note:

We briefly considered an entirely different namespace for security reasons, but figured we didn't need such precaution yet. See the discussion about mirroring to Codeberg for details.

Then the "Actions unit" was enabled in the units section, issues and pull requests were disabled (and the wiki was already disabled). We checked that runners are available so we wouldn't have to run our own.

Then we added a Forgejo action so that the site would build and publish on push, following the pages action guide. At first, we used the built-in codeberg.page wildcard:

      - uses: actions/git-pages@v2
        with:
          site: https://${{ forge.event.repository.owner.username }}.codeberg.page/${{ forge.event.repository.name }}/
          token: ${{ forge.token }}
          source: public/

We picked the runner with the line:

runs-on: codeberg-tiny

This is important because without it, no runner will ever pick it up.

A test was done by pushing this directly to the Codeberg repository (instead of GitLab).

A similar configuration was done to also publish the site to Grebedoc, to compare each host's performance. Our assumption is that Grebedoc performs better. We also setup monitoring to monitor the GitLab pages deployment at gitlab.io.

(Ultimately, Grebedoc was kept.)

Then the repository mirroring was configured in the GitLab project, with the following settings:

  • URL: ssh://git@codeberg.org/torproject/status-site.git
  • Mirror direction: push
  • Authentication method: SSH public key
  • Username: git (likely ignored)
  • Keep divergent refs: unchecked (unchanged)
  • Mirror branches: Mirror all branches (unchanged)

This creates a secret key on the server and the public key can then be copied by clicking on the "copy" icon on the line.

That public key is then uploaded as a deploy key in the Codeberg repository.

Then the domains were registered at Mythic Beasts, and configured with the following:

@                      300 NS    ns1.mythic-beasts.com.
@                      300 NS    ns2.mythic-beasts.com.
; @                    300 ANAME grebedoc.dev.
@                      300 A     185.187.152.7                            ; generated
@                      300 AAAA  2a05:b0c4:1:0:0:0:0:3                    ; generated
_git-pages-challenge   300 TXT   "d600d9e8e2ba80f477c5f79bd131d082df6d5b2bc2f1804f2eff3344f7624850"
_git-pages-forge-allowlist   300 TXT   "https://codeberg.org/torproject/status-site.git"
_git-pages-repository   300 TXT   "https://codeberg.org/torproject/status-site.git"

... as per the Grebedoc directives. Essentially, we allow two methods to update the domain:

  1. Method A: the _git-pages-repository which points at the Codeberg repository, where it will fetch the output of the pages branch on a webhook update or _git-pages-forge-allowlist which is used to check the source of the git-pages action
  2. Method B: the _git-pages-challenge which embeds a secret that allows updating the page completely without Codeberg, it's the output of:
      printf "torprojectstatus.org $(pass tor/dns/grebedoc.dev)" | sha256sum

Once the records were in place, the following action was used instead of the above:

      - uses: actions/git-pages@v2
        with:
          # site: https://${{ forge.event.repository.owner.username }}.grebedoc.dev/${{ forge.event.repository.name }}/
          site: https://torprojectstatus.org/
          token: ${{ forge.token }}
          source: public/
          server: grebedoc.dev

A manual update should be done with the git-pages-cli command:

git-pages-cli http://torprojectstatus.org/ --upload-dir public/ --password="$(pass tor/dns/grebedoc.dev)" --server grebedoc.dev

Note the HTTP here (without TLS) which is necessary to bootstrap the site.

Upgrades

Upgrades to the software are performed by updating the cstate submodule.

Since November, the renovate-cron bot will pass through the project to make sure that submodule is up to date.

There's also a GitLab CI job (check-tags) which ensures the modules are up to date, by running the check-latest.sh script. The script should give you instructions on how to update to the latest submodule, for example with:

(cd themes/cstate && git checkout master && git pull && git reset --hard $latest_tag) && git commit -m"update cstate to $latest_tag" themes/cstate

You can also chase another specific ref by pointing $latest_tag at any commit from the upstream repository. If you have submitted a merge request upstream, you'll first need to change the url in the .gitmodules file, for example with:

git submodule set-url themes/cstate https://github.com/anarcat/cstate

And then do the above reset to follow the right commit.

Hugo itself is managed through the Debian packages provided as part of the bookworm container, and therefore benefit from the normal Debian support policies. Major Debian upgrades need to be manually performed in the .gitlab-ci.yml file and are not checked by renovate.

SLA

This service should be highly available. It should support failure from one or all point of presence: if all fail, it should be easy to deploy it to a third-party provider.

Design and architecture

The status site is part of the static mirror system and is built with cstate, which is a theme for the Hugo static site generator.

The site is managed in a git repository on the GitLab server and mirrors to a Codeberg repository over SSH which, in turn uses Forgejo actions, specifically the checkout and git-pages actions to build and publish the site.

The site is published to Grebedoc, an anycast-hosted service running git-pages that is backed by an S3-like object storage service (currently Wasabi).

Services

No service other than the above external services are required to run this service.

Queues

There are no queues or schedulers for that service, although renovate-cron will pass by the project to check for updates once in a while.

Interfaces

Authentication

Git mirroring from GitLab to mirrors is done over SSH.

The git-pages service has many authentication systems. We use, in particular, the "Forge Authorization (DNS allowlist)" which consists of a _git-pages-forge-allowlist record in DNS announcing the forge, and an ephemeral authentication token issued by Forgejo in the CI job that can be used to verify permissions of the token by git-pages.

We also use the "DNS Challenge" mechanism in that we have that checksum in DNS that allows for manual updates, as a fallback mechanism.

Implementation

Status is mostly written in Markdown, but the upstream code is written in Golang and its templating language.

Issues

File or search for issues in the status-site tracker.

Upstream issues can be found and filed in the GitHub issue tracker.

Users

TPA is the main maintainer of this service and therefore its most likely user, but the network health team are frequent users as well.

Naturally, any person interested in the Tor project and the health of the services is also a potential user.

Upstream

cState is a pretty collaborative and active upstream. It is seeing regular releases and is considered healthy, especially since most of the implementation is actually in hugo, another healthy project.

Codeberg is a friendly bunch, and do their best to run a good service. Like us, they are victims of crawlers and attackers that can make the service unreliable.

Monitoring and metrics

The custom domains behind this service are monitored by the custom script we have to monitor domain expiry. It's defined in the profile::prometheus::domain_exporter class, and runs as the tpa_domain_expiry expiry on monitoring servers, daily.

The sites are also monitored as external targets, as the blackbox_https_codeberg job defined in data/roles/monitoring.yaml.

Tests

New changes to the site are manually checked by browsing a rendered version of the site and clicking around.

This can be done on a local copy before even committing, or it can be done with a review site by pushing a branch and opening a merge request.

Logs

Grebedoc does not keep logs of visitors but keeps logs of site updates, for four weeks.

A history of deployments and past version of the code is of course available in the Git repository history and the GitLab job logs.

Backups

Does not need special backups: backed up as part of the regular git services.

Other documentation

Discussion

Overview

This project comes from two places:

  1. during the 2020 TPA user survey, some respondents suggested to document "down times of 1h or longer" and better communicate about service statuses

  2. separately, following a major outage in the Tor network due to a DDOS, the network team and network health teams asked for a dashboard to inform tor users about such problems in the future

This is therefore a project spanning multiple teams, with different stakeholders. The general idea is to have a site (say status.torproject.org) that simply shows users how things are going, in an easy to understand form.

This happened around 2021. Since then, in 2026, the site was moved to its own separate domain and registry, see tpo/tpa/team#42681.

Security and risk assessment

No security audit was performed of this service, but considering it only manages static content accessed by trusted users, its exposure is considered minimal.

It might be the target of denial of service attacks, but should survive an attack on torproject.org infrastructure, as long as attackers don't think of also targeting Grebedoc.

A compromise of the GitLab infrastructure would also naturally give access to the status site.

Finally, if an outage affects the alternate domain name (torprojectstatus.org) this site could suffer as well, but not on the main domain name.

Technical debt and next steps

The service is pretty healthy at this point.

Goals

In general, the goal is to provide a simple interface to provide users with status updates.

Must have

  • user-friendly: the public website must be easy to understand by the Tor wider community of users (not just TPI/TPA)
  • status updates and progress: "post status problem we know about so the world can learn if problems are known to the Tor team."
    • example: "[recent] v3 outage where we could have put out a small FAQ right away (go static HTML!) and then update the world as we figure out the problem but also expected return to normal."
  • multi-stakeholder: "easily editable by many of us namely likely the network health team and we could also have the network team to help out"
  • simple to deploy and use: pushing an update shouldn't require complex software or procedures. editing a text file, committing and pushing, or building with a single command and pushing the HTML, for example, is simple enough. installing a MySQL database and PHP server, for example, is not simple enough.
  • keep it simple
  • free-software based

Nice to have

  • deployment through GitLab (pages?), with contingency plans
  • separate TLD to thwart DNS-based attacks against torproject.org
  • same tool for multiple teams
  • per-team filtering
  • RSS feeds
  • integration with social media?
  • responsive design

Non-Goals

  • automation: updating the site is a manual process. no automatic reports of sensors/metrics or Nagios, as this tends to complicate the implementation and cause false positives

Approvals required

TPA, network team, network health team.

Proposed Solution

We're experimenting with cstate because it's the only static website generator with such a nice template out of the box that we could find.

Cost

Just research and development time. Hosting costs are negligible.

Alternatives considered

Those are the status dashboards we know about and that are still somewhat in active development:

* Cachet * PHP * MySQL database * demo site (test@test.com, test123) * responsive * not decentralized * no Nagios support * user-friendly * publicly accessible * fairly easy to use * aims for LDAP support * no Twitter, Identica, IRC or XMPP support for now * dropped RSS support * future of the project uncertain (4037, 3968) * cstate, Hugo-based static site generator, tag-based RSS feeds, easy setup on Netlify, GitLab CI integration, badges, read only API * Staytus * ruby * MySQL database * responsive * email notifications * mobile-friendly * not distributed * no Nagios integration * no Twitter notifications * user-friendly - seems to be even nicer than Cachet, as there are links to individual announcements and notifications * no LDAP support * MIT-licensed * similar performance problems than Cachet * vigil-server * tinystatus * uptime kuma more of a monitoring platform

Abandonware

Those were previously evaluated in a previous life but ended up being abandoned upstream:

* Overseer - used at Disqus.com, Python/Django, user-friendly/simple, administrator non-friendly, twitter integration, Apache2 license, development stopped, Disqus replaced it with Statuspage.io * Stashboard - used at Twilio, MIT license, demo, Twitter integration, REST API, abandon-ware, no authentication, no Unicode support, depends on Google App engine, requires daily updates * Baobab - previously used at Gandi, replaced with statuspage.io, Django based

Hacks

Those were discarded because they do not provide an "out of the box" experience:

* use Jenkins to run jobs that check a bunch of things and report a user-friendly status? * just use a social network account (e.g. Twitter) * "just use the wiki" * use Drupal ("there's a module for that") * roll our own with Lektor, e.g. using this template * using GitHub issues

example sites

* Amazon Service Health Dashboard * Disqus - based on statuspage.io * GitLab - based on status.io * Github - "Battle station fully operational", auto-refresh, twitter-connected, simple color coded (see this blog post for more details), not open-source (confirmed in personal email between GitHub support and anarcat on 2013-05-02) * Potager.org - ikiwiki based * Riseup.net - RSS feeds * Signal - simple, plain HTML page * sr.ht - cState * Twilio - email, slack, RSS subscriptions, lots of services shown * Wikimedia - based on proprietary nimsoft software, deprecated in favor of Grafana

Previous implementations

IRC bot

A similar service was ran by @weasel around 2014. It would bridge the status comments on IRC into a website, see this archived version and the source code, which is still available.

Jenkins jobs

The site used to be built with Jenkins jobs, from a git repository on the git server. This was setup this way because that is how every other static website was built back then.

This involved:

We also considered using GitLab CI for deployment but (a) GitLab pages was not yet setup and (b) it didn't integrate well with the static mirror system for now. See the broader discussion of the static site system improvements.

Both issues have now been fixed thanks to the static-shim service.

Static mirror system

Until June 2026, this was implement as the https://status.torproject.org site in the static mirror system. It was moved to Grebedoc, see ADR-0109 for details.