Incident and emergency response: what to do in case of fire¶

This documentation is for sysadmins to figure out what to do when things go wrong. If you don't have the required accesses and haven't been trained for such situation, you might be better off just trying to wake up someone that can deal with them. See the support documentation instead.

This page lists situations that are not service-specific, generic issue that can happen on any server (or even on your home network). They are, in a sense, the default location for "pager playbooks" that would otherwise live in the service documentation.

Therefore, if the fault concerns a specific service, you will more likely find what you are looking for in the service listing.

Incident response procedures¶

If you are faced with an incident, try to follow this procedure:

1. File an incident in GitLab

   ... or at least start taking notes of what is happening if GitLab is down. Use the .gitlab/issue_templates/Incident.md template from the tpo/tpa/team repository in a Etherpad or local text file, in the worst case.

   When in doubt, file an incident. We prefer to over-document those and close spurious reports than under-document and wonder what happened after.

2. Get help

   By default, you're in Lead, and responsible for everything. Except in the simplest cases, you should delegate at least Communications and Planning.

   Find someone to help with Comms, at least, to shield you off from interrupts and help with updating the status page.

   Eventually, for "all hands on deck", major incidents, this will include delegating Lead itself or handing off Operations and keeping Lead.

   Don't hesitate in pinging your colleagues over text messages and out of band. Try to be considerate of time zones, but we're a team, and you're not expected to solve everything alone.

3. Work the problem!

   Depending on the role (and you accumulate all of those by default):

   - Operations

   Find and fix the problem! try to poke at systems and try to find
   a solution. don't hesitate to ask for help and keep others
   informed of your progress.
   
   ONLY ONE ROLE MODIFIES THE SYSTEMS DURING AN INCIDENT.
   
   This is like flying a plane: when you switch pilot, you say
   "the plane is yours", and the other pilot confirms and says "the
   plane is mine".
   
   Use `tmux` to allow others to see what you're doing, and survive
   outages.
   

   - Communications

   Talk internally and possibly externally about this issue.
   
   - stay in close contact with operations to know what's going on
   - relay and summarize information about the outage by instant
     messaging / IRC / Matrix or email
   - update the [status site](../service/status.md) as soon as possible, and keep it
     up to date
   - for major incidents, consider involving the Communications
     Director and social media outlets
   - avoid talking to the media, direct to the communications
     director
   

   - Planning

   You're the note taker and take care of assisting operations with
   long term issues. Your role is:
   
   - document progress in the issue (e.g. add timeline events for
     comments made by operations)
   - file new issues for possible issues / improvements that are
     uncovered
   - document how much systems diverged from norm
   - look for related issues and incidents in the past to figure
     out long term patterns and possible resolutions
   - keep track of new tasks that are piling up ("Next steps")
   - order dinner and breaks, you're the time keeper
   

   - Lead

   Tag! you're it! the buck stops here, and you might have all the
   above roles on your hands. if all roles have been delegated or
   rotated, your role is:
   
   - keep a cool head
   - act as a tie-breaker to make tough calls
   - remove roadblocks from other administrative boundaries
   - do *not* micro-manage everyone else
   - all of the above, which will be delegated as incidents grow
   
   Note that quite often, the best person for this role is the team
   lead, as they have the residual authority from their normal role
   to negotiate with others. But the role does not *necessarily*
   have to be the team lead.
   
   Most often than not, before delegation, the "star of the week"
   will be in Lead.
   

4. Post-mortem

   Once things have settled down, write and communicate a post-mortem

   This is typically done by the Planning role, but this can be shifted around.

   The post-mortem, if it becomes too big, can be moved to the status site or even a blog post. See this comment for examples of more exhaustive post-mortems.

Always remember that your priority is to fix things, not red tape, but it will be easier for long term recovery and coordination with others if you follow that procedure.

Rotating roles¶

The above roles are not set in stone, and can be rotated rather liberally. Roles can be delegated by the person holding it or rotated by someone jumping in the team. Generally, consent is sought, but don't wait for confirmation before taking up roles like communications and planning: it's better to shield your operations/lead from interruptions than doing nothing while waiting for confirmation.

Someone, for example, can walk into an incident and say "I think we need to have someone on communications, so I'm going to start doing that now, okay?"

Sometimes, however, it's possible a more forceful approach is needed. For example, "you've been going at this problem for four hours, I think you should let Arnie over here give it a try, okay?" Just like when flying a plane, however, you need to somehow manage to get that person off the wheel. Be gentle, but firm, and consider that everyone is under stress and doing their best: it's not a criticism of someone to be rotated out of a role.

Severity¶

We inherited a "severity" parameter from the GitLab incidents. Try to use it to describe the severity of an incident. This can be done in post-mortem.

GitLab, oddly, does not define what the 4 levels actually means and they do not seem to map to existing literature, which generally refer to either three (atlassian) or five (PagerDuty, Splunk, Zenduty). We are therefore using the top four levels from PagerDuty, ignoring the last one which is essentially "cosmetic, open an issue":

• S1, "Critical": total outage or major security affecting a all users, existential threat, examples:

  • total DNSSEC outage
  • local privilege escalation via sudo/systemd

• S2, "High": major outage affecting a majority of users

  • internal DNSSEC outage
  • GitLab ran out of disk space

• S3, "Medium": limited outage affecting only a subset of users, significant performance issues or hardware errors that will lead to a major or critical incident if unhandled

  • GitLab increased latency
  • pg backups filled up on bungei
  • failed disk on fsn-node-03

• S4, "Low": minor performance issues or warnings that could lead to problems for a small number of users, minor security issue, examples:

  • puppet not running on Ganeti cluster
  • container deployments bypass firewall rules

The distinction between those severity levels does not matter that much: make a guess. When in doubt, over-estimate. The distinction between "medium" and "low" is particularly unclear and irrelevant.

When in doubt, pick "medium" or nothing, and handle it in post-mortem.

An incident severity can of course be changed at any time as we learn more information about the issue, even in post-mortem or after an incident is closed.

Note that there is an overlap between the incident severity and the Support levels defined in TPA-RFC-2. Consider all code "red" and "yellow" to be incidents, and everything else to be issues.

More background¶

Keep in mind this advice from Google SRE book:

Best Practices for Incident Management¶

• Prioritize. Stop the bleeding, restore service, and preserve the evidence for root-causing.

• Prepare. Develop and document your incident management procedures in advance, in consultation with incident participants.

• Trust. Give full autonomy within the assigned role to all incident participants.

• Introspect. Pay attention to your emotional state while responding to an incident. If you start to feel panicky or overwhelmed, solicit more support.

• Consider alternatives. Periodically consider your options and re-evaluate whether it still makes sense to continue what you’re doing or whether you should be taking another tack in incident response.

• Practice. Use the process routinely so it becomes second nature.

• Change it around. Were you incident commander last time? Take on a different role this time. Encourage every team member to acquire familiarity with each role.

See TPA-RFC-91 for a discussion on this procedure and the roles.

Specific situations¶

Server down¶

Note that the HostDown alerts uses this as a playbook. But that alert currently uses the node exporter up metric to determine if a host is up, and it's possible that the node exporter is down while the host is still up. See tpo/tpa/team#42560 for an example of such a failure mode.

If a server is reported as non-responsive, this situation can be caused by:

1. a network outage at our provider
   • sometimes the network outage can be happening between two of our providers so make sure to test network reachability from more than one place on the internet.
2. RAM and swap being full
3. the host being offline or crashed

You can first check if it is actually reachable over the network:

ping -4 -c 10 server.torproject.org
ping -6 -c 10 server.torproject.org
ssh server.torproject.org

If it does respond at least from one point on the internet, you can try to diagnose the issue by looking at prometheus and/or Grafana and analyse what, exactly is going on. If you're lucky enough to have SSH access, you can dive deeper in the logs and systemd unit status: for example it might just be that the node exporter has crashed.

If the host does not respond, you should see if it's a virtual machine, and in this case, which server is hosting it. This information is available in service/ldap (or the web interface, under the physicalHost field). Then login to that server to diagnose this issue. If the physical host is a ganeti node, you can use the serial console and if it's not a ganeti node, you can try to access the console on the hosting provider's web site.

Once you have access to the console, look out for signs of errors like OOM-Kill, disk failures, kernel panics, network-related errors. If you're still able to login and investigate, you might be able to bring the machine back online. Otherwise, look in subsections below for how to perform hard resets.

If the physical host is not responding or is empty (in which case it is a physical host), you need to file a ticket with the upstream provider. That information is available in LDAP, under the physicalHost field.

Check if the parent server is online. If you still can't figure out which server that is, use traceroute or mtr to see the hops to the server. Normally, you should see a reverse DNS matching one of our point of presence. This will also show you whether or not the upstream routers are responsive. This is an example of a healthy trace to fsn-node-01, hosted at Hetzner robot, as seen from the other cluster in Dallas:

root@ssh-dal-01:~# mtr -c 10 -w fsn-node-01.torproject.org
Start: 2024-06-19T18:40:03+0000
HOST: ssh-dal-01                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- gw-01.gnt-dal-01.torproject.org  0.0%    10    0.5   4.2   0.4  35.2  10.9
  2.|-- e0-7.switch3.dal2.he.net        90.0%    10    1.7   1.7   1.7   1.7   0.0
  3.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
  6.|-- port-channel9.core2.par3.he.net  0.0%    10  103.5 105.5 102.3 126.5   7.5
  7.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
  8.|-- hetzner-online.par.franceix.net 10.0%    10  102.0 102.0 101.9 102.2   0.1
  9.|-- core12.nbg1.hetzner.com          0.0%    10  120.6 121.4 120.4 125.5   1.6
 10.|-- core22.fsn1.hetzner.com          0.0%    10  122.9 123.5 122.7 126.2   1.3
 11.|-- 2a01:4f8:0:3::5fe                0.0%    10  122.8 122.8 122.7 123.0   0.1
 12.|-- fsn-node-01.torproject.org       0.0%    10  123.1 123.1 122.8 124.0   0.4

In the above, you can see when the packets leave the continent from Dallas (hop 2) to land in Paris (hop 6), although the other hops in the middle are not responding and therefore hidden.

Here's a healthy trace to a hetzner-hel1-01, hosted in Hetzner cloud:

root@ssh-dal-01:~# mtr -c 10 -w hetzner-hel1-01.torproject.org
Start: 2024-06-19T18:41:22+0000
HOST: ssh-dal-01                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- gw-01.gnt-dal-01.torproject.org  0.0%    10    1.0   0.6   0.4   1.0   0.2
  2.|-- e0-7.switch3.dal2.he.net        70.0%    10    1.2   1.2   1.1   1.3   0.1
  3.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
  6.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
  7.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
  8.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
  9.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
 10.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
 11.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
 12.|-- 2a03:5f80:4:2::236:87            0.0%    10  130.2 130.9 129.7 138.9   2.8
 13.|-- core32.hel1.hetzner.com          0.0%    10  129.8 129.9 129.7 130.1   0.1
 14.|-- spine16.cloud1.hel1.hetzner.com  0.0%    10  128.5 130.6 128.4 145.2   5.3
 15.|-- spine2.cloud1.hel1.hetzner.com   0.0%    10  129.4 129.6 129.1 131.4   0.7
 16.|-- ???                             100.0    10    0.0   0.0   0.0   0.0   0.0
 17.|-- 12995.your-cloud.host            0.0%    10  128.6 128.7 128.5 129.1   0.2
 18.|-- hetzner-hel1-01.torproject.org   0.0%    10  130.7 131.1 130.3 135.4   1.5

What follows are per-provider instructions:

Hetzner robot (physical servers)¶

If you're not sure yet whether it's the server or Hetzner, you can use location-specific Hetzner targets:

ash.icmp.hetzner.com
fsn.icmp.hetzner.com
hel.icmp.hetzner.com
nbg.icmp.hetzner.com

... and so on.

If all fails, you can try to reset or reboot the server remotely:

1. Visit the Heztner Robot server page (password in tor-passwords/hosts-extra-info)
2. Select the right server (hostname is the second column)
3. Select the "reset" tab
4. Select the "Execute an automatic hardware reset" radio button and hit "Send". This is equivalent to hitting the "reset" button on a computer.
5. Wait for the server to return for a "few" (2? 5? 10? 20?) minutes, depending on how hopeful you are this simple procedure will work.
6. If that fails, Select the "Order a manual hardware reset" option and hit "Send". This will send an actual human to attend the server and see if they can bring it back online.

If all else fails, Select the "Support" tab and open a support request.

DO NOT file a ticket with support@hetzner.com. That email address is notoriously slow to get an answer from. See incident 40432 for a 3+ days delay.

Hetzner Cloud (virtual servers)¶

1. Visit the Hetzner Cloud console (password in tor-passwords/hosts-extra-info)
2. Select the project (usually "default")
3. Select the affected server
4. Open the console (the >_ sign on the top right), and see if there are any error messages and/or if you can login there (using the root password in tor-passwords/hosts)
5. If that fails, attempt a "Power cycle" in the "Power" tab (on the left)
6. If that fails, you can also try to boot a rescue system by selecting "Enable Rescue & Power Cycle" in the "Rescue" tab

If all else fails, create a support request. The support menu is in the "Person" menu on the top right of the page.

DO NOT file a ticket with support@hetzner.com. That email address is notoriously slow to get an answer from. See incident 40432 for a 3+ days delay.

Cymru¶

Open a ticket by writing support@cymru.com.

Sunet / safespring¶

TBD

Intermittent problems¶

If you have an intermittent problem that takes a while to manifest itself, you can increase the count. If that takes too long, you can enable "flood" mode which will decrease the interval between packets, waiting less between each failing probe or sending as soon as a reply is received, up to a certain rate.

Here is, for example, a successful 1000 packet ping executed in 100ms:

root@tb-build-03:~# ping -f -c 1000 dal-node-01.torproject.org
PING dal-node-01.torproject.org(2620:7:6002:0:3eec:efff:fed5:6b2a) 56 data bytes

--- dal-node-01.torproject.org ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 101ms
rtt min/avg/max/mdev = 0.075/0.086/0.211/0.012 ms, ipg/ewma 0.101/0.095 ms

And here is a failing ping aborted after 14 seconds:

root@tb-build-03:~# ping -f -c 1000 maven.mozilla.org
PING maven.mozilla.org(2600:9000:24f8:fa00:1b:afe8:4000:93a1) 56 data bytes
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C
--- maven.mozilla.org ping statistics ---
878 packets transmitted, 0 received, 100% packet loss, time 14031ms

(See tpo/tpa/team#41654 for a discussion and further analysis of that specific issue.)

MTR can help diagnose issues in this case. Vary parameters like IPv6 (-6) or TCP (--tcp). In the above case, the problem could be reproduced with mtr --tcp -6 -c 10 -w maven.mozilla.org.

Tools like curl can also be useful for quick diagnostics, but note that it supports the happy eyeballs standard so it might hide (e.g. IPv6) issues that might otherwise be affecting other clients.

Unexpected reboot¶

If a host reboots without a manual intervention, there might be different causes for the reboot to happen. Identifying exactly what happened after the fact can be challenging or even in some cases impossible since logs might not have been updated with information about the issues.

But in some cases the logs do have some information. Some things that can be investigated:

• syslog. look particularly for disk errors, OOM kill messages close to the reboot, kernel oops messages
• dmesg from previous boots, e.g. journaltcl -k -b -1, or see journalctl --list-boots for a list of boot IDs available
• smartctl -t long and smartctl -A / nvme [device-self-test|self-test-log] on all devices
• /proc/mdadm and /proc/drbd: make sure that replication is still all right

Also note that it's possible this is a spurious warning, or that a host took longer than expected to reboot. Normally, our Fabric reboot procedures issue a silence for the monitoring system to ignore those warnings. It's possible those delays are not appropriate for this host, for example, and might need to be tweaked upwards.

Network-level attacks¶

This section should guide you through network availability issues.

Confirming network-level attacks with Grafana¶

In case of degraded service availability over the network, it's a good idea to start by looking at metrics in Grafana. Denial of service attacks against a service over the network will often cause a noticeable bump in network activity, both in terms of ingress and egress traffic.

The traffic per class dashboard is a good place to start.

Finding traffic source with iftop¶

Once you have found there is indeed a spike of traffic, you should try to figure out what it consists of exactly.

A useful tool to investigate this is iftop, which displays network activity in realtime via the console. Here are some useful keyboard shortcuts when using it:

• n toggle DNS resolution
• D toggle destination port
• T toggle cumulative totals
• o freeze current order
• P pause display

In addition, the -f command-line argument can be used to filter network activity. For example, use iftop -f 'port 443' to only monitor HTTPS network traffic.

Firewall blocking¶

If you are sure that a specific $IP is mounting a Denial of Service attack on a server, you can block it with:

iptables -I INPUT -s $IP -j DROP

$IP can also be a network in CIDR notation, e.g. the following drops a whole Google /16 from the host:

iptables -I INPUT -s 74.125.0.0/16 -j DROP

Note that the above inserts (-I) a rule into the rule chain, which puts it before other rules. This is most likely what you want, as it's often possible there's an already existing rule that will allow the traffic through, making a rule appended (-A) to the chain ineffective.

This only blocks one network or host, and quite brutally, at the network level. From a user's perspective, it will look like an outage. A gentler way way is to use -j REJECT to actually send a reset packet to let the user know they're blocked.

See also our nftables documentation.

Note that those changes are gone after reboot or firewall reloads, for permanent blocking, see below.

Server blocking¶

An even "gentler" approach is to block clients at the server level. That way the client application can provide feedback to the user that the connection has been denied, more clearly. Typically, this is done with a web server level block list.

We don't have a uniform way to do this right now. In profile::nginx, there's a blocked_hosts list that can be used to add CIDR entries which are passed to the Nginx deny directive. Typically, you would define an entry in Hiera with something like this (example from data/roles/gitlab.yaml):

profile::nginx::blocked_hosts:
  # alibaba, tpo/tpa/team#42152

  - "47.74.0.0/15"

For Apache servers, it's even less standardized. A couple servers (currently donate and crm) have a blocklist.txt file that's used in a RewriteMap to deny individual IP addresses.

Extracting IP range lists¶

A command like this will extract the IP addresses from a webserver log file and group them by number of hits:

awk '{print $1}' /var/log/nginx/gitlab_access.log | grep -v '0.0.0.0' | sort | uniq -c | sort -n

This assumes log redaction has been disabled on the virtual host, of course, which can be done in emergencies like this. The most frequent hosts will show up first.

You can lookup which netblock the relevant IP addresses belong to a command like ip-info (part of the libnet-abuse-utils-perl Debian package) or asn (part of the asn package). Or this can be done by asking the asn.cymru.com service, with, for example:

nc whois.cymru.com 43 <<EOF
begin
verbose
216.90.108.31
192.0.2.1
198.51.100.0/24
203.0.113.42
end
EOF

This can be used to group IP addresses by netblock and AS number, roughly. A much more sophisticated approach is the asncounter project developed by anarcat, which allows AS and CIDR-level counting and can be used to establish a set of networks or entire ASNs to block.

The asncounter(1) manual page has detailed examples for this. That tool has been accepted in Debian unstable as of 2025-05-28 and should slowly make its way down to stable. An old version was backported to Debian 13 "trixie-backports".

The profile::prometheus::asncounter_exporter Puppet class can be used to collect per ASN and prefix metrics. The following queries are then useful:

sum(rate(asncounter_asn_total[5m])) by (asn, name)
sum(rate(asncounter_prefix_total[5m])) by (cidr, asn, name)

Web server down¶

Apache web server diagnostics¶

If you get an alert like ApacheDown, that is:

Apache web server down on test.example.com

It means the apache exporter cannot contact the local web server over its control address http://localhost/server-status/?auto. First, confirm whether this is a problem with the exporter or the entire service, by checking the main service on this host to see if users are affected. If that's the case, prioritize that.

It's possible, for example, that the webserver has crashed for some reason. The best way to figure that out is to check the service status with:

service apache2 status

You should see something like this if the server is running correctly:

● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-09-10 14:56:49 UTC; 1 day 5h ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 475367 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
   Main PID: 338774 (apache2)
      Tasks: 53 (limit: 4653)
     Memory: 28.6M
        CPU: 11min 30.297s
     CGroup: /system.slice/apache2.service
             ├─338774 /usr/sbin/apache2 -k start
             └─475411 /usr/sbin/apache2 -k start

Sep 10 17:51:50 donate-01 systemd[1]: Reloading apache2.service - The Apache HTTP Server...
Sep 10 17:51:50 donate-01 systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
Sep 10 19:53:00 donate-01 systemd[1]: Reloading apache2.service - The Apache HTTP Server...
Sep 10 19:53:00 donate-01 systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
Sep 11 00:00:01 donate-01 systemd[1]: Reloading apache2.service - The Apache HTTP Server...
Sep 11 00:00:01 donate-01 systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
Sep 11 01:29:29 donate-01 systemd[1]: Reloading apache2.service - The Apache HTTP Server...
Sep 11 01:29:29 donate-01 systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
Sep 11 19:50:51 donate-01 systemd[1]: Reloading apache2.service - The Apache HTTP Server...
Sep 11 19:50:51 donate-01 systemd[1]: Reloaded apache2.service - The Apache HTTP Server.

With the first dot (●) in green and the line Active saying active (running). If it isn't, the logs should show why it failed to start.

It's possible you don't see the right logs in there if the service is stuck in a restart loop. In this case, that use this command instead to see the service logs:

journalctl -b -u apache2

That shows the logs for the server from the last boot.

If the main service is online and it's only the exporter having trouble, try to reproduce the issue with curl from the affected server, for example:

root@test.example.com:~# curl http://localhost/server-status/?auto

Normally, this should work, but it's possible Apache is misconfigured and doesn't listen to localhost for some reason. Look at the apache2ctl -S output, and the rest of the Apache configuration in /etc/apache2, particularly the Ports and Listen directives.

See also the Apache exporter scraping failed instructions in the Prometheus documentation, a related alert.

Filesystem set to readonly¶

If a filesystem is switched to readonly, it prevents any process from writing to the concerned disk, which can have consequences of differing magnitude depending on which volume is readonly.

If Linux automatically changes a filesystem to readonly, it usually indicates that some serious issues were detected with the disk or filesystem. Those can be:

• physical drive errors
• bad sectors or other detected ongoing data corruption
• hard drive driver errors
• filesystem corruption

Look out for disk- or filesystem-related errors in:

• syslog
• dmesg
• physical console (e.g. IMPI console)

In some cases with ext4, running fsck can fix issues. However, watch out for files disappearing or being moved to lost+found if the filesystem encounters serious enough inconsistencies.

If the hard disk seems to be showing signs of breakage. Usually that disk will get ejected from the RAID array without blocking the filesystem. However if disk breakage did impact the filesystem consistency and caused it to switch to readonly, migrate the data away from that drive ASAP for example by moving the instance to its secondary node or by rsync'ing it to another machine.

In such a case, you'll also want to review what other instances are currently using the same drive and possibly move all of those instances as well before replacing the drive.

Disk is full or nearly full¶

When a disk is filled up to 100% of its capacity, some processes can have issues with continuing to work normally. For example PostgreSQL will purposefully exit when that happens in order to avoid the risk of data corruption. MySQL is not so graceful and it can end up with data corruption in some of its databases.

The first step is to check how long you have. For this, a good tool is the Grafana disk usage dashboard. Select the affected instance, and look at the "change rate" panel, it should show you how much time is left per partition.

To clear up this situation, there are two approaches that can be used in succession:

• find what's using disk space and clear out some files
• grow the disk

The first thing that should be attempted is to identify where disk space is used and remove some big files that occupy too much space. For example, if the root partition is full, this will show you what is taking up space:

ncdu -x /

Examples¶

Maybe the syslog grew to ridiculous sizes? Try:

logrotate -f /etc/logrotate.d/syslog-ng

Maybe some users have huge DB dumps laying around in their home directory. After confirming that those files can be deleted:

rm /home/flagada/huge_dump.sql

Maybe the systemd journal has grown too big. This will keep only 500MB:

journalctl --vacuum-size=500M

If in the cleanup phase you can't identify files that can be removed, you'll need to grow the disk. See how to grow disks with ganeti.

Note that it's possible a suddenly growing disk might be a symptom of a larger problem, for example bots crawling a website abusively or an attacker running a denial of service attack. This warrants further (and more complex) investigation, of course, but can be delegated to after the disk usage alert has been handled.

/boot is too small¶

With the addition of non-free modules, the kernel images have become fairly large and the default size of partition that the debian installer suggests has become too small.

If a machine is out of space on /boot, it will impact and block upgrades when a new kernel is installed.

We can start by removing any old kernel image that's not part of the current release, if there's any still around. You can inspect the current kernel images with:

dpkg -l | grep linux-image

If you can't remove enough kernel images and accompanying packages, you can then implement another trick to get rid of some kernel modules.

Keep in mind that the following trick can cause some surprising issues upon boot in some situations. One important example is if disks for the OS are plucked out of a machine and implanted in another -- in that case the required kernel modules might change and so some of the modules might be missing: in such a case you may need to boot to a live image, chroot to the OS partition and run update-initramfs in order to update the included modules.

To limit the modules included with each kernel, edit the file /etc/initramfs-tools/initramfs.conf and change the line starting with MODULES= to the following:

MODULES=dep

Service-specific procedures¶

• PostgreSQL backups filling up
• MinIO server filling up
• Prometheus running out of disk space

Failed systemd units¶

To check out what's happening with failed systemd units on a host:

systemctl --failed

You can, of course, run this check on all servers with Cumin:

cumin '*' 'systemctl --failed'

If you need further information you can dive into the logs of the units reported by the command above:

journalctl -xeu failed-unit.service

It's also possible the SystemdFailedUnits alert fires but the unit has actually recovered, or that the alert is flapping, in which case you will not find the alert in the systemctl --failed listing, but you can list the failures with:

journalctl JOB_RESULT=failed

Those are log entries from the systemd process that look like this:

Aug 31 22:19:05 crm-int-01 systemd[1]: Failed to start civicron@prod.service - "Run CiviCRM cron for prod".

In this case, you can look at the civicron@prod.service unit logs with:

journalctl -u civicron@prod.service -b

... then search for Failed to start in the pager (with /).

Service-specific instructions for failed systemd units:

• tails-update-time-based-apt-snapshots-*: Tails time-based APT snapshots systemd service unit failures

Host clock desynchronized¶

Moved to the time documentation.

Resource exhaustion investigations¶

When running out of sockets or file descriptors or anything, really, one sometimes can't use normal tools to diagnose problems.

In tpo/tpa/team#42560, for example, the ss command would take about 12 seconds to return. It was slow enough that, by the time it completed, the displayed data was completely stale and useless.

Using BPF resolves a lot of those issues because it bypasses normal procedures and uses in-kernel trace points to report back. A good start is the bpftrace command:

apt install bpftrace

You can list the possible traces with the -l argument:

bpftrace -l

In the case of UDP tracing, we wanted to know which processes were using the bind system call, so we looked for that:

bpftrace -l | grep syscall | grep bind

and then traced that syscall, for 10 seconds, with:

bpftrace -c 'sleep 10' -e 'tracepoint:syscalls:sys_enter_bind { @[comm] = count(); }'

Breaking that up, this:

1. runs the sleep 10 command and terminates after, which gives us a time limit for collection (otherwise use control c to abort

2. sets up a BPF hook to execute on the tracepoint:syscalls:sys_enter_bind trace point

3. count() the number of calls based on the command (@[comm]) name

This showed us the following:

root@metricsdb-03:~# bpftrace -c 'sleep 10' -e 'tracepoint:syscalls:sys_enter_bind { @[comm] = count(); }'
Attaching 1 probe...


@[BgSchPool]: 1
@[pasta.avx2]: 1748
@[tokio-rt-worker]: 3081
@[unbound]: 24093

This was showing that the pasta.avx2, tokio-rt-worker, and unbound processes were opening a lot of sockets. From there, we were able to resolve the issue by stopping unbound, and eventually the Network Health team looked into the root issue with tokio-rt-worker.

Failed Power Supply¶

On physical machines, if we get warned of a PSU failure, for example through a PowerSupplyOffline alert, some actions are required:

1. If no issue exists regarding the power supply failure for that machine, create a new issue in gitlab.
2. Take a look at the history of the state of the power supply. Here we look specifically at prometheus-03.tpo since it holds metrics data for longer. Look at least at the last full year to see if the same power supply showed issues more than once.
   • Note that the metric in the graph linked above, ipmi_sensor_state, can take on three values. 0 means all is well and anything above that is problematic.
3. If this is the first time that issues are found, schedule a visit to the data center and if it's possible to identify which one is at fault -- for example if one of them has its LEDs off -- yank (take out) the faulty power supply and plug it back in. This will reset it and might take it back online.
   • Note: while the power supply is out of its server slot you will want to identify its mark and model for example by taking a picture of its label.
4. If the above did not work or it at step 2 you saw that errors had already happened with the same power supply, then order a new power supply to replace the faulty one.

Support policies¶

Please see TPA-RFC-2: support.