Git¶

TPA uses Git in several places in its infra. Several services are managed via repos hosted in GitLab, but some services are managed by repos stored directly in the target systems, such as Puppet, LDAP, DNS, TLS, and probably others.

This page contains documentation about repositories primarily hosted in target systems (even though they may also be mirrored in GitLab for convenience).

Commit signature verification¶

In order to resist tampering attempts such as GitLab compromise, some key repositories are configured to verify commit signatures before accepting ref updates. For that, TPA uses sequoia-git to authenticate operations against certificates and permissions stored in a centralized OpenPGP policy file. See TPA-RFC-90: Signed commits for the initial proposal.

Terminology¶

Throughout this section, we use the term "certificate" to refer to OpenPGP Transferable Public Keys (see section 11.1 of RFC 4880).

sequoia-git basics¶

In order to authenticate changes in a Git repository, sequoia-git uses two pieces of information:

• an OpenPGP policy file, containing authorized certificates and a list of permissions for each certificate, and
• a "trust-root", which is the ID of a commit that is considered trusted.

With these, sequoia-git goes through commit by commit checking whether the signature is valid and authorized to perform operations.

By default, sequoia-git uses the openpgp-policy.toml file in the root of the repo being checked, but a path to an external policy file can be passed instead. In TPA, we do the former on the client side and the latter on the server side, as we'll see in the next section.

The TPA setup¶

In TPA we use one OpenPGP policy file to authenticate changes for all our repositories, namely the [openpgp-policy.toml][] file in the root of the Puppet repository. Using one centralized file allows for updating certificates and permissions in only one place and have it deployed to the relevant places.

For authenticating changes on the server-side:

• the TPA OpenPGP policy file is deployed to /etc/openpgp-policy/policies/tpa.toml,
• trust-roots for the Puppet repos (stored in hiera data for the puppetserver role in the Puppet repo) are deployed to /etc/openpgp-policy/gitconfig/${REPO}.conf, and
• per-repo Git hooks use the above info to authenticate changes.

On the client-side:

• we use the TPA OpenPGP policy file in the root of the Puppet repo,
• trust-roots are stored in the [.mrconfig][] file in tpo/tpa/repos> and set as Git configs in the relevant repos by mr update (see doc on [repos.git][]), and
• per-repo Git hooks use the above info to authenticate changes.

Note: When the trust-root for a repository changes, it needs to be updated in the hiera data for the puppetserver role and/or the [.mrconfig][] file, depending on whether it's supposed to be authenticated on server and/or client side.

Authentication on the client-side¶

Assuming you've setup [repos.git][] and ran mr update there (see The TPA setup above), authentication should happen automatically after every Git checkout, commit and merge. Because we use one OpenPGP policy file and per-repo trust-roots, the simplest way to re-run the verification is to use the hooks we maintain, for example in tor-puppet, tor-puppet-hiera-enc or account-keyring:

.githooks/post-checkout

If you have the trust-root and path to the OpenPGP policy file, you can also use params to pass them to sq-git log:

sq-git log --trust-root COMMIT_ID --policy-file POLICY_FILE

Handling authentication errors¶

Some authentication errors output by sq-git can be misleading, as they may be referring to specific paths that can't be authenticated while there still exists one authenticated path, which is enough to authenticate the changes. For example:

$ ./.githooks/post-checkout
Authenticating 51ed4234506a3fb60539001352fb0f75bfc7238a with a3827ef189da2fddb30ae18cafab743443341086:
  Error: Commit 51ed4234506a3fb60539001352fb0f75bfc7238a is not signed
  Uninstall ruby-net-dns on isoworkers

  Hint: when using an external policy, do

git show 51ed4234506a3fb60539001352fb0f75bfc7238a

  and verify that the commit is good.  If satisfied, do

sq-git policy goodlist --policy-file openpgp-policy.toml 51ed4234506a3fb60539001352fb0f75bfc7238a
Authenticating f68a6bcbe44645e9be8d6a2de7174e252d1c03bd with 0b4df50dc2026c966436d3b8dd0178e842b8b74c:
  Error: Commit f68a6bcbe44645e9be8d6a2de7174e252d1c03bd is not signed
  Make formatting consistent with shfmt
Authenticating 0b4df50dc2026c966436d3b8dd0178e842b8b74c with 8c11cbd2b7fe5a78f41ecf39f3aed0f823d9cc60:
  Error: Commit 0b4df50dc2026c966436d3b8dd0178e842b8b74c is not signed
  Add pipefail guardrail
Authenticating 8c11cbd2b7fe5a78f41ecf39f3aed0f823d9cc60 with bcc5af351f5457c68d041fecad36a5242c1ccdf4:
  Error: Commit 8c11cbd2b7fe5a78f41ecf39f3aed0f823d9cc60 is not signed
  Make comment more accurate
Authenticating bcc5af351f5457c68d041fecad36a5242c1ccdf4 with a355e624043dd9b48b3d9e3e205f1a232b34b38a:
  Error: Commit bcc5af351f5457c68d041fecad36a5242c1ccdf4 is not signed
  Fix build breakage after aborting test suite at unfortunate time
Authenticating 8877eaac223b9f66ffeddf09c1800be44a7fd45b with 3030013cce02a5837f7d45b43fa4804c2c5e0a59:
  Error: Commit 8877eaac223b9f66ffeddf09c1800be44a7fd45b is not signed
  Install tshark on isoworkers
Authenticating 3122fe1c650c47ec1c175b430855779665b9c25b with 622f36ad369bd5e9a50e83b2512a337f3f2af4ed:
  Error: Commit 3122fe1c650c47ec1c175b430855779665b9c25b is not signed
  docker@gitlab-runner-3.lizard: allow using any Tor image under /tpo/
Authenticating 622f36ad369bd5e9a50e83b2512a337f3f2af4ed with de00ab11765133c3a24029b2d67cf07e80da5794:
  Error: Commit 622f36ad369bd5e9a50e83b2512a337f3f2af4ed is not signed
  dind@gitlab-runner-3.lizard: allow using TPA-maintained base images
Authenticating 552310ef0792d27bed904478f5629104462ebe29 with a4f94404d74f7d974258366fd16fe9eabf68d7a9:
  Error: Commit 552310ef0792d27bed904478f5629104462ebe29 is not signed
  Add missing word in comment
Authenticating a4f94404d74f7d974258366fd16fe9eabf68d7a9 with fe0a7b9287e7ac94e6c8e1ae287a980f8f34ebc9:
  Error: Commit a4f94404d74f7d974258366fd16fe9eabf68d7a9 is not signed
  Add modules needed for the build_IUKs Jenkins job
Verified that there is an authenticated path from the trust root
fe0a7b9287e7ac94e6c8e1ae287a980f8f34ebc9 to 2aa09722e472615c91ccea63b7707bf5c014048c.

The above is not a real authentication error. The reason we see several error messages is because in this case we merged a branch that contained unsigned commits. In this case, it is expected to see sq-git log output several authentication errors for specific commits.

What matters is the exit status of the sq-git log command and, when it succeeds, you should always see Verified that there is an authenticated path from the trust root followed by the commit IDs of the start and end of the verified path.

A real authentication error ends in something like this:

(...)
Error: 11 errors occurred while verifying the commits.  11 commits couldn't be authenticated.  Note: not all errors are fatal.  The first error is shown:

Caused by:
    0: Could not verify commits fe0a7b9287e7ac94e6c8e1ae287a980f8f34ebc9..c8a72d9f2079caa2cf8523f6695fbe163e898f51
    1: While verifying commit c8a72d9f2079caa2cf8523f6695fbe163e898f51
    2: Commit c8a72d9f2079caa2cf8523f6695fbe163e898f51 is not signed

If you see this, then something is wrong. In this case, a commit is not signed.

Another problem may be that a commit is signed with a certificate that is not allowed, for example:

(...)
Error: 11 errors occurred while verifying the commits.  11 commits couldn't be authenticated.  Note: not all errors are fatal.  The first error is shown:

Caused by:
    0: Could not verify commits fe0a7b9287e7ac94e6c8e1ae287a980f8f34ebc9..fafb98f0c7b15408d1721e4518aba8f000ce51ef
    1: While verifying commit fafb98f0c7b15408d1721e4518aba8f000ce51ef
    2: Could not find a certificate with B26CCBBF078BFD694E6D463595DCEA6A53ED68B8 in the policy file

If you see an authentication error on the client side, identify and fix the cause of the failure before trying to push commits. If you do try to push unauthenticated commits, the server should refuse to accept them if it also fails to authenticate them.

Authentication on the Puppet Server¶

The Puppet repositories stored in the Puppet server are configured with hooks to verify authentication of the incoming commits before performing ref updates.

Puppet deploys in the Puppet server:

- the TPA OpenPGP policy file ([openpgp-policy.toml][]) to /etc/openpgp-policy/policies/tpa.toml, - global Git configuration containing per-repo policy file and trust-root configs to /etc/openpgp-policy/gitconfig/, and - Git update-hooks to the Puppet repositories that only allow ref updates if authentication is valid

See the [profile::openpgp_policy][] Puppet profile for the implementation.

With this, ref updates in the Puppet Git repos are only performed if all commits since the trust-root are signed with authorized certificates contained in the installed TPA OpenPGP policy file.

Certificate updates¶

While a certificate is still valid and has the [sign_commit][] capability, it's allowed to update any certificate contained in the openpgp-policy.toml file.

To update one or more certificates, first make sure you have up-to-date versions in your local store. One way to do that is by using sq to import the certificate from Tor's Web Key Directory:

sq network wkd search <ADDRESS>

Then use sq-git to update the OpenPGP policy file with certificates from your local store:

sq-git policy sync --disable-keyservers

Note that, if you don't use --disable-keyservers, expired subkeys may end up being included by a sync, and you may think that there are updates to the key when there really aren't. So it's better to just do as suggested above.

You can also edit the openpgp-policy.toml file manually and perform the needed changes.

Note that, because we use a centralized OpenPGP policy file, when permissions are removed for a certificate, we may need to update the trust-root, otherwise old commits may fail to be authenticated against the new policy file.

As of this writing, when a TPA person renews their keys, they need to do this in the puppet-control.git repository:

sq-git policy sync --disable-keyservers --policy-file=openpgp-policy.toml
git commit openpgp-policy.toml
git push

See also the OpePGP key renewal guide.

Expired certificates¶

If a certificate expires before it's been updated in the openpgp-policy.toml file, changes signed by that certificate will not be accepted, and you'll need to (1) ask another sysadmin with a valid certificate to perform the needed changes and (2) wait for or force deployment of the new file in the server.

See the above section for instructions on how to update the OpenPGP policy file.

Manual override¶

There may be extreme situations in which you need to override the authentication check, for example if your certificate expired and you're the only sysadmin in duty. In these cases, you can manually remove/update the corresponding Git hooks in the server and push the needed changes. If you do this, make sure to:

• update the trust root both in the hiera data for the puppetserver role and in tpo/tpa/repos>.
• instruct the other sysadmins to pull tpo/tpa/repos> and run mr update, so their local Git configs for trust-roots is automatically updated. If you don't do that, their local checks will start failing when they pull commits that can't be authenticated.

Other repositories¶

Even though we initially deployed this mechanism to the Puppet repositories only, the current implementation of the OpenPGP policy profile allows for configuration of the same setup for arbitrary repositories, which can be configured via hiera. See the hiera data for the puppetserver role for an example.

Setting trust-roots is mandatory, while policy files are optional. If no policy file is explicitly set, the Git hook will perform the authentication checks against the policy file in the root of the repository itself.

Handling external contributions¶

Some repositories are mirrored in GitLab and allow external contributors to push to non-protected branches and create Merge Requests (for example, tpo/tpa/puppet-control>). In these repos, there may exist a CI job that fails if commit signatures fail to verify.

In these cases, to merge the contribution, a merge commit should be signed by an authorized key to create an authenticated path from the trust root to tip of the branch. Such merging and signing must be done locally (in the authorized party's computer) and pushed to the canonical repository, from where it'll be mirrored back to GitLab.

Known issues¶

We reported the following issues that have been fixed upstream but will only land in Debian Forky:

• sq-git log fails when merging more than one unsigned or unauthorized commit
• sq-git log fails when checking a branch with more than one root
• output of sq-git log gets too long
• sq-git update-hook ignores the --policy-file option