Git repository integrity solutions¶
Advanced section
This section is a summary of the discussion in ticket tpo/tpa/gitlab#81. A broader discussion of the security issues with GitLab vs Gitolite and the choices made during that migration are available in Gitolite: security concerns.
Some developers expressed concerns about using GitLab as a canonical location for source code repositories, mainly because of the much broader attack surface GitLab provides, compared to the legacy, gitolite-based infrastructure, especially considering that the web application basically has write access to everything.
One solution to this problem is to use cryptographic signatures. We already use OpenPGP extensively in the Tor infrastructure, and it's well integrated in git, so it's an obvious candidate. But it's not necessarily obvious how OpenPGP would be used to sign code inside Tor, so this section provides a short review of existing solutions in that space.
Tip
An actual solution of this, inspired by Gerwitz below, has been deployed by TPA, see commit signature verification.
Guix: sign all commits¶
Guix uses OpenPGP to sign commits, using an approach that is basically:
- The repository contains a .guix-authorizations file that lists the OpenPGP key fingerprints of authorized committers.
- A commit is considered authentic if and only if it is signed by one of the keys listed in the .guix-authorizations file of each of its parents. This is the authorization invariant.
[...] Since .guix-authorizations is a regular file under version control, granting or revoking commit authorization does not require special support.
Note the big caveat:
It has one downside: it prevents pull-request-style workflows. Indeed, merging the branch of a contributor not listed in .guix-authorizations would break the authorization invariant. It’s a good tradeoff for Guix because our workflow relies on patches carved into stone tablets (patch tracker), but it’s not suitable for every project out there.
Also note there's a bootstrapping problem in their design:
Which commit do we pick as the first one where we can start verifying the authorization invariant?
They solve this with an out of band "channel introduction" mechanism which declares a good hash and a signing key.
This also requires a custom client. But it serves as a good example of an extreme approach (validate everything) one could take.
Note that GitLab Premium (non-free) has support for push rules and in particular a "Reject unsigned commits" rule.
Another implementation is SourceWare's gitsigur which verifies all commits (200 lines Python script), see also this discussion for a comparison. A similar project is Gentoo's update-02-gpg bash script.
Arista: sign all commits in Gerrit¶
Arista wrote a blog post called Commit Signing with Git at Enterprise Scale (archive) which takes a radically different approach.
- all OpenPGP keys are centrally managed (which solves the "web of trust" mess) in a Vault
- Gerrit is the gatekeeper: for patches to be merged, they must be signed by a trusted key
It is a rather obtuse system: because the final patches are rebased on top of the history, the git signatures are actually lost so they have a system to keep a reference to the Gerrit change id in the git history, which does have a copy of the OpenPGP signature.
Gerwitz: sign all commits or at least merge commits¶
Mike Gerwitz wrote an article in 2012 (which he warns is out of date) but which already correctly identified the issues with merge and rebase workflows. He argues there is a way to implement the desired workflow by signing merges: because maintainers are the one committing merge requests to the tree, they are in a position to actually sign the code provided by third-parties. Therefore it can be assume that if a merge commit is signed, then the code it imported is also signed.
The article also provides a crude checking script for such a scenario.
Obviously, in the case of GitLab, it would make the "merge" button less useful, as it would break the trust chain. But it's possible to merge "out of band" (in a local checkout) and push the result, which GitLab generally correctly detect as closing the merge request.
Note that sequoia-git implements this pattern, according to this.
Torvalds: signed tags¶
Linus Torvalds, the original author and maintainer of the Linux
kernel, simply signs the release tags. In an article called "what
does a pgp signature on a git commit prove?", Konstantin Ryabitsev
(the kernel.org sysadmin), provides a good primer on OpenPGP signing
in git. It also shows how to validate Linux releases by checking the
tag and argues this is sufficient to ensure trust.
Vick: git signatures AKA git notes¶
The git-signatures project, authored by Lance R. Vick, makes it possible to "attach an arbitrary number of GPG signatures to a given commit or tag.":
Git already supports commit signing. These tools are intended to compliment that support by allowing a code reviewer and/or release engineer attach their signatures as well.
Downside: third-party tool not distributed with git and not packaged in Debian.
The idea of using git-notes was also proposed by Owen Jacobsen.
Walters: extended validation tags¶
The git-evtag projects from Colin Walters tries to address the perceived vulnerability of the SHA-1 hash by implementing a new signing procedure for tags, based on SHA-512 and OpenPGP.
Ryabitsev: b4 and patch attestations¶
Konstantin Ryabitsev (the kernel.org sysadmin, again) proposed a new cryptographic scheme to sign patches in Linux, he called "patch attestation". The protocol is designed to survive mailing list transports, rebases and all sorts of mangling. It does not use GnuPG and is based on a Trust On First Use (TOFU) model.
The model is not without critics.
Update, 2021-06-04: there was another iteration of that concept, this time based on DKIM-like headers, with support for OpenPGP signatures but also "native" ed25519.
One key takeaway from this approach, which we could reuse, is the way public keys are stored. In patatt, the git repository itself holds the public keys:
On the other hand, within the context of git repositories, we already have a suitable mechanism for distributing developer public keys, which is the repository itself. Consider this:
- git is already decentralized and can be mirrored to multiple locations, avoiding any single points of failure
- all contents are already versioned and key additions/removals can be audited and “git blame’d”
- git commits themselves can be cryptographically signed, which allows a small subset of developers to act as “trusted introducers” to many other contributors (mimicking the “keysigning” process)
The idea of using git itself for keyring management was originally suggested by the did:git project, though we do not currently implement the proposed standard itself.
<https://github.com/dhuseby/did-git-spec/blob/master/did-git-spec.md>
It's unclear, however, why the latter spec wasn't reused. To be investigated.
Update, 2022-04-20: someone actually went through the trouble of auditing the transparency log, which is an interesting exercise in itself. The verifier source code is available, but probably too specific to Linux for our use case. Their notes are also interesting. This is also in the kernel documentation and the logs themselves are in this git repository.
Ryabitsev: Secure Scuttlebutt¶
A more exotic proposal is to use the Secure Scuttlebutt (SSB) protocol instead of emails to exchange (and also, implicitly) sign git commits. There is even a git-ssb implementation, although it's hard to see because it's been migrated to .... SSB!
Obviously, this is not quite practical and is shown only as a more radical example, as a stand-in for the other end of the decentralization spectrum.
Stelzer: ssh signatures¶
Fabian Stelzer made a pull request for git which was actually merged in October 2021 and therefore might make it to 2.34. The PR adds support for SSH signatures on top of the already existing OpenPGP and X.509 systems that git already supports.
It does not address the above issues of "which commits to sign" or "where to store keys", but it does allow users to drop the OpenPGP/GnuPG dependency if they so desire. Note that there may be compatibility issues with different OpenSSH releases, as the PR explicitly says:
I will add this feature in a follow up patch afterwards since the released 8.7 version has a broken ssh-keygen implementation which will break ssh signing completely.
We do not currently have plans to get rid of OpenPGP internally, but it's still nice to have options.
Lorenc: sigstore¶
Dan Lorenc, an engineer at Google, designed a tool that allows users to sign "artifacts". Typically, those are container images (e.g. cosign is named so because it signs "containers"), but anything can be signed.
It also works with a transparency log server called rekor. They run a public instance, but we could also run our own. It is currently unclear if we could have both, but it's apparently possible to run a "monitor" that would check the log for consistency.
There's also a system for signing binaries with ephemeral keys which seems counter-intuitive but actually works nicely for CI jobs.
Seems very promising, maintained by Google, RedHat, and supported by the Linux foundation. Complementary to in-toto and TUF. TUF is actually used to create the root keys which are controlled, at the time of writing, by:
- Bob Callaway (Google)
- Dan Lorenc (Google)
- Luke Hinds (RedHat)
- Marina Moore (NYU)
- Santiago Torres (Purdue)
Update: gitsign is specifically built to use this infrastructure for Git. GitHub and GitLab are currently lacking support for verifying those signatures. See tutorial.
Similar projects:
- SLSA, which has a well documented threat model
- Trillian (Google)
- sigsum, similar to sigstore, but more minimal
Sirish: gittuf¶
Aditya Sirish, a PhD student under TUF's Cappos is building gittuf a "security layer for Git repositories" which allows things like multiple signatures, key rotation and in-repository attestations of things like "CI ran green on this commit".
Designed to be backend agnostic, so should support GPG and sigstore, also includes in-toto attestations.
Other caveats¶
Also note that git has limited security guarantees regarding checksums, since it uses SHA-1, but that is about to change. Most Git implementations also have protections against collisions, see for example this article from GitHub.
There are, of course, a large number of usability (and some would say security) issues with OpenPGP (or, more specifically, the main implementation, GnuPG). There has even been security issues with signed Git commits, specifically.
So I would also be open to alternative signature verification schemes. Unfortunately, none of those are implemented in git, as far as I can tell.
There are, however, alternatives to GnuPG itself. This article from Saoirse Shipwreckt shows how to verify commits without GnuPG, for example. That still relies on OpenPGP keys of course...
... which brings us to the web of trust and key distribution problems. The OpenPGP community is in this problematic situation right now where the traditional key distribution mechanisms (the old keyserver network) has been under attack and is not as reliable as it should be. This brings the question of keyring management, but that is already being discussed in tpo/tpa/team#29671.
Finally, note that OpenPGP keys are not permanent: they can be revoked, or expired. Dealing with this problem has its specific set of solutions as well. GitHub marks signatures as verified for expired or revoked (but not compromised) keys, but has a special mouse-over showing exactly what's going on with that key, which seems like a good compromise.
Related¶
- gitid: easier identity management for git
- signed git pushes
- TUF: generic verification mechanism, used by Docker, no known Git implementation just yet (update: gittuf in pre-alpha as of dec 2023)
- SLSA: "security framework, a check-list of standards and controls to prevent tampering, improve integrity, and secure packages and infrastructure", built on top of in-toto
- jcat: used by fwupd
- git-signify: using signify, a non-OpenPGP alternative
- crev: Code REView system, used by Rust (and Cargo) to vet dependencies, delegates sharing signatures to git, but cryptographically signs them so should be resilient against a server compromise
- arch linux upstream tag verifications
- Linux kernel OpenPGP keys distribution repository
- sequoia authenticate commits - to be evaluated