Skip to content

GitLab user guide

GitLab is a web-based DevOps lifecycle tool that provides a Git-repository manager providing wiki, issue-tracking and continuous integration/continuous deployment pipeline features, using an open-source license, developed by GitLab Inc (Wikipedia). Tor uses GitLab for issue tracking, source code and wiki hosting, at https://gitlab.torproject.org, after migrating from Trac and gitolite.

Note that continuous integration is documented separately, in the CI page.

System administrators should consult the service documentation.

Basic usage

We also have training documents here:

If you haven't been onboarded with GitLab properly, please book an appointment for a training session with TPA.

How to get an account?

If want a new account, you should request a new one at https://anonticket.torproject.org/user/gitlab-account/create/.

But you might already have an account! If you were active on Trac, your account was migrated with the same username and email address as Trac, unless you have an LDAP account, in which case that was used. So head over to the password reset page to get access to your account.

How to report an issue in Tor software?

You first need to figure out which project the issue resides in. The project list is a good place to get started. Here are a few quick links for popular projects:

If you do not have a GitLab account or can't figure it out for any reason, you can also use the mailing lists. The tor-dev@lists.torproject.org mailing list is the best for now.

How to report an issue in the bugtracker itself?

If you have access to GitLab, you can file a new issue after you have searched the GitLab project for similar bugs.

If you do not have access to GitLab, you can email gitlab-admin@torproject.org.

Note about confidential issues

Note that you can mark issues as "confidentials" which will make them private to the members of the project the issue is reported on (the "developers" group and above, specifically).

Keep in mind, however, that it is still possible issue information gets leaked in cleartext, however. For example, GitLab sends email notifications in cleartext for private issue, an known upstream issue.

We have deployed a workaround for this which redacts outgoing mail, by replacing the email's content with a notification that looks like:

A comment was added to a confidential issue and its content was redacted from this email notification.

If you have an OpenPGP key in the account-keyring repository and a @torproject.org email associated with your GitLab account, the contents will instead be encrypted to that key. See tpo/tpa/gitlab#151 for that work and How do I update my OpenPGP key?

Note that there's still some metadata leaking there:

  • the issue number
  • the reporter
  • the project name
  • the reply token (allowing someone to impersonate a reply)

This could be (partly) fixed by using "protected headers" for some of those headers.

Some repositories might also have "web hooks" that notify IRC bots in clear text as well, although at the time of writing all projects are correctly configured. The IRC side of things, of course, might also leak information.

Note that internal notes are currently not being redacted, unless they are added to confidential issues, see issue 145.

How to contribute code?

As reporting an issue, you first need to figure out which project you are working on in the GitLab project list. Then, if you are not familiar with merge requests, you should read the merge requests introduction in the GitLab documentation. If you are unfamiliar with merge requests but familiar with GitHub's pull requests, those are similar.

Note that we do not necessarily use merge requests in all teams yet, and Gitolite still has the canonical version of the code. See issue 36 for a followup on this.

Also note that different teams might have different workflows. If a team has a special workflow that diverges from the one here, it should be documented here. Those are the workflows we know about:

If you do not have access to GitLab, please use one of the mailing lists: tor-dev@lists.torproject.org would be best.

How to quote a comment in a reply?

The "Reply" button only creates a new comment without any quoted text by default. It seems the solution to that is currently highlighting the text to quote and then pressing the r-key. See also the other keyboard shortcuts.

Alternatively, you can copy-paste the text in question in the comment form, select the pasted text, and hit the Insert a quote button which look like a styled, curly, and closing quotation mark ”.

Setting up two-factor authentication (2FA)

We strongly recommend you enable two-factor authentication on GitLab. This is well documented in the GitLab manual, but basically:

  1. first, pick a 2FA "app" (and optionally a hardware token) if you don't have one already

  2. head to your account settings

  3. register your 2FA app and save the recovery codes somewhere. if you need to enter a URL by hand, you can scan the qrcode with your phone or create one by following this format:

      otpauth://totp/$ACCOUNT?secret=$KEY&issuer=gitlab.torproject.org
    

    where...

    * `$ACCOUNT` is the `Account` field in the 2FA form
    * `$KEY` is the `Key` field in the 2FA form, without spaces
    
  4. register the 2FA hardware token if available

GitLab requires a 2FA "app" even if you intend to use a hardware token. The 2FA "app" must implement the TOTP protocol, for example the Google Authenticator or a free alternative (for example free OTP plus, see also this list from the Nextcloud project). The hardware token must implement the U2F protocol, which is supported by security tokens like the YubiKey, Nitrokey, or similar.

Compatibility note - our generated "account name" used in the OTP generation is quite long, which may cause problems when enrolling with some authenticator apps. For the yubikey app, you can work around this by shortening the pre-filled "issuer" field, which shares storage quota with the "account name", but is only used for display. See tpa/gitlab#185.

Advanced usage

Use over Tor

GitLab's web interface won't work in Tor Browser's "safest" mode, so first downgrade to "safer".

To access GitLab over Tor, use http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/.

The Onion-Location header missing, we know. That's because nginx ignores add-header directives when we return 400, if someone has an idea, let us know.

Git pulls and push

Git should work unchanged if SSH is still reachable over clearnet.

Unauthenticated pulls won't work over Tor, so you can try this to replace the HTTP service with SSH in your .gitconfig:

[url "ssh://git@gitlab.torproject.org/"]
insteadOf = https://gitlab.torproject.org/

TODO: figure out how to do Git over tor?

2FA

Two-factor authentication is a problem for YubiKey users, but there are workarounds.

You should be able to use recovery codes. If you're missing recovery codes, follow this procedure which allows you to generate new codes over SSH:

ssh git@gitlab.torproject.org 2fa_recovery_codes

OTP authenticator apps should work.

gitlab might not prompt you for an OTP code if you have an enrolled webauthn security key and have enabled webauthn in your browser. A workaround is to disable security.webauth.webauthn in about:config (i.e. revert to the default).

Our generated "account name" used in the OTP generation is quite long, which may cause problems when enrolling with some authenticator apps. See tpa/gitlab#185. (Not related to the onion site itself).

Your YubiKey (webauthn) won't work.

The (first) reason why your YubiKey is not working is that, by design, it's not going to authenticate you against another URL (which is the .onion site instead of gitlab.tpo). That is a good thing.

Here are the steps that ought to work to re-enroll via the .onion site, but no one has succeeded thus far:

  • in about:config, set security.webauth.webauthn to true and restart. (WARNING: the safety of this has not been audited, which is why it's disabled by default)
  • Log in some other way (e.g. recovery code)
  • re-enroll your YubiKey

No on has been able to actually login with a YubiKey. On login, GitLab will say:

failed to connect to your device. Try again (NotAllowedError)

We've also found some people have trouble with enrolling a key in the first place, where it fails with:

Unable to use this authentication method. Try a different authentication method.

Continuous Integration (CI)

All CI documentation resides in a different document see CI.

Publishing notifications on IRC

Note: this might be deprecated soon. We're experimenting with a Matrix bot instead, see tpo/tpa/team#42556.

By default, new projects do not have notifications setup in #tor-bots like all the others. To do this, you need to configure a "Webhook", in the Settings -> Webhooks section of the project. The URL should be:

https://kgb-bot.torproject.org/webhook/

... and you should select the notifications you wish to see in #tor-bots. You can also enable notifications to other channels by adding more parameters to the URL, like (say) ?channel=tor-foo.

Important note: do not try to put the # in the channel name, or if you do, URL-encode it (e.g. like %23tor-foo), otherwise this will silently fail to change the target channel.

Other parameters are documented the KGB documentation. In particular, you might want to use private=yes;channel=tor-foo if you do not want to have the bot send notifications in #tor-bots, which is also does by default.

IMPORTANT: Again, even if you tell the bot to send a notification to the channel #tor-foo, the bot still defaults to also sending to #tor-bots, unless you use that private flag above. Be careful to not accidentally leak sensitive information to a public channel, and test with a dummy repository if you are unsure.

The KGB bot can also send notifications to channels that require a password. In the /etc/kgb.conf configuration file, add a secret to a channel so the bot can access a password-protected channel. For example:

channels:
    -
        name: '#super-secret-channel
        network: 'MyNetwork'
        secret: 'ThePasswordIsPassw0rd'
        repos:

            - SecretRepo

Note: support for channel passwords is not implemented in the upstream KGB bot. There's an open merge request for it and the patch has been applied to TPA's KGB install, but new installs will need to manually apply that patch.

Note that GitLab admins might be able to configure system-wide hooks in the admin section, although it's not entirely clear how does relate to the per-project hooks so those have not been enabled. Furthermore, it is possible for GitLab admins with root access to enable webhooks on all projects, with the webhook rake task. For example, running this on the GitLab server (currently gitlab-02) will enable the above hook on all repositories:

sudo gitlab-rake gitlab:web_hook:add URL='https://kgb-bot.torproject.org/webhook/'

Note that by default, the rake task only enables Push events. You need the following patch to enable others:

modified   lib/tasks/gitlab/web_hook.rake
@@ -10,7 +10,19 @@ namespace :gitlab do
       puts "Adding webhook '#{web_hook_url}' to:"
       projects.find_each(batch_size: 1000) do |project|
         print "- #{project.name} ... "

-        web_hook = project.hooks.new(url: web_hook_url)
+        web_hook = project.hooks.new(
+          url: web_hook_url,
+          push_events: true,
+          issues_events: true,
+          confidential_issues_events: false,
+          merge_requests_events: true,
+          tag_push_events: true,
+          note_events: true,
+          confidential_note_events: false,
+          job_events: true,
+          pipeline_events: true,
+          wiki_page_events: true,
+        )
         if web_hook.save
           puts "added".color(:green)
         else

See also the upstream issue and our GitLab issue 7 for details.

You can also remove a given hook from all repos with:

sudo gitlab-rake gitlab:web_hook:rm URL='https://kgb-bot.torproject.org/webhook/'

And, finally, list all hooks with:

sudo gitlab-rake gitlab:web_hook:list

The hook needs a secret token to be operational. This secret is stored in Puppet's Trocla database as profile::kgb_bot::gitlab_token:

trocla get profile::kgb_bot::gitlab_token plain

That is configured in profile::kgb_bot in case that is not working.

Note that if you have a valid personal access token, you can manage the hooks with the gitlab-hooks.py script in gitlab-tools script. For example, this created a webhook for the tor-nagios project:

export HTTP_KGB_TOKEN=$(ssh root@puppet.torproject.org trocla get profile::kgb_bot::gitlab_token plain)
./gitlab-hooks.py -p tpo/tpa/debian/deb.torproject.org-keyring create --no-releases-events --merge-requests-events --issues-events --push-events --url https://kgb-bot.torproject.org/webhook/?channel=tor-admin

Note that the bot is poorly documented and is considered legacy, with no good replacement, see the IRC docs and the Matrix integration below.

Publishing notifications on Matrix

You can enable notifications on certain events in your project (or even all projects inside a group!) to Matrix rooms. We use the Matrix integration here, but we found the upstream GitLab documentation to be lacking, so here's a more complete procedure:

  1. in a private browser window": create a new Matrix account, following the Matrix docs, with the following variations:

    1. username: add a -bot suffix, prefix with tpa- if this is a TPA bot; we named it tpa-gitlab-notifications-bot

    2. password: store the password in the team's password manager

    3. email: set the email address to the team's alias with the username as an extension (we used torproject-admin+tpa-gitlab-notifications-bot@torproject.org)

    4. optionally, turn off everything in the settings: notification, read receipts, integrations, you don't need any of those things for a bot

    5. go to the "Help & About" section, in "Advanced", click the "Access token" button and copy the access token to your password manager, you're likely going to use it more than once

    6. join (or get the bot invited) to the room you want to send notifications to

    7. go to the room information (the circle "i" on the top right"), click "settings", then "advanced" and then copy the "Internal room ID"

    8. do NOT log off from element, but you can close that browser window to discard the cookies

  2. in the group or project, head to Settings -> Integrations

  3. find the "Matrix notifications" group, hit the + Add button

  4. leave the "Active" box checked

  5. leave the "Hostname" empty, unless you are not on matrix.org, in which case you need to enter your Matrix home server base_url setting

    Note: that is not exactly the same as the "host" part of your URL. For example, for @foo:matrix.org, the base_url is actually https://matrix-client.matrix.org. You can find the base_url by visiting the /.well-known/matrix/client path on the server, that is https://matrix.org/.well-known/matrix/client. This will give you the URL:

      curl https://matrix.org/.well-known/matrix/client | jq -r '.["m.homeserver"].base_url'
    
  6. paste the access token from your password manager in the Token field

  7. set the triggers, we like to set the following:

    - [x] A push is made to the repository - [x] A work item is created, closed, or reopened - [ ] A confidential work item is created, closed, or reopened - [x] A merge request is created, merged, closed, or reopened - [ ] A comment is added - [ ] An internal note or comment on a confidential work item is added - [x] A tag is pushed to the repository or removed - [x] A pipeline status changes - [ ] A wiki page is created or updated - [x] An incident is created, closed, or reopened - [ ] A new, unique vulnerability is recorded (available only in GitLab Ultimate)

    That is: everything but updates, wiki pages, and confidential/internal issues and comments.

  8. set the "Room identifier" to the "Internal room ID" found above

  9. keep the "Notify only broken pipelines" box as is (checked)

  10. check the "Notify only when pipeline status changes" box

  11. keep the "Branches for which notifications are to be sent " drop down to the default ("Default branch")

Container registry operations

Enabling

The container registry is disabled by default in new GitLab projects.

It can be enabled via the project's settings, under "Visibility, project features, permissions".

Logging in

To upload content to the registry, you first need to login. This can be done with the login command:

podman login

This will ask you for your GitLab username and a password, for which you should use a personal access token.

In the case where you need a VM to be able to pull an image from a private project registry, you can use a Deploy token instead, with the read_registry permission. By default, podman login saves the credentials to /run/user/$uid/containers/auth.json. To make this persistent across reboots, you can move this file to ~/.config/containers/auth.json and ensure that the file is only accessible by the user itself.

Uploading an image

Assuming you already have an image built (below we have it labeled with containers.torproject.org/anarcat/test/airsonic-test), you can upload it with:

podman push containers.torproject.org/anarcat/test/airsonic-test containers.torproject.org/anarcat/test

Notice the two arguments: the first is the label of the image to upload and the second is where to upload it, or "destination". The destination is made of two parts, the first component is the host name of the container registry (in our case containers.torproject.org) and the second part is the path to the project to upload into (in our case anarcat/test.

The uploaded container image should appear under Deploy -> Container Registry in your project. In the above case, it is in:

https://gitlab.torproject.org/anarcat/test/container_registry/4

Cleanup policy

If your project builds container images and upload them to the registry in CI jobs, it's important to consider setting up a registry cleanup policy.

This is especially important if the uploaded image name or tag is based on a variable property like branch names or commit IDs. Failure to set up a cleanup policy will result in container images accumulating indefinitely and wasting valuable container storage space.

Email interactions

You can interact with GitLab by email too.

Creating a new issue

Clicking on the project issues gives a link at the bottom of the page, which says say "Email a new issue to this project".

That link should go into the "To" field of your email. The email subject becomes the title of the issue and the body the description. You can use shortcuts in the body, like /assign @foo, /estimate 1d, etc.

See the upstream docs for more details.

Commenting on an issue

If you just reply to the particular comment notification you received by email, as you would reply to an email in a thread, that comment will show up in the issue.

You need to have email notifications enabled for this to work, naturally.

You can also add a new comment to any issue by copy-pasting the issue-specific email address in the right sidebar (labeled "Issue email", introduced in GitLab 13.8).

This also works with shortcuts like /estimate 1d or /spend -1h. Note: for those you won't get notification emails back, though, while for others like /assign @foo you would.

See the upstream docs for more details.

Quick status updates by email

There are a bunch of quick actions available which are handy to update an issue. As mentioned above they can be sent by email as well, both within a comment (be it as a reply to a previous one or in a new one) or just instead of it. So, for example, if you want to update the amount of time spent on ticket $foo by one hour, find any notification email for that issue and reply to it by replacing any quoted text with /spend 1h.

How to mirror a Git repository from GitLab to GitHub

Some repositories are mirrored to the torproject organization on GitHub. This section explains how that works and how to create a new mirror from GitLab. In this example, we're going to mirror the tor browser manual.

  1. head to the "Mirroring repositories" section of the settings/repository part of the project

  2. as a Git repository URL, enter:

      ssh://git@github.com/torproject/manual.git
    
  3. click "detect host keys"

  4. choose "SSH" as the "Authentication method"

  5. don't check any of the boxes, click "Mirror repository"

  6. the page will reload and show the mirror in the list of "Mirrored repositories". click the little "paperclip" icon which says "Copy SSH public key"

  7. head over to the settings/keys section of the target GitHub project and click "Add deploy key"

      Title: https://gitlab.torproject.org/tpo/web/manual mirror key
      Key: <paste public key here>
    
  8. check the "Allow write access" checkbox and click "Add key"

  9. back in the "Mirroring repositories" section of the GitLab project, click the "Update now" button represented by circling arrows

If there is an error, it will show up as a little red "Error" button. Hovering your mouse over the button will show you the error.

If you want retry the "Update now" button, you need to let the update interval pass (1 minute for protected branch mirroring, 5 minutes for all branches) otherwise it will have no effect.

How to find the right emoji?

It's possible to add "reaction emojis" to comments and issues and merge requests in GitLab. Just hit the little smiley face and a dialog will pop up. You can then browse through the list and pick the right emoji for how you feel about the comment, but remember to be nice!

It's possible you get lost in the list. You can type the name of the emoji to restrict your search, but be warned that some emojis have particular, non-standard names that might not be immediately obvious. For example, 🎉, U+1F389 PARTY POPPER, is found as tada in the list! See this upstream issue for more details.

Publishing GitLab pages

GitLab features a way to publish websites directly from the continuous integration pipelines, called GitLab pages. Complete documentation on how to publish such pages is better served by the official documentation, but creating a .gitlab-ci.yml should get you rolling. For example, this will publish a hugo site:

image: registry.gitlab.com/pages/hugo/hugo_extended:0.65.3
pages:
  script:

    - hugo
  artifacts:
    paths:
      - public
  only:
    - main

If .gitlab-ci.yml already contains a job in the build stage that generates the required artifacts in the public directory, then including the pages-deploy.yml CI template should be sufficient:

include:

  - project: tpo/tpa/ci-templates
    file: pages-deploy.yml

GitLab pages are published under the *.pages.torproject.org wildcard domain. There are two types of projects hosted at the TPO GitLab: sub-group projects, usually under the tpo/ super-group, and user projects, for example anarcat/myproject. You can also publish a page specifically for a user. The URLs will look something like this:

Type of GitLab page Name of the project created in GitLab Website URL
User pages username.pages.torproject.net https://username.pages.torproject.net
User projects user/projectname https://username.pages.torproject.net/projectname
Group projects tpo/group/projectname https://tpo.pages.torproject.net/group/projectname