Skip to content

Automated ISO/IMG builds and tests on Jenkins

⚠ This service will change during policy/tpa-rfc-73-tails-infra-merge-roadmap and this page should be updated when that happens.

Access

jenkins.mtl-libvirt-02

  • SSH onion service: 3sboqgpxwoihrwufe3jzwcoh752xtp4zanwjafqzhpqqivehrmzmvtid.onion
  • SSH fingerprints:
    • SHA256:atYprceRhgpiWNs0RwOq33pNEzsaQMruap3iJ7rhefU root@jenkins (ECDSA)
    • SHA256:00zqcuIhnrztFDKs0tiNWFeF+kE8ypFaMVY3dRq1O+s root@jenkins (ED25519)
    • SHA256:I8FOoPTIGHUEGUFSmh5EHIkFwkRlbIGrN27NC2ju6t4 root@jenkins (RSA)

isoworker17.mtl-libvirt-02

  • SSH onion service: obnxiivfhpsosum3stt57eic6fu4mjwmzft7mumjbtccingguc5v5eid.onion
  • SSH fingerprints:
    • SHA256:4Y906QN/0qcYFEpvSCLd7aFvGuKGwItIxiX6QhUtSxs root@isoworker17 (ECDSA)
    • SHA256:u2WsR5TYinhGolInWQKlR7yNCweX38Q1i6t1UZDECP4 root@isoworker17 (ED25519)
    • SHA256:fMIQyNqzfL/6q+ZUE2KoO1gqpdqfCO3uGBtAQ1RR9c8 root@isoworker17 (RSA)

isoworker18.mtl-libvirt-02

  • SSH onion service: uaeggouawkpj6av3chu6fosqlibh5ubljdeizxa5apispiwyhmqxz7id.onion
  • SSH fingerprints:
    • SHA256:J/YkrmxT3pstdU4sqs9AmwT+F3l2pOllmXMmk6DVKVk root@isoworker18 (ECDSA)
    • SHA256:YmEe9WPL3iAxEeWoEH5SC+RD9hB7PmDKMOku2uygdBI root@isoworker18 (ED25519)
    • SHA256:Gp3YoojFvNPz/NSBbVWnkiX0UUut0yCKsHGd27BbyIw root@isoworker18 (RSA)

isoworker19.mtl-libvirt-02

  • SSH onion service: oxza4wt5vcnclquotatgjdxlcoj6ztdkt6clruszowkjmtebtagpykid.onion
  • SSH fingerprints:
    • SHA256:FEbkzMRLjBFb2lN3xJoeS3ZRw7O8Wak3xQMoIKDo+a0 root@isoworker19 (ECDSA)
    • SHA256:q6u3qZKeh0KCGlCc0cgok2sk0gPBVbWSLArIk75wvUA root@isoworker19 (ED25519)
    • SHA256:DW3sL7A16MSWAsLhhp2n/IBpbvOiTJFIlLr4mM4PhKg root@isoworker19 (RSA)

isoworker20.mtl-libvirt-02

  • SSH onion service: zvddi6qdltsy737qkjz3fstb3ng5r7f2pzoaf6decoh27u3imh432zqd.onion
  • SSH fingerprints:
    • SHA256:NdxVxZWTaP9UML5sm8WG0EsNZLU0J7TqDVpno3DO4Aw root@isoworker20 (ECDSA)
    • SHA256:3/DTvwWHb4V6x22r4qdRjDRofsDkQuAj0Q7PczsTC9w root@isoworker20 (ED25519)
    • SHA256:kgvJ/gnHfoc0aje3t+IshOLR7NrFfk7GY0Cg6hESqw4 root@isoworker20 (RSA)

isoworker21.mtl-libvirt-03

  • SSH onion service: 2vtuurwb4ndkqczgomoaefa65cstwuedjiua2tw6crcihnothrbyaeqd.onion
  • SSH fingerprints:
    • SHA256:JkJMlJPv/yZStuavDEWrgYlOYpI/eH/uEvslShE2VCQ root@isoworker21 (ECDSA)
    • SHA256:jOmDCOnr1aQhadiqodP7Tn4Kln/+F7SZqBWa4moHWxk root@isoworker21 (ED25519)
    • SHA256:pksMjyQhi3HIpW3Dg89h64jWRBYcu4gm9R6xGRTM7GY root@isoworker21 (RSA)

isoworker22.mtl-libvirt-03

  • SSH onion service: jqm6xbya6ckhq3ahqgrrnat2aji7y7h3zwjzhpyaey2zonld4ijrbiqd.onion
  • SSH fingerprints:
    • SHA256:ffKX5ctdjxP0PFsRRR2RCfupIrAE9iyAI+IzFA/ect4 root@isoworker22 (ECDSA)
    • SHA256:ZvGmLbOv9rHGBHY2fzV19t1xC+/iWI34jyH5jp/3ypg root@isoworker22 (ED25519)
    • SHA256:DoaO//t/gpNmG3AqrFxvFC9n1TfvuO2QHLBB97aEL3Q root@isoworker22 (RSA)

isoworker23.mtl-libvirt-03

  • SSH onion service: fscivkmdxpc6z66srfqagnhvsyu2u27vdj4zeshgjxt4zg4pmsbcojqd.onion
  • SSH fingerprints:
    • SHA256:OiJAYI+LDD8x6P7+MnyWs5obpXMTytcyJlEBB+OI6ow root@isoworker23 (ECDSA)
    • SHA256:nT4AsSjRwcDJI/s9/tZ1IrX2hiy0WYljqCeZnerwTF8 root@isoworker23 (ED25519)
    • SHA256:wrnG4Z/aLYFmEybn95PZ5J/JPRYQhPRb/oYPdK4DuVI root@isoworker23 (RSA)

isoworker24.mtl-libvirt-03

  • SSH onion service: tdlgl35lipgnwwi6opnv4itxee4rmnkthi6any7uijwbkb5ick4srgad.onion
  • SSH fingerprints:
    • SHA256:VKdKhceTI1CdG60p8VQc1AC0XYeQQ5Th9Anq+FGhTzc root@isoworker24 (ECDSA)
    • SHA256:jdnFFXX1SsUzfSJilwf0W4MXXb07FXEtEhCiU1OLjwo root@isoworker24 (ED25519)
    • SHA256:uVMhnxe60nmRP2snZT37a++Uh00OraAgN69kyLumHMo root@isoworker24 (RSA)

Configuration

Controller

* Puppet code. * YAML jobs configuration lives in a dedicated Git repository; Jenkins Job Builder uses it to configure Jenkins * Manual configuration (not handled by Puppet): - In the Jenkins web interface: - Security → Agents → TCP port for inbound agents → Fixed: 42585 - System → # of executors: 8 (actually, set to the same number of configured agents) - System → Git plugin → Global Config user.name Value: jenkins - System → Git plugin → Global Config user.email Value: sysadmins@tails.net - System → Priority Sorter → Only Admins can edit job priorities: checked - Job Priorities → Add 2 job groups: - Description: Top priority - Jobs to include: Jobs marked for inclusion - Job Group Name: 1 - Priority: 1 - Description: Test suite - Jobs to include: Jobs marked for inclusion - Job Name: 2 - Priority: 2 - Create one node for each agent, in Nodes → New node: - Node name: use the hostname of the agent (eg. "isoworker17.mtl-libvirt-02") - Number of executors: 1 - Remote root directory: /var/lib/jenkins - Usage: Use this node as much as possible - Launch method: Launch agent by connecting it to the controller - Disable WorkDir: checked - Internal data directory: remoting - Availability: Keep this agent online as much as possible - Preference of Node: choose a preference depending on the node specs - In the Jenkins VM: - For backups: Make sure there exists an SSH key for root and it's public part is configured in profile::tails::backupserver::backupagents for stone.tails.net (or the current backup server). - Document the onion server address and SSH fingerprints for the VM. - The configuration for the build_IUKs job is only stored in /var/lib/jenkins and nowhere else. - Create 4 different "Views": - RM: - Use a regular expression to include jobs into the view - Regular expression: ^(build_IUKs|(reproducibly_)?(test|build)_Tails_ISO_(devel|stable|testing|feature-trixie|experimental|feature-tor-nightly-master)(-force-all-tests)?) - Tails Build: - Use a regular expression to include jobs into the view - Regular expression: build_Tails_ISO_.* - Tails Build Reproducibility: - Use a regular expression to include jobs into the view - Regular expression: reproducibly_build_.* - Tails Test Suite: - Use a regular expression to include jobs into the view - Regular expression: test_Tails_ISO_.*

Manual controller reboots

Some times, the Jenkins controller needs to be manually rebooted (example), so we have sudo config in place that allows the jenkins user in the Jenkins controller VM to do that.

When logged in to the controller as the jenkins user, this should work:

jenkins@jenkins:~$ sudo reboot

Agents

* Puppet code.

* To disable an agent in a way that survives reboots and Puppet runs, create the /var/lib/jenkins/disabled file on that agent. This will prevent jenkins-slave.service from starting.

Web server

* Puppet code.

Upgrades

Upgrade policy

Here are some guidelines to triage security vulnerabilities in Jenkins and the plugins we have installed:

  1. Protecting our infra from folks who have access to Jenkins

    → Upgrading quarterly is sufficient.

  2. Protecting our infra from attacks against folks who have access to Jenkins

    For example, XSS that could lead a legitimate user to perform unintended actions with Jenkins credentials (i.e. root in practice).

    → We should stay on top of security advisories and react more quickly than "in less than 3 months".

  3. Protecting our infra from other 3rd-parties that affect Jenkins' security

    For example, say some Jenkins plugin, that connects to remote services, has a TLS certificate checking bug. This could potentially allow a MitM to run arbitrary code with Jenkins controller or workers permissions, i.e. root.

    → We should stay on top of security advisories and react more quickly than "in less than 3 months".

Upgrade procedure

  • Preparation:
    • Go through the changelog, paying attention to changes on how agents connect to controller, config changes that may need update, important changes in plugins, etc.
  • Deployment:
    • Take note of currently running builds before starting the upgrades.
    • Deploy Jenkins upgrade to latest version available using Puppet.
    • Generate a list of up-to-date plugins by running this Groovy script in the Jenkins Script Console. Make sure to update the initial list containing actively used plugins if there were changes.
    • Generate updated Puppet code for tails::jenkins::master using this Python3 script and the output of the above script.
    • Deploy plugin upgrades using the code generated above.
    • Restart all agents.

    • Manually run the Update jobs script (may be needed so XML is valid with current Jenkins):

      sudo -u jenkins /usr/local/sbin/deploy_jenkins_jobs update
  • Wrap up:
    • Go through warnings in Jenkins interface.
    • Manually remove unneeded plugins from /var/lib/jenkins/plugins.
    • Restart builds that were interrupted by Jenkins restart.
    • Update this Jenkins upgrade procedure in case there were changes.
    • Schedule next update.

Agent to controller connections

These are the steps a Jenkins agent does when connecting to the controller:

  1. Fetch connection info from http://jenkins.mtl-libvirt-02:8080 (see the tails::jenkins::slave Puppet class).
  2. Receive the connection URL https://jenkins.tails.net ("Jenkins URL", manually configured in Configure System).
  3. Resolve jenkins.tails.net to 192.168.122.1 (because of libvirt config).
  4. Connect using HTTPS to jenkins.tails.net:443.
  5. Learn about port 42585 (fixed "TCP port for inbound agents", manually configured in Configure Global Security).
  6. Finally, connect using HTTP to jenkins.tails.net:42585.

Generating jobs

We generate automatically a set of Jenkins jobs for branches that are active in the Tails main Git repository.

The first brick extracts the list of active branches and output the needed information:

This list is parsed by the generate_tails_iso_jobs script run by a cronjob and deployed by our puppet-tails tails::jenkins::iso_jobs_generator manifest.

This script output YAML files compatible with jenkins-job-builder. It creates one project for each active branch, which in turn uses several JJB job templates to create jobs for each branch:

  • build_Tails_ISO_*
  • reproducibly_build_Tails_ISO_*
  • test_Tails_ISO_*

This changes are pushed to our jenkins-jobs git repo by the cronjob, and thanks to their automatic deployment in our tails::jenkins::master and tails::gitolite::hooks::jenkins_jobs manifests in our puppet-tails repo, these new changes are applied to our Jenkins instance.

Passing parameters through jobs

We pass information from build job to follow-up jobs (reproducibility testing, test suite) via two means:

  • the Parameterized Trigger plugin, whenever it's sufficient
  • the EnvInject plugin, for more complex cases:
    • In the build job, a script collects the needed information and writes it to a file that's saved as a build artifact.
    • This file is used by the build job itself, to setup the variables it needs (currently only $NOTIFY_TO).
    • Follow-up jobs imported this file in the workspace along with the build artifacts, then use an EnvInject pre-build step to load it and set up variables accordingly.

Builds

See jenkins/automated-builds-in-jenkins.

Tests

See jenkins/automated-tests-in-jenkins.