Jenkins CICD Docker Pipeline Using Ephemeral Slaves and HTTPS

23. January 2017 Docker 0

Introduction

Having great tools to support a task only makes things easier. I’ve written about using GitLab CE with Docker in the past and have come to really embrace Docker for all of my development tasks. Keeping true to the adoption of best practices, I felt it was time to start using a CICD pipeline due to its many benefits. For example, my unit tests automatically run when code is deployed to my source control repository. In addition, my latest build is automatically deployed to my integration environment. These are only a few benefits and the list goes on for a mile.

There are a few build tools or task runners available on the market. One that always pops up is Jenkins. It is very powerful, fully featured, extendable, and best of all, it’s open source. It has a strong community backing and is in active development. It integrates well with tools such as GitHub and GitLab and as you’ll come to find out, plays nice with Docker.

Using Jenkins with Docker

The folks behind Jenkins have made it easy to get started using it with Docker. The image they provide is excellent and serves as a great starting point for most people. This will allow Jenkins to run in a Docker environment but isn’t configured to work with build slaves. Having dedicated machines to use in the build process is powerful. Multiple machines can take part in the build and automation process and be configured differently based on the needs of a project.

Something else that is quite important is setting everything up securely using HTTPS. After all, who wants to pass along important build information in plain text? Looking up how to setup Jenkins with HTTPS returned various results and setting it up to meet my needs was challenging at first. The most difficult part of the equation was setting up ephemeral build slaves.

Ephemeral Build Slaves

Maxfield Stewart wrote an article on the subject of ephemeral build slaves using Docker. It is extremely informative and inspired me to create my own solution. This is because he left out details on securing everything with HTTPS. Granted, I don’t blame him for leaving this out as it would have only added to the complexity of his article. The good news is I have filled in the gaps and identified a working solution using HTTPS along with the proper setup with NGINX as a reverse proxy. I also had to figure out how to get JNLP (the protocol used for the build slaves) traffic forwarded through NGINX as well. Therefore, all the heavy lifting has been done for you and I hope this solution helps to meet your needs and save you time.

In contrast to ephemeral build slaves, normal build slaves in Jenkins are static. In other words, they are allocated like a VM that is constantly running, taking up precious compute resources. However, with the magic of Docker these build slaves can be dynamic; they can be provisioned and removed when needed. Specific build slaves can be created for a particular purpose and using Yet Another Docker Plugin, configured to work for certain jobs.

Diving into the README.md

I spent some time creating the README.md which is included with the GitHub repository. This section will list that file in its entirety (as of version 1.0.0). It provides all the relevant information needed to understand how it works and the benefits provided.

README.md

Jenkins CICD Docker Pipeline Using Ephemeral Slaves and HTTPS

This repository contains custom Docker files for running Jenkins using ephemeral build slaves over a NGINX reverse proxy. Everything is setup to run on HTTPS using a self-signed certificate (this needs to be created). This is a great way to setup the ultimate Jenkins CICD pipeline.

Be sure to see the change log if interested in tracking changes leading to the current release.

Getting Started

  1. Ensure Docker Compose is installed along with Docker Engine.
  2. Secure the Docker Daemon using TLS.
  3. Create a Docker network named development.
  4. Clone this repository into the desired location.
  5. In the Docker file for Jenkins Master, modify the -Duser.timezone setting found in the JAVA_OPTS environment variable to match the desired time zone. For more information, see this.
  6. Generate a self-signed certificate to use with the NGINX reverse proxy.
  7. Run the following command (geared toward Linux):sudo make build
  8. Run the following command (geared toward Linux):sudo make run
  9. Configure Jenkins to use ephemeral build slaves with JNLP.
  10. Create a new pipeline job and enter the following for the script.
  11. Save and run the job and take note of the results. If everything was setup properly, Docker should dynamically provision a Jenkins slave and then remove it when it’s no longer needed.

Please read the rest of the content found within in order to understand additional configuration options.

Securing the Docker Daemon Using TLS

The following will configure the Docker Daemon using TLS (geared toward Linux). Before proceeding, be sure to research the official article from Docker on the subject.

  1. Create a directory where the generated keys will be stored during the creation process.cd ~sudo mkdir dockercd docker
  2. Create the CA key by running the following commands.sudo openssl genrsa -aes256 -out ca-key.pem 4096The above command will require entering a passphrase to protect ca-key.pem. Enter a strong password and make note of what was entered as it’ll be needed later. Then enter the next command:sudo openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pemThe above command will request input in the following areas shown below.

    It’s important that for Common Name (e.g. server FQDN or YOUR name) to enter the hostname that the current OS is using (e.g. ubuntu-server). Also, keep in mind that the key will be valid for one year. If longer is desired, change the integer specified after -days.

  3. Create the server key by running the following commands.sudo openssl genrsa -out server-key.pem 4096Then enter the next command:sudo openssl req -subj "/CN=$HOST" -sha256 -new -key server-key.pem -out server.csrFor the above command, replace $HOST with the hostname that the current OS is using (e.g. ubuntu-server).Then enter the next command:echo subjectAltName = IP:$IPADDRESS,IP:127.0.0.1 > extfile.cnf

    For the above command, replace $IPADDRESS with the IP address of the current host. Running the command ifconfigshould show the current IP address.

    Then enter the next command:

    sudo openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf

    In the above command, keep in mind that the key will be valid for one year. If longer is desired, change the integer specified after -days.

  4. Create the client key by running the following commands.sudo openssl genrsa -out key.pem 4096Then enter the next command:sudo openssl req -subj '/CN=client' -new -key key.pem -out client.csrThen enter the next command:echo extendedKeyUsage = clientAuth > extfile.cnfThen enter the next command:

    sudo openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cert.pem -extfile extfile.cnf

    In the above command, keep in mind that the key will be valid for one year. If longer is desired, change the integer specified after -days.

  5. Run the following commands on all files ending with .pem (e.g. ca-key.pem, cert.pem, key.pem). This needs to happen for the reasons referenced in this discussion.sudo mv ca-key.pem ca-key.baksudo mv cert.pem cert.baksudo mv key.pem key.bakThen enter the next commands:sudo openssl rsa -in ca-key.bak -text > ca-key.pemsudo openssl rsa -in cert.bak -text > cert.pem

    sudo openssl rsa -in key.bak -text > key.pem

    Remove the .bak files as they are no longer needed.

    sudo rm ca-key.bak cert.bak key.bak

  6. Remove the certificate signing requests as they are no longer necessary.sudo rm -v client.csr server.csr
  7. Protect the keys and certificates by assigning the appropriate permissions.sudo chmod -v 0400 ca-key.pem key.pem server-key.pemThen enter the next command:sudo chmod -v 0444 ca.pem server-cert.pem cert.pem
  8. Copy the keys into the appropriate folder so Docker can use them.sudo mkdir /etc/dockerThen enter the next command:sudo cp ca.pem /etc/docker/.Then enter the next command:sudo cp server*.pem /etc/docker/.
  9. Create and configure the docker.conf file by running the following commands.sudo mkdir /etc/systemd/system/docker.service.dThen enter the next command:sudo vim /etc/systemd/system/docker.service.d/docker.confNext copy and paste the following into the file.

  10. Reload and restart Docker by running the following commands.sudo service docker stopThen enter the next command:sudo systemctl daemon-reloadThen enter the next command:sudo service docker start

If everything worked correctly, issuing the command sudo docker ps should work fine without any errors.

Generating a Self-Signed Certificate for NGINX

In order to generate a self-signed certificate (using OpenSSL) to secure all HTTP traffic, follow these instructions (geared toward Linux).

  1. Run the command sudo openssl genrsa -out server.key 4096 which will generate a secure server key.
  2. Run the command sudo openssl req -new -key server.key -out server.csr which will generate the certificate signing request.
  3. The above command will request input in the following areas shown below.

    It’s important that for Common Name (e.g. server FQDN or YOUR name) to enter jenkins-nginx.

  4. Run the command sudo openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt to create the signed certificate. The certificate will be valid for one year unless the value used for days is different.
  5. Copy both server.crt and server.key into ./jenkins-nginx/config/ssl (part of this repository) and overwrite the dummy files. These files will be copied into the Jenkins NGINX container and used to secure the Jenkins Master instance.
  6. Copy server.crt into ./jenkins-slave/config/ssl (part of this repository) and overwrite the dummy file. This file will be copied into the Jenkins Slave container and used to allow it to communicate with the Jenkins Master instance.
  7. Delete the leftover certificate signing request file: sudo rm server.csr.

Configuring Jenkins to Use ephemeral Build Slaves

With the Docker Daemon secured using TLS and Jenkins Master running behind a NGINX reverse proxy using HTTPS, work can proceed to configure the Jenkins slave options.

  1. While in Jenkins (e.g. https://192.168.1.50:9155), on the left sidebar go to Credentials and then on the sidebar below Credentials, click System.
  2. Click on Global credentials (unrestricted) and then click Add Credentials in the left sidebar.
  3. For Kind, select Docker Host Certificate Authentication and for the Scope, choose Global (Jenkins, nodes, items, all child items, etc).
  4. Locate the client key used to secure the Docker Daemon. If it wasn’t deleted, it should still be in /home/user/docker. Check by issuing the following commands.cd ~/dockerlsIf key.pem is there then proceed to the next step. Otherwise, revisit Securing the Docker Daemon Using TLS.
  5. Copy the contents of key.pem and paste into the Client Key field in Jenkins.
  6. Locate the client certificate used to secure the Docker Daemon. It should be in /etc/docker/. Copy the contents of cert.pem and paste into the Client Certificate field in Jenkins.
  7. Locate CA certificate used to secure the Docker Daemon. It should be in /etc/docker/. Copy the contents of ca.pem and paste into the Server CA Certificate field in Jenkins.
  8. Click OK when finished.
  9. On the left sidebar, click Manage Jenkins then Configure System.
  10. Scroll down to the section titled Cloud and find Yet Another Docker. For the Cloud Name field, enter a desired name (e.g. hostname of Docker Machine) and for the Docker URL field, enter tcp://192.168.1.50:2376. Be sure to replace the IP address with that of the one running Docker Machine.
  11. Under the Host Credentials field, select the recently added credentials created in the previous steps.
  12. Under the Type field, select NETTY and then click on the Test Connection button. If no errors are displayed, move on to the next step. Otherwise, retrace/retry previous steps.
  13. Under the Max Containers field, the default is 50. This is the maximum amount of Jenkins slave containers that will be provisioned at any given time. Change this value to the desired amount or leave it as default.
  14. Under the Images section, click the Add Docker Template button and select Docker Template. For the Docker Image Name, enter danieleagle/jenkins-slave:1.0-ubuntu-16.04.
  15. For the Pull Strategy field, select Pull never. Since the image is local there is no need to pull it, so ensure this setting is set correctly.
  16. Under the Remove Container Settings section, check Remove volumes.
  17. For the Labels field, enter testslave. Note: This will likely be a different name later on when moving past the testing phase of this initial setup. This name should match the node found in the pipeline script.
  18. Under the Usage field, select Only build jobs with label expressions matching this node.
  19. Under the Launch method field, select Docker JNLP launcher.
  20. Under the Linux user field, enter jenkins.
  21. Under the Slave JVM options field, enter -Xmx8192m -Djava.awt.headless=true -Duser.timezone=America/Chicago. Be sure the change the timezone to the appropriate value.
  22. Under the Different jenkins master URL field, enter https://jenkins-nginx. Also, if everything was done correctly in creating the self-signed certificate for NGINX to use, then Ignore certificate check will not need to be checked. However, if SSL errors present themselves, this may be a good way to get around them but it’s not recommended to keep that setting in production.
  23. When done with everything, click the Save button at the bottom of the page.

Jenkins Secure Email with TLS

When setting up email using TLS for notifications, alerts, etc., be sure to uncheck SSL but instead use port 587. This will still send email securely using TLS and get around any resulting errors from checking the SSL box but using it with TLS. The Dockerfile for Jenkins Master adds -Dmail.smtp.starttls.enable=true to the JAVA_OPTS environment variable to ensure TLS will work.

Forwarding JNLP traffic

Since Jenkins Master is configured to use a NGINX reverse proxy with HTTPS, all HTTP traffic directed at Jenkins will go through this proxy so that HTTPS is enforced. In addition, a special configuration setting had to be specified for NGINX to forward the appropriate JNLP traffic for use by Jenkins slaves. This was configured by adding the following to the nginx.conf file.

Container Network

The network specified (can be changed to the desired value) by these Docker containers is named development. It is assumed that this network has already been created prior to using the included Docker Compose file. The reason for this is to avoid generating a default network so that other Docker containers can access the services these containers expose using the Docker embedded DNS server.

If no network has been created, run the following Docker command (geared toward Linux): sudo docker network create network-name. Be sure to replace network-name with the name of the desired network. For more information on this command, go here.

Port Mapping

The external ports used to map to the internal ports that Jenkins uses are 9155 (maps to 443 for HTTPS) and 9156 (maps to 50000 for JNLP). These ports can certainly be changed but please be mindful of the effects. Additional configuration may be required as a result. In addition, even though JNLP is being exposed as port 9156 it may not be needed. In other words, currently it’s being used internally. However, it has been exposed externally incase the need to use it that way presents itself.

Data Volumes

It is possible to change the data volume folders mapped to the Jenkins Master container to something other than volume_data/x if desired. It is recommended to choose a naming scheme that is easy to recognize.

Notes About the Included Make File

Instead of accessing Docker Compose directly, a makefile is included and should be used instead. The reason for this is that the Jenkins slave shouldn’t be running right away; it should only run when it’s required. All that will be handled dynamically by the Yet Another Docker Plugin which is included with the Jenkins Master container. Thus, using the makefile will prevent the Jenkins slave from running right away.

Included Logrotate Examples

In order to properly rotate the logs that Jenkins outputs, logrotate can be used. Included is an example logrotate file for Jenkins Master as well as a file for Jenkins NGINX that can be copied to /etc/logrotate.d so that the logs get rotated based on the settings specified in those files.

Also, another approach would be to create a file with logrotate settings for all Docker containers and copy it to /etc/logrotate.d. This would rotate all the logs for all Docker containers.

Jenkins Plugins

The file plugins.txt is used to install plugins for Jenkins when creating the Jenkins Master container. Additional plugins can be added (they can be removed as well) to this file before creating the container. Also, from time to time the plugins listed within this file may become out of date and require specifying the newest versions.

Jenkins Slave OS

It is possible to use a different OS for Jenkins slaves. However, it will require a specific configuration that works for the desired OS. In addition, the OS should work with the plugin used for the Jenkins slaves, Yet Another Docker Plugin. An example discussion on using Alpine Linux as the OS for this can be found here.

Jenkins Slave Tools

The included Jenkins slave container doesn’t have all the tools needed for a given objective or build task. It will need to be modified to include the appropriate tools. In addition, certain Jenkins plugins may need to be installed in order to achieve a specific task to complement the Jenkins slave.

Further Reading

This solution was inspired from an article by Maxfield Stewart from Riot Games found here. He also offers a repository that complements the article found here.

Special Thanks

Special thanks goes to David Hale for streamlining the process to secure the Docker Daemon and Maxfield Stewart as mentioned above for his amazing article that served as inspiration for this solution.

Conclusion

Getting Jenkins working with HTTPS using a self-signed certificate and ephemeral build salves wasn’t a straight forward process. Certainly there are a few examples that exist in the wild but I could never find anything that worked perfectly for me. This is why I created my own and shared it with the world. My hope is you find this useful for your current needs. Feel free to ask any questions or submit pull requests to improve what I’ve shared and good luck with your future endeavors.

Daniel Eagle

Currently residing in the Austin area, Daniel Eagle is an avid gamer, writer, technology/science enthusiast, software developer, and educator. He takes great pride in spreading knowledge and helping others.

More Posts - Website

Follow Me:
TwitterLinkedInYouTube