Build a CI/CD Pipeline with Docker and GitLab

Deploy is easier than ever!

Continuous Integration and Continuous Delivery are not new things this days. This practice can help to make your product or project be up and running the last version of your code faster and easier than ever. Today I will show you my way to create a pipeline using a custom Docker image and GitLab CI/CD tools to deploy to a VPS/KVM Linux server.

Requirements:

• Basic understanding of Linux, Docker and CI/CD.
• GitLab account (free plan is okay).
• Linux server with SSH access (root user is not required). I’ll be using Ubuntu 16.04 LTS with LAMP stack.
• Lightweight Docker image with SSH and LFTP.

Before continue, make sure you are logged-in into your GitLab account, you are owner of a project/repository and you have access to that repo in your local computer through Git, you must to be able to pull and push. I use GitKraken (Git GUI) to make it even easier.

About GitLab’s CI/CD

GitLab offers a simple way to handle CI/CD pipelines using Docker and shared runners. Every time you run your pipeline GitLab will create an isolated virtual machine and build a Docker image, which you can setup using a YAML configuration file. Those pipelines can have multiple jobs, but more jobs means more build time and we don’t want that, right? You can have up to 2000 build minutes per month using the free plan.

“Shared Runners on GitLab.com run in autoscale mode and are powered by DigitalOcean. Autoscaling means reduced wait times to spin up builds, and isolated VMs for each project, thus maximizing security.”

– GitLab Docs

Create SSH key for GitLab’s runner

Note: Even if you already have SSH access to your server, I recommend to create a new pair just for CI/CD, as well as a new non-root user for deploys.

We’ll connect to our server using SSH in Docker, this means we cannot type our user’s password (a.k.a. non-interactive login), so we need create a passwordless SSH key pair in our local computer. I usually create RSA keys with 2048 bits, which is secure enough.

$ ssh-keygen rsa -b 2048

Follow the creation steps — if any doubts, man ssh-keygen— and remember to not set a password for that key pair. After that, we have to import the private key into our server:

$ ssh-copy-id -i /path/to/key user@host

Now you can try to connect using:

$ ssh -i /path/to/key user@host

It should not ask for password. We’ll use the private key later.

Select the Dockerfile

I am using Docker Hub to host my custom Dockerfile, which is a lightweight (~8 Mb) Alpine-based container with OpenSSH and LFTP installed. We need this image to make GitLab’s CI/CD run our jobs and scripts, and less weight means less pulling/download time. You can host your own image or check the one that I use right here.

Pipeline configuration

You’ll need to create a “.gitlab-ci.yml” file in your repo’s root directory. I’m gonna paste and explain my configuration here, you can have a further read about this configuration file and all available options in this link.

My config file looks like this:

image: jimmyadaro/gitlab-ci-cd:latest
Deploy:
 stage: deploy
 only:
 — ‘master’
 when: manual
 allow_failure: false
 before_script:
 #Create .ssh directory
 — mkdir -p ~/.ssh
 #Save the SSH private key
 — echo “$SSH_PRIVATE_KEY” > ~/.ssh/id_rsa
 — chmod 700 ~/.ssh
 — chmod 600 ~/.ssh/id_rsa
 — eval $(ssh-agent -s)
 — ssh-add ~/.ssh/id_rsa
 script:
 #Backup everything in /var/www/html/
 — ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa $USERNAME@$HOST “zip -q -r /var/backups/www/01-Deploy-$(date +%F_%H-%M-%S).zip /var/www/html/”
 #Deploy new files to /var/www/html
 — lftp -d -u $USERNAME, -e ‘set sftp:auto-confirm true; set sftp:connect-program “ssh -a -x -i ~/.ssh/id_rsa”; mirror -Rnev ./ /var/www/html — ignore-time — exclude-glob .git* — exclude .git/; exit’ sftp://$HOST
 — rm -f ~/.ssh/id_rsa
 — ‘echo Deploy done: $(date “+%F %H:%M:%S”)’

Let’s review each line so we can understand what is going on here.

image: jimmyadaro/gitlab-ci-cd:latest

This says to the runner “pull and run this container’s last version from the Docker Hub”. Here you can set any image you want to use, but don’t forget we need SSH and LFTP.

Deploy:

This is the pipeline job name, you need to set this for create a job.

stage: deploy

This is the job’s stage name, which can help you to identify your current pipeline status since you can have multiple stages, such as “backup”, “build”, “deploy”, etc. I’ll use one job with one stage since I don’t really need any other stage. Both job name and stage name are totally customizable, you can use “ASDF” as job name and “GHJK” as stage name, but you may like to identify the stages during the pipeline execution so I suggest to normalize those names.

only:
 — ‘master’

This indicates the pipeline to trigger only when your repo’s masterbranch receives an update (such a git merge). The pipeline can also be triggered by changes on tags or even via webhooks. I recommend to work on any other branch (development, wip, whatever you like) and keep master as “production branch” since any changes will trigger the pipeline.

when: manual

This means you need to go into your project’s CI/CD configuration in order to manually make the whole deploy thing start. You can skip this, I just prefer to trigger this pipeline by hand. I you remove this line, any changes on your selected branch or tag files will run the pipeline.

allow_failure: false

This means that, if you have any other stage on your pipeline, having a failure in this job won’t allow the remaining jobs execution, so everything will just stop. This is optional too.

before_script:
 #Create .ssh directory
 — mkdir -p ~/.ssh
 #Save the SSH private key
 — echo “$SSH_PRIVATE_KEY” > ~/.ssh/id_rsa
 — chmod 700 ~/.ssh
 — chmod 600 ~/.ssh/id_rsa
 — eval $(ssh-agent -s)
 — ssh-add ~/.ssh/id_rsa

The before_script block will run the specified commands in your container before start executing the main (deploy) script. As you can see, each shell command is specified using a dash per line. This block will save our SSH private key in the container’s default SSH path so we can connect to our server using passwordless authentication.

This private key is stored as a protected variable in my project CI/CD configuration — currently under Settings > CI/CD > Variables on GitLab’s web UI. I do also have my server host and deploy username (non-root user) in variables to use them later.

Settings UI for GitLab’s CI/CD Variables

script:
 #Backup everything in /var/www/html/
 — ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa $USERNAME@$HOST “zip -q -r /var/backups/www/01-Deploy-$(date +%F_%H-%M-%S).zip /var/www/html/”
 #Deploy new files to /var/www/html
 — lftp -d -u $USERNAME, -e ‘set sftp:auto-confirm true; set sftp:connect-program “ssh -a -x -i ~/.ssh/id_rsa”; mirror -Rnev ./ /var/www/html — ignore-time — exclude-glob .git* — exclude .git/; exit’ sftp://$HOST
 — rm -f ~/.ssh/id_rsa
 — ‘echo Deploy done: $(date “+%F %H:%M:%S”)’

The block above is the main script that GL’s runner will be executing. First I connect to my server and backup everything in a ZIP file* with the current date formatted as yyyy-mm-dd_hh-mm-ssas filename:

— ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa $USERNAME@$HOST “zip -q -r /var/backups/www/01-Deploy-$(date +%F_%H-%M-%S).zip /var/www/html/”

*You may need to install ZIP CLI in your server.

After that backup under /var/www/html, LFTP connects to my server and uploads the new repo files — here I use SFTP, the FTP configuration may be different:

— lftp -d -u $USERNAME, -e ‘set sftp:auto-confirm true; set sftp:connect-program “ssh -a -x -i ~/.ssh/id_rsa”; mirror -Rnev ./ /var/www/html — ignore-time — exclude-glob .git* — exclude .git/; exit’ sftp://$HOST

Using mirror -Rnev ./ /var/www/htmltells LFTP to upload all files under ./(the repo root directory) to our remote path (var/www/htmlin my server). The options mean:

• -u sets the SSH username to our sftp://$HOST.
• -e for execute a command (the one we set on simple quotes).
• -R for reverse mirror (upload).
• -n for only newer files.
• -e will delete the files not present in our source**.
• -v , as usual, for verbose log.
• ignore-time will ignore time when deciding whether to download.
• exclude-glob .git* will exclude .git* files (such as .gitignore and .gitkeep) from any directory. You can add other file extensions or file names there.
• exclude .git/ will help us to be sure we don’t upload our git file.
• exit will stop LFTP and SSH execution.

**The files present on our server that are not in the repository will be deleted from the server. Remember the source is our GitLab’s repository.

Finally, the script removes the private key from the shared runner’s container (just a security step) and echoes a plain text string with the current date:

— rm -f ~/.ssh/id_rsa
 — ‘echo Deploy done: $(date “+%F %H:%M:%S”)’

And that’s it! Now you have your new files up and running in your server.

This is what a successful pipeline execution looks like in GitLab:

Running Docker image

Final pipeline status

Conclusion

I tried other approaches like using rsync instead of LFTP, jobs with multiple stages and artifacts and cache dependencies so I can reuse the SSH key, using Docker’s ENTRYPOINTs and CMDs… but this way seems like the faster and easier way to me.

Thanks for reading! I would really appreciate spelling checks as private notes 😇

Any doubts or comments? Just write them down there! 👇