CI/CD with Gitlab
Sept. 29, 2020/
CI/CD stands for continuous integration and continuous delivery (or deployment). These practices essentially automate the integration of code from different developers and the deployment of the new changes.
CI/CD reduces the time from when a feature is completed on a developer's machine, to when the feature is deployed and reaches the customer, all whilst maintaining code quality. By streamlining the integration and deployment process, a business can react to changing needs, new features, or fix bugs very quickly and have the final product deployed to customers as soon as the code has been finalised.
Continuous integration can help ensure code quality. Thorough tests can be written beforehand and can be set to run every time a new change is pushed to the repository. These tests can ensure functionality or enforce code style (and thus improve maintainability) with tools like Pylint for Python.
Continuous delivery may involve writing scripts that are executed after the testing phase passes. The aim is to deploy the new changes to production quickly, and this step is often triggered manually after a human review.
Continuous deployment takes this one step further and automates the deployment process in a sustainable way. This may involve creating a thorough testing and reviewing process, while also having a quick way to revert changes in case a bug slips through.
GitLab's CI/CD Pipelines are configured by adding a YAML file called
.gitlab-ci.yml to the project's root. This file determines
.gitlab-ci.yml will be provided in the next section as an example.
A pipeline consists of
Usually, a pipeline only proceeds to the next stage once all jobs in the current stage succeed. Otherwise, the pipeline terminates early. For example, if any job in the build stage fails, the test stage will not run.
A typical pipeline might consist of four stages, executed in this order:
The information in the secion above is based on the CI/CD section of GitLab's Docs.
Documentation about the
.gitlab-ci.yml file can be found here.
Further documentation about GitLab Pipelines can be found here.
GitLab Runners run the jobs in the pipeline. Further documentation can be found here.
This website (Daruweb) uses GitLab's CI/CD pipeline to ensure changes can be pushed to production quickly and easily. Daruweb is containerised in a Docker container. To deploy a change without CI/CD, one would have to manually
With some simple scripting and the use of GitLab's Pipelines, this entire process can be done in one click straight from GitLab's web GUI.
Daruweb utilises a
deploy stage with an additional
before stage that sets up the runner's environment. Steps 1 and 2 are done in the
build stage and are automatically executed whenever a change is pushed to GitLab.
Once the build succeeds, the developer can choose to deploy to staging (or UAT in this case) or production via a single click on GitLab's web interface. Once the
deploy stage is triggered, the runner copies and executes a script from the repository that performs steps 3 and onwards onto the respective VM. One advantage of this last step is that the script can be modified without needing to SSH into the VM.
The following example is the actual
.gitlab-ci.yml file used by Daruweb as of September 2020. Comments have been added to better explain the contents of the file. Sensitive data like passwords and keys can be configured as environmental variables from within GitLab. One such example is the key stored in
# Use the official Docker image image: docker:latest services: - docker:dind # before_script runs before the other jobs and is used to set up the runner's environment before_script: # Log in to Docker - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY # Install ssh-agent if not already installed, it is required by Docker - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' # Run ssh-agent (inside the build environment) - eval $(ssh-agent -s) # Add the SSH key stored in the daruweb variable to the agent store # tr is used to fix line endings for ed25519 keys - echo "$daruweb" | tr -d '\r' | ssh-add - # Create the SSH directory and give it the right permissions - mkdir -p ~/.ssh - chmod 700 ~/.ssh # ssh-keyscan scans the keys of the UAT and production servers - ssh-keyscan uat.darumatic.com >> ~/.ssh/known_hosts - ssh-keyscan darumatic.com >> ~/.ssh/known_hosts - chmod 644 ~/.ssh/known_hosts # This job builds the Docker image and pushes it up to Docker Hub build: stage: build script: - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" . - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" # This job deploys to the UAT server deploy-uat: stage: deploy script: # Copy the update_container script up to UAT - scp update_container.sh email@example.com:~ # SSH into UAT, login to Docker and run the update_container script - ssh firstname.lastname@example.org "docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY && ~/update_container.sh $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" # This manual job deploys to the production server deploy-prod: stage: deploy script: # Copy the update_container script up to prod - scp update_container.sh email@example.com:~ # SSH into prod, login to Docker and run the update_container script - ssh firstname.lastname@example.org "docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY && ~/update_container.sh $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" # Set deploy-prod to be manual so it can be triggered on GitLab's web GUI when: manual
GitLab's Pipelines allows CI/CD principles to be set up and applied to any project. Daruweb uses this feature to deploy new changes to production quickly and effortlessly.
© 2017-2020 Darumatic Pty Ltd. All Rights Reserved.