Hierarchical Continuous Delivery System

Photo by Brett Jordan on Unsplash

Introduction

Organizing deployments at a large scale has historically posed a significant challenge for organizations. Some companies, such as Netflix, have developed their solutions, such as Spinnaker, to manage continuous deployment across various environments like AWS or Kubernetes. One prevalent model for deployment management involves utilizing a hierarchy structure to inherit common settings across different levels of the organization. In this article, we’ll compare GitLab with other popular deployment management tools like GitHub, Jenkins, and Spinnaker, focusing on key criteria such as support for hierarchy groups, pipeline code management, inheritance of variables, Cron scheduling, and self-hosted capabilities.

What’s a Hierarchical CD model?

A hierarchy CD model is a concept for managing your deployments across different environments by inheriting common settings between them and categorizing them in some group. It helps reduce the need for defining common properties and cross-cutting concerns (like security) across your deployments. For example, when deploying two applications in a specific Kubernetes cluster, they may share common settings such as KUBECONFIG and CLUSTER_ADDRESS. Additionally, you can manage access permissions for them within the same group, limiting permissions to apply to all projects inside it.

It is recommended to utilize the following architecture for organizing your deployments within GitLab or Jenkins. GitLab and Jenkins both supports an infinite group and sub-group hierarchy, unlike other tools such as GitHub. You can leverage this feature as follows:

.
└── CD
    ├── kubernetes cluster 1
    │   ├── kubernetes namespace 1
    │   │   ├── project deployment 1
    │   │   ├── project deployment 2
    │   │   └── project deployment 3
    │   └── kubernetes namespace 2
    │       ├── project deployment 4
    │       ├── project deployment 5
    │       └── project deployment 6
    └── kubernetes cluster 2
        ├── kubernetes namespace 1
        │   ├── project deployment 7
        │   ├── project deployment 8
        │   └── project deployment 9
        └── kubernetes namespace 2
            ├── project deployment 10
            ├── project deployment 11
            └── project deployment 12

In the above model, the leaf nodes represent the Git repositories storing your CD pipelines for each application, corresponding to a GitLab project. Other parent nodes are GitLab groups that store common configurations as environment variables for their child projects.

Gitlab vs GitHub vs Jenkins vs Spinnaker

A CD deployment model for managing deployments involves using a hierarchy model to inherit common settings throughout the hierarchy. You can see the comparison between some popular tools used for an enterprise CD tool in the following table:

          Critiria          |  GitLab |  GitHub | Jenkins | Spinnaker
----------------------------|---------|---------|---------|---------
   support hierarchy group  |  yes    |   no    |   yes   |  no
----------------------------|---------|---------|---------|---------
    manage pipeline code    |  yes    |   yes   |   no    |  no
and execution in one place  |         |         |         |
----------------------------|---------|---------|---------|---------
  inherit variables from    |  yes    |    -    |   yes   |  -
      hierarchy group       |         |         |(limited)|
----------------------------|---------|---------|-------- |---------
  cron scheduling pipeline  |  yes    |   no    |   yes   |  yes
----------------------------|---------|---------|-------- |---------
  self-hosted (on-premises) |  yes    |   yes   |   yes   |  yes
                            |         |(limited)|         |

As you can see in the above table, some tools like Github and Spinnaker does not support hierarchy group specification and you can’t use them for this kind of concept.

Choosing between the above tools depends on your requirements, and there may be some features not mentioned in the table. However, based on the criteria mentioned above, GitLab appears to be the best choice as it supports all the required features for an enterprise deployment management system.

How to define Hierarchy model in Gitlab?

You can set the KUBECONFIG of your cluster or namespace as an environment variable for the corresponding group. If you define a variable for a group, all sub-groups, and sub-groups of sub-groups, and so on, will inherit the variable, allowing you to use it in all sub-projects.

For example, if you want to define the KUBECONFIG for CD/kubernetes cluster 2/kubernetes namespace 2, then all its child projects (project deployment 10, 11, 12) automatically inherit the KUBECONFIG variable from their parent. You can navigate to Settings → CI/CD → Variables → Add Variable and define your variable accordingly:

If you want to create the KUBECONFIG file from your Service Account, you can use the following script, which generates a kubeconfig file based on the default token secret:

# your server name goes here
api_server=https://{YOUR_API_SERVER}
# the name of the secret containing the service account token goes here
secret_name=${YOUR_SECRET_NAME}
 
ca=$(kubectl get secret/$secret_name -o jsonpath='{.data.ca\.crt}')
token=$(kubectl get secret/$secret_name -o jsonpath='{.data.token}' | base64 --decode)
namespace=$(kubectl get secret/$secret_name -o jsonpath='{.data.namespace}' | base64 --decode)
 
echo "
apiVersion: v1
kind: Config
clusters:
- name: default-cluster
  cluster:
    certificate-authority-data: ${ca}
    server: ${api_server}
contexts:
- name: default-context
  context:
    cluster: default-cluster
    namespace: default
    user: default-user
current-context: default-context
users:
- name: default-user
  user:
    token: ${token}
" > KUBECONFIG

In the above script file, you need to replace the values of api_server and secret_name with your desired values, and then run it.

You can find more details about how to configure the Gitlab pipeline in my other blog post here:

Managing multiple Kubernetes clusters using Git

Running a pipeline

To create a pipeline, you first need to create a Git repository project inside GitLab and then create a file named .gitlab-ci.yml. Define your pipeline as code inside this file:

stages:
  - prepare
  - deploy
 
prepare:
  stage: prepare
  image: debian:buster
  script: |
    echo "preparing the project manifests..."
 
deploy:
  stage: deploy
  image: kubectl
  script: |
    echo "deploying the manifests..."

As you can see in the above pipeline, it has two stages: build and prepare. A production-ready pipeline could look like this:

stages:
  - prepare
  - deploy
 
prepare:
  stage: prepare
  image: debian:buster
  script: |
    set -xe
 
    namespace=`echo $CI_PROJECT_NAMESPACE | awk -F'/' '{print $NF}'`
    export HOME=/tmp
 
    helm -n $namespace template ${CI_PROJECT_NAME} . > all.yaml
     
    kubectl-slice --input-file=all.yaml --output-dir=manifests
 
    cd manifests
 
    for filename in *; do
      if [[ $filename == job-* ]]; then
        kubectl -n $namespace get -f $filename || kubectl -n $namespace diff -f $filename || true
      else
        kubectl -n $namespace diff -f $filename || true
      fi
    done
 
  artifacts:
    paths:
      - manifests
    expire_in: 1 month
    when: always
 
deploy:
  stage: deploy
  image: kubectl
  script: |
    set -xe
    namespace=`echo $CI_PROJECT_NAMESPACE | awk -F'/' '{print $NF}'`
     
    cd manifests
 
    for filename in *; do
      if [[ $filename == job-* ]]; then
        kubectl -n $namespace get -f $filename || kubectl -n $namespace apply -f $filename
      else
        kubectl -n $namespace diff -f $filename || kubectl -n $namespace apply -f $filename || true
      fi
    done
     
   
  when: manual

As you can see in the above script, the KUBECONFIG variable is not mentioned, but in reality, GitLab will automatically inject all the variables defined in the parent groups into all the child projects, and kubectl will read the variable and use it.

For more details on how to write a production-ready CD pipeline, you can refer to my other blog post:

Monitoring the Rollout Process in the CD Pipeline

Conclusion

GitLab emerges as a robust deployment management solution, offering comprehensive support for hierarchical organization, pipeline management, and self-hosted deployment capabilities. By leveraging GitLab’s features and installation methods, organizations can streamline their deployment processes and achieve greater efficiency in managing deployments at scale.

Support My Blog ☕️

If you enjoy my free technical blog posts and find them valuable, please consider buying me a coffee at here. Your support goes a long way in helping me produce more quality articles and content. If you have any feedback or suggestions for improving my code, please leave a comment on this post or send me a message on my LinkedIn.