In the world of DevOps, automation is one of the primary goals. This includes automating how you deploy your software. Rather than relying on someone to rsync/FTP/write their software on the machine it is being deployed upon, there is the concept of CI/CD.

CI, or Continuous Integration, is the step of creating an artifact from code commits. This could be a Docker image, deployed using commits in the Master branch of a Git repository.

CD, or Continuous Deployment/Delivery, is the step of deploying the artifact. This could be anything from the Deployment system running a docker pull and docker run on a machine, telling an AWS Lambda to update to the latest code in an S3 bucket, or it could be to tell a Kubernetes cluster to update an existing Deployment to use a new image.

In this post, I’m going to quickly go through how to setup a Jenkins pipeline that will trigger ArgoCD. ArgoCD is a CD tool used to update Kubernetes clusters, based upon the manifests within a Git repository. It can deploy standard Kubernetes manifests, use Kustomize to update them or Helm charts.

Prerequisites

I won’t go into the details of how to setup ArgoCD itself, as their documentation covers this very well already.

What you will need to do to use Jenkins with ArgoCD is: -

  • argocd-cli installed on the Jenkins worker/runner
  • A Jenkins Deployment role within ArgoCD, that has access to update Applications
  • One or more repositories that ArgoCD has access to in your Git-based version control system (e.g. GitHub, Gitlab, Gitea, Git with no frontend)
  • The Jenkins worker/runner needs to be able to access ArgoCD via the API, so ensure correct firewall rules are in place

ArgoCD CLI

We configure/prepare the images for our Jenkins workers using Packer. Because of this, adding in additional tools is quite straightforward. If however you run static workers, then just ensure that the argocd binary is installed somewhere that is runnable by the Jenkins user. An example would be: -

$ pwd
/home/jenkins

## Download the tool 
$ curl -LO https://github.com/argoproj/argo-cd/releases/download/v1.2.0/argocd-linux-amd64 

## Move it to the /usr/local/bin
$ sudo mv argocd-linux-amd64 /usr/local/bin/argocd

## Ensure it is exectutable
$ sudo chmod 755 /usr/local/bin/argocd

## Check it works
$ argocd version
argocd: v1.2.0+674978c
  BuildDate: 2019-09-04T21:26:04Z
  GitCommit: 674978cd587701b39e81fce6d5c960b6d76d5882
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.2.0+674978c
  BuildDate: 2019-09-04T21:27:17Z
  GitCommit: 674978cd587701b39e81fce6d5c960b6d76d5882
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: 0.13.1

ArgoCD Jenkins Deploy Role

To create a Deployment role in ArgoCD, go to the ArgoCD dashboard, click on the Gears icon on the sidebar (to take you to settings) and go to Projects.

ArgoCD Settings

In Projects, choose whatever project your application is running in. If you haven’t created any yet, this is going to be Default.

ArgoCD Projects

In the project, go to Roles, and click Add Role

ArgoCD Add Role

In the role, you can give the Role the name you would like, and a description to make it easier to work out its purpose in future. You can apply fine-grained policy here to say what Jenkins can do (e.g. only create, only update, only sync existing applications). Alternatively you can give it full access to do anything, but obviously this is not recommended in a production environment

ArgoCD Add Role Jenkins

When all of the above is done, you will then need to create a JWT (JSON Web Token). This is used to authenticate the user, ensuring only the client (in this case, Jenkins) can assume this role and use it’s permissions. This can be done in the Web UI, or it can be done via the CLI.

To create the JWT from the CLI, you need to do argocd proj role create-token {PROJECT-NAME} {PROJECT-ROLE}, with an example below: -

$ argocd proj role create-token default jenkins-deploy-role
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1Njg3MjEyMjEsImlzcyI6ImFyZ29jZCIsIm5iZiI6MTU2ODcyMTIyMSwic3ViIjoicHJvajpkZWZhdWx0OmplbmtpbnMtZGVwbG95LXJvbGUifQ.UyXNZtdbDyllzGl7PbLPhMgqNMFE1oJqONaLHV8RK-k

Keep this somewhere safe (i.e. some form of password manager)

Adding the Token to Jenkins

To add the token into Jenkins itself (ready to be used in a Pipeline), first go to your Jenkins instance and go to Credentials on the sidebar, then select System, then Global credentials

Jenkins Configure

In here, you’ll be given the option to Add Credentials

Jenkins Add Credentials

In here, fill in the fields as following: -

Field Value
Kind Secret Text
Scope Global
Secret This is the JWT you created earlier
ID Pick a name, e.g. argocd-deploy-role
Description Choose a description that is relevant, e.g. Jenkins Deploy Token for ArgoCD

You can now make use of this Token within your Jenkins pipelines

Allow ArgoCD to access your repositories

The repositories that ArgoCD needs access to is those that host your Kubernetes manifests. You could keep the manifests in the same repository as your code, or you could have them in an entirely separate repository.

There are a number of options you can do to set this up. You can connect using SSH to your Version Control system, or HTTPS. I chose SSH.

Go to the Settings page in ArgoCD again, but rather than clicking on Projects, click on Repositories. In here, click on Connect repo using SSH

ArgoCD Add Repository

To generate an SSH key for this, choose your favourite method. I’m running Linux, so I generate them as such: -

$ ssh-keygen -o -f argocd-deploy -C "[email protected]"
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in argocd-deploy.
Your public key has been saved in argocd-deploy.pub.
The key fingerprint is:
SHA256:kYa25wSePx/R1aLd/p8Z+b6MgPWDNMVsS+4Tp64JrHE [email protected]
The keys randomart image is:
+---[RSA 3072]----+
|                 |
|       . .  o  . |
|      + +    *o .|
|     o = . .=+.o |
|      + S .+o+...|
|       =. +.= +..|
|       .+E.o * o.|
|        +o..+ = *|
|       .  .o.o *B|
+----[SHA256]-----+

Take the contents of the Private Key (in my case, argocd-deploy) and paste them into the SSH private key data field in ArgoCD. Supply the URL of the repository that your Kubernetes manifests sit in (the HTTPS URL if you are connecting with HTTPS, SSH URL if via SSH).

You’ll need the public key (argocd-deploy.pub in my case) for setting up a Deploy Key in your chosen Git server.

For more information on setting up deploy tokens in GitHub or Gitlab, see: -

If you use another Git server for your version control, please refer to their documentation

Firewall Rules

Your Jenkins worker/runner needs to be able to access the ArgoCD API. We expose it on port TCP:443 (i.e. standard HTTPS) so you will need to ensure your firewall allows this through.

Kubernetes Manifests

As mentioned, you can use standard Kubernetes manifests, Kustomize and/or Helm charts to deploy your applications. I am using Kustomize to allow us to update the Docker image hash used in the Kubernetes Deployment. Without this, even if a new image is available in your Docker registry (private or public), the Deployments never update.

This is because the Kubernetes API doesn’t think the Deployment has changed. The tag is the same as before, so it doesn’t know that it needs to do anything.

As of Kubernetes version v1.15, they now support a rolling restart. Used in combination with ImagePullPolicy: Always, this will force the deployment to pick up the new image.

Pre-Kubernetes v1.15, the Deployment would never update. It is not seen as best practice to use :latest anyway, so referring to actual hashes is probably a good idea anyway.

Example Manifest

My very basic Kubernetes manifest is below: -

## ------------------- Debian Deployment ------------------- #

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: debian-test
  name: debian-test
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: debian-test
  template:
    metadata:
      labels:
        k8s-app: debian-test
    spec:
      containers:
      - name: debian-test
        image: {AWS-ACCOUNT}.dkr.ecr.{AWS-REGION}.amazonaws.com/k8s-debian-test:latest 
        imagePullPolicy: Always
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 80
          protocol: TCP
        livenessProbe:
          httpGet:
            scheme: HTTP
            path: /
            port: 80
          initialDelaySeconds: 30
          timeoutSeconds: 30

---
## ------------------- Debian Service ------------------- #

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: debian-test
  name: debian-test
spec:
  ports:
    - port: 80
      targetPort: 80
  type: NodePort
  selector:
    k8s-app: debian-test

We use Amazon’s ECR private Docker registry for our images. As you can see, the above says :latest as the image tag. However, we also use Kustomize to change this per deployment. Our kustomization.yaml file is very simple: -

resources:
- debian-test.yaml

As of v1.2.0, ArgoCD can leverage Kustomize natively (if you specify the correct arguments), so there is no need to add anything further in this YAML file.

Jenkins Pipeline

Now all the prerequisites are done, you have ArgoCD connected to your Repository with your manifests, and you have created your manifests, you can create a Jenkinsfile ready to deploy the application

pipeline {
    agent {
        node {
            label 'testing'
        }
    }
    
    stages {       
        stage('Prepare') {
            steps {
                checkout([$class: 'GitSCM',
                branches: [[name: "origin\master"]],
                doGenerateSubmoduleConfigurations: false,
                submoduleCfg: [],
                userRemoteConfigs: [[
                    url: 'ssh:\\[email protected]\argocd-test\argocd-test.git']]
                ])
            }
        }
        stage ('Docker_Build') {
            steps {
                \\ Build the docker image
                sh'''
                    # Build the image
                    $(aws ecr get-login --region eu-west-1 --profile global --no-include-email)
                    docker build . -t k8s-debian-test
                '''
            }
        }
        
        stage ('Deploy_K8S') {
             steps {
                     withCredentials([string(credentialsId: "jenkins-argocd-deploy", variable: 'ARGOCD_AUTH_TOKEN')]) {
                        sh '''
                        ARGOCD_SERVER="argocd-prod.example.com"
                        APP_NAME="debian-test-k8s"
                        CONTAINER="k8s-debian-test"
                        REGION="eu-west-1"
                        AWS_ACCOUNT="$ACCOUNT_NUMBER"
                        AWS_ENVIRONMENT="staging"

                        $(aws ecr get-login --region $REGION --profile $AWS_ENVIRONMENT --no-include-email)
                        
                        # Deploy image to ECR
                        docker tag $CONTAINER:latest $AWS_ACCOUNT.dkr.ecr.$REGION.amazonaws.com\$CONTAINER:latest
                        docker push $AWS_ACCOUNT.dkr.ecr.$REGION.amazonaws.com\$CONTAINER:latest
                        IMAGE_DIGEST=$(docker image inspect $AWS_ACCOUNT.dkr.ecr.$REGION.amazonaws.com\$CONTAINER:latest -f '{{join .RepoDigests ","}}')
                        # Customize image 
                        ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app set $APP_NAME --kustomize-image $IMAGE_DIGEST
                        
                        # Deploy to ArgoCD
                        ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app sync $APP_NAME --force
                        ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app wait $APP_NAME --timeout 600
                        '''
               }
            }
        }
    }
}

I won’t cover the Jenkins specifics (e.g. the stages, steps, labels etc) as there are a number of resources out there that cover this. Instead I’ll go through the relevant parts that make it a little different from your standard Jenkinsfile

ECR Login

This is used to login to AWS ECR, to retrieve Docker images (e.g. a base Debian image), and then to tag and push based upon our Dockerfile. If you are using standard images, or your own Private registry, you can ignore these parts.

Image Digest

For whatever reason, the Docker image digests are not always seen in docker images --digests. Instead, we inspect the image from the latest Push to ECR, and retrieve the .RepoDigests tag instead. This gives the following: -

$ docker image inspect ${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com/k8s-debian-test:latest -f '{{join .RepoDigests ","}}'
${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com/k8s-debian-test@sha256:###REALLY-LONG-SHA256-HASH-DIGEST###

We can then use this to pull the latest generated image

ArgoCD

Due to our ArgoCD API and dashboard being fronted by an AWS Application Load Balancer, we currently prefix all of ArgoCD commands with --grpc-web, e.g. argocd --grpc-web app sync TEST --force. This is because AWS ALBs do not support GRPC by default. If the load balancer/ingress you have in front of ArgoCD does support it, remove this from all the commands.

Customize Image

ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app set $APP_NAME --kustomize-image $IMAGE_DIGEST

The above command sets the image within the Kubernetes manifest to be that of the generated Image Digest variable. As ArgoCD supports Kustomize by default, it can manipulate the manifests itself. This means that we do not get into a situation where the Deployment never updates, due to the Image tag never changing.

This is a really nice feature, and means we do not need to run the Kustomize binary as well as ArgoCD on our Jenkins workers.

Sync the App

ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app sync $APP_NAME --force
ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app wait $APP_NAME --timeout 600

The above simply asks ArgoCD to trigger Kubernetes to deploy the app. Within ArgoCD, this is using the Manifest from the Git repository, that has been updated with Kustomize to use the new image tag. This will then go out and deploy a new version of it, based upon the image generated by Jenkins.

Jenkins Pipeline Run

Below is all the output you get from the build

Jenkins Run

Kubernetes Output

Pre-ArgoCD Run

kubectl get pods
NAME                                     READY   STATUS      RESTARTS   AGE
debian-test-8648f969ff-hrsvp             1/1     Running     0          4d22h

Post-ArgoCD Run

kubectl get pods
NAME                                     READY   STATUS      RESTARTS   AGE
debian-test-7664c648bb-sq6h7             1/1     Running     0          22s

Jenkins Console Output - ArgoCD


+ ARGOCD_SERVER=argocd-prod.example.com argocd \ 
  --grpc-web app set debian-test-k8s \ 
  --kustomize-image ${AWS-ACCOUNT}.dkr.ecr.${AWS-REGION}.amazonaws.com/k8s-debian-test@sha256:###SHA256-IMAGE-HASH###

+ ARGOCD_SERVER=argocd-prod.example.com argocd --grpc-web app sync debian-test-k8s --force
TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE
2019-09-17T13:05:27+00:00   apps  Deployment     default           debian-test    Synced  Healthy              
2019-09-17T13:05:27+00:00            Service     default           debian-test    Synced  Healthy              
2019-09-17T13:05:27+00:00   apps  Deployment     default           debian-test  OutOfSync  Healthy              

Name:               debian-test-k8s
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://argocd-prod.example.com/applications/debian-test-k8s
Repo:               [email protected]:yeti/argocd-test.git
Target:             HEAD
Path:               yaml
Sync Policy:        <none>
Sync Status:        Synced to HEAD (f5f91ad)
Health Status:      Progressing

Operation:          Sync
Sync Revision:      f5f91ad16296ecab90f337e5dbf3f4f927b61799
Phase:              Succeeded
Start:              2019-09-17 13:05:27 +0000 UTC
Finished:           2019-09-17 13:05:29 +0000 UTC
Duration:           2s
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME         STATUS  HEALTH       HOOK  MESSAGE
       Service     default    debian-test  Synced  Healthy            service/debian-test unchanged
apps   Deployment  default    debian-test  Synced  Progressing        deployment.apps/debian-test configured
+ ARGOCD_SERVER=argocd-prod.example.com argocd --grpc-web app wait debian-test-k8s --timeout 600
TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH            HOOK  MESSAGE
2019-09-17T13:05:29+00:00            Service     default           debian-test    Synced  Healthy                  service/debian-test unchanged
2019-09-17T13:05:29+00:00   apps  Deployment     default           debian-test    Synced  Progressing              deployment.apps/debian-test configured

Name:               debian-test-k8s
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://argocd-prod.example.com/applications/debian-test-k8s
Repo:               [email protected]:yetiops/argocd-test.git
Target:             HEAD
Path:               yaml
Sync Policy:        <none>
Sync Status:        Synced to HEAD (f5f91ad)
Health Status:      Healthy

Operation:          Sync
Sync Revision:      f5f91ad16296ecab90f337e5dbf3f4f927b61799
Phase:              Succeeded
Start:              2019-09-17 13:05:27 +0000 UTC
Finished:           2019-09-17 13:05:29 +0000 UTC
Duration:           2s
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME         STATUS  HEALTH   HOOK  MESSAGE
       Service     default    debian-test  Synced  Healthy        service/debian-test unchanged
apps   Deployment  default    debian-test  Synced  Healthy        deployment.apps/debian-test configured

ArgoCD Dashboard

ArgoCD Deployment

Summary

Why are we using Jenkins and ArgoCD? Jenkins is pretty good at building artifacts, bringing code commits together and accepting webhooks from something like GitHub or GitLab to kick off a job. However ArgoCD gives greater control on how we deploy the images, as everything can be described in native Kubernetes manifests.

This removes the need for exposing Kubernetes to Jenkins directly, as well as having to manipulate kubectl commands to make it deploy (or use extra plugins).

ArgoCD also gives us a live view of how the deployments went, and where we can roll back to if required. It also shows them in a way that is Kubernetes centric (i.e. how the Deployments and Services tie together) and so is much easier for developers and operations teams to see how everything ties together.

I hope this helps people looking for information on using Jenkins and ArgoCD together!