This post is the next in the series on how I overhauled my personal infrastructure to make it easier to manage, make changes and integrate new applications.

Previous posts in the series are: -

As we have already covered how to use set up Drone CI, as well as a working pipeline with Ansible, it’s time to show it working with Salt.

Background

As mentioned before, I use Ansible to bootstrap nodes, set up monitoring of devices that can’t be managed by Salt, build my DNS and DHCP configuration and some other minor.

For most other tasks though, I use Salt to manage my machines. This covers everything from setting up Prometheus and exporters, NFS, Promtail/Loki, Samba, Wireguard, Gitea and more.

Differences

Because Salt (unlike Ansible) uses agents and a central Salt server to apply changes to nodes, Drone can’t run Salt directly from within a container. I could configure Salt Masterless, but this requires a lot of extra work (including cloning the entire Salt configuration to each managed node), removing the speed benefits that Salt has over Ansible.

With this being the case, Drone needs some way of access the Salt central server (known as a “master”) to check, test and apply changes.

In the Introduction to Drone CI post, I referred to different kinds of runners (Docker, Exec, SSH) that perform different functions. In this case, Drone will use the Exec Runner. The Exec Runner can run commands directly on a machine (rather than inside of a Docker container), which fits with how to manage Salt.

Full Drone Pipeline

---
kind: pipeline
name: pr
type: exec

clone:
  disable: true

node:
  salt: server

trigger:
  branch:
  - main
  event:
  - pull_request

steps:
- name: Check all minions are up 
  commands:
    - "sudo salt --force-color '*' test.ping"
  when:
    event:
    - pull_request

- name: Salt Lint
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "git checkout ${DRONE_SOURCE_BRANCH}" 
    - "find /srv/salt -type f -name '*.sls' -print0 | xargs -0 --no-run-if-empty salt-lint -x 204 -x 203"
  when:
    event:
    - pull_request

- name: Highstate test
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "git checkout ${DRONE_SOURCE_BRANCH}" 
    - "sudo salt --force-color '*' saltutil.pillar_refresh"
    - "sudo salt --force-color '*' state.highstate --state-verbose=False test=True"
  when:
    event:
    - pull_request

---
kind: pipeline
name: slack-pr
type: docker

clone:
  disable: true

depends_on:
- pr

trigger:
  branch:
  - main
  event:
  - pull_request
  status:
  - success
  - failure

steps:
- name: slack-pr
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{#success build.status}}
        {{repo.name}} PR build passed. 
        Merge in to apply.
        PR: https://git.noisepalace.co.uk/YetiOps/{{repo.name}}/pulls/{{build.pull}}
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{else}}
        {{repo.name}} PR build failed. 
        Please investigate. 
        PR: https://git.noisepalace.co.uk/YetiOps/{{repo.name}}/pulls/{{build.pull}}
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{/success}}      
  when:
    status:
    - failure
    - success
    event:
      - pull_request

---
kind: pipeline
name: slack-push-start
type: docker

trigger:
  branch:
  - main
  event:
  - push
  - tag

clone:
  disable: true

steps:
- name: slack-push-start
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{repo.name}} build is starting.
      Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
  when:
    branch:
    - main
    event:
    - push
    - tag

---
kind: pipeline
name: push
type: exec

node:
  salt: server

trigger:
  branch:
  - main
  event:
  - push
  - tag

depends_on:
  - slack-push-start

clone:
  disable: true

steps:
- name: Salt Lint
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "find /srv/salt -type f -name '*.sls' -print0 | xargs -0 --no-run-if-empty salt-lint -x 204 -x 203"
  when:
    branch:
    - main
    event:
    - push
    - tag

- name: Salt Highstate 
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "sudo salt --force-color '*' --state-verbose=False state.highstate"
  when:
    branch:
    - main
    event:
    - push
    - tag

---
kind: pipeline
name: slack-push
type: docker

trigger:
  branch:
  - main
  event:
  - push
  - tag
  status:
  - success
  - failure

depends_on:
- push

clone:
  disable: true

steps:
- name: slack-push
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{#success build.status}}
        {{repo.name}} build passed.
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{else}}
        {{repo.name}} build {{build.number}} failed. Please investigate. 
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{/success}}      
  when:
    status:
    - failure
    - success
    branch:
    - main
    event:
    - push
    - tag

Compared to the Ansible pipeline, there are many similarities but also a few key differences.

The first is that the Drone file contains multiple pipelines. Each separator (the ---) is the start of a new pipeline. The reason I need to do this is because each pipeline has a type. I already have a working Slack notification step, so it makes sense to continue using this. However the Slack notification step uses Docker. As noted already, I am using the Exec runner for commands on the Salt server, which requires a use of the exec type. To use them all in the same “build” (i.e. a run through of all the steps in the Drone file), each “type” of build needs to be separated.

Also, you’ll notice the disabling of clones (i.e. performing a Git clone). Drone by default will run an implicit Clone step in every pipeline, to make code available within a container. However this doesn’t work for the Exec runners, at which point this step will fail. I also don’t need to clone any code for Slack notifications, so I disable it in every Pipeline defined in the Drone file.

Another difference is the node selector. Node selectors in Drone use key-value pairs, which are defined in the Exec Runner configuration as a label - DRONE_RUNNER_LABELS=salt:server. You can add more labels with commas, and these can also be set as environment variables rather than in a configuration file. What this means is that the Pipeline will run on nodes that match the key-value pair and nowhere else. This ensures that Drone doesn’t run the Pipeline on a server other than that which is running the Salt server.

Finally, there is the depends_on field. By default, Drone will run pipelines defined in a file in parallel. This is really useful when deploying to multiple locations, or running concurrent (parallel) steps or pipelines. However in this Drone file, sending Slack notifications at the same time as running Salt would notify before the stage is finished. Using depends_on ensures each Pipeline runs sequentially, with the status of each Pipeline affecting any Pipelines that depend on it.

Pipeline Steps

As the Slack notification steps are identical to the steps in the Ansible pipeline, I’ll cover the Salt specific steps.

Check all minions are up

- name: Check all minions are up 
  commands:
    - "sudo salt --force-color '*' test.ping"
  when:
    event:
    - pull_request

As this step (and others in the Drone file) use the Exec runner, there is no need to specify a Docker Image to use as part of the process. Instead, I supply the commands to run on the machine.

The test.ping command is used to ensure all the managed minions are up and return a response. If this fails, it means an agent is not reachable, and therefore configuration cannot be applied to all nodes at once. This ensures Salt is working everywhere before trying to apply changes.

The --force-color argument is used because if a real terminal (i.e. SSH or someone physically logged in to a machine) is not present, Salt will not display colours. Red is quite an easy colour to spot for problems, rather than the output being same colour whether a command passed or failed!

Salt Ping

Salt Lint

- name: Salt Lint
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "git checkout ${DRONE_SOURCE_BRANCH}" 
    - "find /srv/salt -type f -name '*.sls' -print0 | xargs -0 --no-run-if-empty salt-lint -x 204 -x 203"
  when:
    event:
    - pull_request

As you can see in this, the commands used are the same as those that a user would need to run directly on the machine. This is where the Exec runner can become unwieldy compared to running in Docker, as you aren’t guaranteed to be running in a clean state (i.e. no hanging commits/changes, the code is cloned fresh).

In this, the Exec runner will change directory to the Salt base path (/srv/salt in this case), checkout the main branch, run a Git pull (to bring in all previously merged changes and committed branches), and then checkout the branch opened for this Pull Request.

Finally, Drone runs salt-lint over the Salt files to ensure there are no glaring mistakes or bad practices. For example, it will catch if file permissions are not being set as part of a file creation task, or modal permissions are valid, and much more.

I ignore two rules, which are 203 - Most files should not contain tabs and 204 - Lines should be no longer than 160 chars. The first is because of setting some file contents in a file on OpenBSD that requires tabs, and the second is because I use a lot of URLs that would become really hard to read (and reuse for other states) if I started to split them down. Most other Salt Lint rules are valid for my codebase, but these two would require making changes that I don’t feel are all that beneficial.

Drone Salt Lint

Highstate Test

- name: Highstate test
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "git checkout ${DRONE_SOURCE_BRANCH}" 
    - "sudo salt --force-color '*' saltutil.pillar_refresh"
    - "sudo salt --force-color '*' state.highstate --state-verbose=False test=True"
  when:
    event:
    - pull_request

In this, the commands are similar to those in the previous steps. The main difference here is running a “pillar refresh”, and a “highstate” in test mode.

The pillar refresh is used to ensure that each host that Salt manages has the latest pillar data (analogous to host/group variables in Ansible, or custom facts in Puppet). I have had issues in the past where states haven’t applied correctly because the pillar data is not up to date on each node being managed. This makes sure that states don’t fail erroneously because the agents haven’t pulled in the latest data yet.

The highstate all states for each host that they assigned to. The test=True flag makes sure that changes are not applied (i.e. a dry run). The state-verbose=False flag tells Salt to only output tasks that have changed, rather than outputting the status of every task that ran on a node.

Drone Salt Highstate Test

Highstate

- name: Salt Highstate 
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "sudo salt --force-color '*' --state-verbose=False state.highstate"
  when:
    branch:
    - main
    event:
    - push
    - tag

This state performs a very similar function to the previous step. The main differences are: -

  • Checking out the main branch, pulling in the changes, and NOT checking out another branch (i.e. running from the primary/default branch)
  • No pillar refresh, as this happens in the previous step, at which point it is very unlikely to be out of date

This is where changes are actually applied, based upon all the new code being merged in. Other than Slack notifications, this is the final step that Drone will run.

Drone Salt Highstate

Demonstration

The below video is a demonstration of making a change to the Salt repository and committing it. In this case I had no actual changes I needed to make, so this pipeline is based upon removing a line from the README file. The same process is used when making actual changes within Salt: -

Summary

In this, we have now seen how we can make use of the Exec runners and how to use them in the same pipelines as Docker runners.

This has also shown why it is preferred to use Docker runners, as Docker runners clone the repository directly and can run commands as if the repository is the base path. Exec runners are effectively like running a script on a server, meaning that you need to think about paths, Git flow (i.e. checking out code, making sure you have the most recent branch etc) whereas this isn’t a consideration for the Docker runners.

In the next post, I’m going to cover using Drone with Terraform. This will also involve using Hashicorp’s Consul for a remote state backend.