Previous posts in the series are: -

• Introduction
• Ansible Improvements
• SaltStack Improvements
• Introduction to Drone CI
• Using Drone with Ansible
• Using Drone with Salt

This post will cover using Drone to manage and deploy resources with Terraform.

Background

When I first started using Drone, I wasn’t using it with Terraform. In fact very little of my infrastructure was managed by Terraform. Recently though, I decided to start moving most external services to it. In keeping with my theme of trying to automate my infrastructure and avoid manual changes, it made sense to finally start using Terraform for my personal usage of external services.

Ansible and Salt still control what runs on my machines (including VPSs/instances on cloud providers), but managing external services makes a lot of sense with Terraform. While Ansible and Salt are good at making changes, they don’t manage “desired state” very well, in the sense that Ansible can create a VPS on Hetzner, but it does not know that one may already exist.

With Terraform, it knows what resources it manages (using State management), and whether they should already exist or not. Terraform may still be incorrect as to the state of a resource, but usually only if the resource changed since the last time Terraform ran. Something like Ansible will just try to make the same changes it did last time, which could lead to duplicate resources, or the Playbook failing because it can’t create a resource (because it already exists and can’t be created again).

The following covers what I use: -

• Cloudflare for external DNS
  • This includes externally accessibly domains
  • It also includes “internal” domains I use that I want valid Let’s Encrypt certificates for
• Digital Ocean and Hetzner for VPSs
  • My blog and other external services (RSS, Read-It-Later, Black Box Exporter for monitoring) are hosted on these
• Google Cloud Platform and Oracle Cloud Infrastructure for the always-free virtual machines
• Lab/Proxmox for testing technologies

For most of these, the steps are the same, with some minor differences in variable/secrets for each provider). In some cases, extra steps are required. I’ll first show the most basic pipeline (the one used for Cloudflare DNS) and then show some of the extra steps in others.

Full Drone Pipeline - Cloudflare

kind: pipeline
name: default
type: docker

trigger:
  branch:
    - main

steps:
  - name: Terraform FMT PR
    image: jmccann/drone-terraform:latest
    settings:
      actions:
        - fmt
      fmt_options:
        write: false
        diff: true
        check: true
    when:
      event:
      - pull_request

  - name: Terraform Plan
    image: jmccann/drone-terraform:latest
    settings:
      actions:
        - validate
        - plan
    environment:
      DIGITALOCEAN_TOKEN:
        from_secret: digitalocean_token 
      HCLOUD_TOKEN:
        from_secret: hcloud_token
      CLOUDFLARE_API_TOKEN:
        from_secret: cloudflare_api_token
    when:
      event:
      - pull_request

  - name: slack-pr
    image: plugins/slack
    settings:
      webhook:
        from_secret: drone_builds_slack_webhook 
      channel: builds
      template: >
        {{#success build.status}}
          {{repo.name}} PR build passed. 
          Merge in to apply.
          PR: https://git.noisepalace.co.uk/YetiOps/{{repo.name}}/pulls/{{build.pull}}
          Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
        {{else}}
          {{repo.name}} PR build failed. 
          Please investigate. 
          PR: https://git.noisepalace.co.uk/YetiOps/{{repo.name}}/pulls/{{build.pull}}
          Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
        {{/success}}            
    when:
      status:
      - failure
      - success
      event:
        - pull_request

  - name: slack-push-start
    image: plugins/slack
    settings:
      webhook:
        from_secret: drone_builds_slack_webhook 
      channel: builds
      template: >
        {{repo.name}} build is starting.
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}      
    when:
      branch:
      - main
      event:
      - push
      - tag
  
  - name: Terraform FMT
    image: jmccann/drone-terraform:latest
    settings:
      actions:
        - fmt
      fmt_options:
        write: false
        diff: true
        check: true
    when:
      branch:
      - main
      event:
      - push
      - tag


  - name: Terraform Apply
    image: jmccann/drone-terraform:latest
    settings:
      actions:
        - validate
        - plan
        - apply
    environment:
      DIGITALOCEAN_TOKEN:
        from_secret: digitalocean_token 
      HCLOUD_TOKEN:
        from_secret: hcloud_token
      CLOUDFLARE_API_TOKEN:
        from_secret: cloudflare_api_token
    when:
      branch:
      - main
      event:
      - push
      - tag

  - name: slack-push
    image: plugins/slack
    settings:
      webhook:
        from_secret: drone_builds_slack_webhook 
      channel: builds
      template: >
        {{#success build.status}}
          {{repo.name}} build passed.
          Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
        {{else}}
          {{repo.name}} build {{build.number}} failed. Please investigate. 
          Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
        {{/success}}            
    when:
      status:
      - failure
      - success
      branch:
      - main
      event:
      - push
      - tag

This pipeline is very similar to the one used for Ansible. This uses the same Slack notification steps, the same kind of triggers/conditionals for running steps, and runs as a single Docker-based pipeline (rather than the multiple pipelines required for Salt).

The bulk of the Terraform-specific steps in the pipelines use the Terraform Drone plugin. It is possible to use the official Hashicorp Terraform Docker image, but the Drone plugin is a little more convenient. If you use the official Hashicorp Terraform Docker image, you need to define each command you want to run, whereas the plugin reduces most commands to fields.

Steps - Terraform FMT

- name: Terraform FMT PR
  image: jmccann/drone-terraform:latest
  settings:
    actions:
      - fmt
    fmt_options:
      write: false
      diff: true
      check: true
  when:
    event:
    - pull_request

This step is the same as running the terraform fmt command with the flags -diff and -check. While this isn’t a necessary step, it does apply some consistent formatting rules to Terraform files.

In the diff and check mode, it doesn’t make any changes, but will fail if terraform fmt mentions any required changes (as it will exit with a non-zero exit code).

Steps - Terraform Plan

  - name: Terraform Plan
    image: jmccann/drone-terraform:latest
    settings:
      actions:
        - validate
        - plan
    environment:
      DIGITALOCEAN_TOKEN:
        from_secret: digitalocean_token 
      HCLOUD_TOKEN:
        from_secret: hcloud_token
      CLOUDFLARE_API_TOKEN:
        from_secret: cloudflare_api_token
    when:
      event:
      - pull_request

This step runs terraform validate and terraform plan. Validate ensures that the Terraform configuration is correct (i.e. brackets in the correct place, variables defined correctly etc), and the Plan stage shows the changes Terraform wants to make.

In this Drone expose a set of environment variables, namely my Digital Ocean API token, Hetzner HCloud API token and the Cloudflare API token. It uses the Digital Ocean and Hetzner tokens to create DNS records for the public IPs of my VPSs within Hetzner and Digital Ocean, and the Cloudflare API token is used to read/write updates to the Cloudflare API. An example of the relevant Terraform code is shown below: -

data.tf

data "hcloud_server" "hcloud-vps-a" {
  name = "vps-shme-hcloud-a"
}

data "hcloud_server" "hcloud-vps-b" {
  name = "vps-shme-hcloud-b"
}

data "digitalocean_droplet" "vps-shme" {
  name = "vps-shme"
}

main.tf

# VPS Records - IPv4

resource "cloudflare_record" "vps-a" {
  zone_id = local.this_zone_id
  name    = "vps-a"
  type    = "A"
  proxied = "false"
  value   = data.hcloud_server.hcloud-vps-a.ipv4_address
}

resource "cloudflare_record" "vps-b" {
  zone_id = local.this_zone_id
  name    = "vps-b"
  type    = "A"
  proxied = "false"
  value   = data.hcloud_server.hcloud-vps-b.ipv4_address
}

resource "cloudflare_record" "vps-shme" {
  zone_id = local.this_zone_id
  name    = "vps"
  type    = "A"
  proxied = "false"
  value   = data.digitalocean_droplet.vps-shme.ipv4_address
}

# VPS Records - IPv6

resource "cloudflare_record" "vps-a-v6" {
  zone_id = local.this_zone_id
  name    = "vps-a"
  type    = "AAAA"
  proxied = "false"
  value   = data.hcloud_server.hcloud-vps-a.ipv6_address
}

resource "cloudflare_record" "vps-b-v6" {
  zone_id = local.this_zone_id
  name    = "vps-b"
  type    = "AAAA"
  proxied = "false"
  value   = data.hcloud_server.hcloud-vps-b.ipv6_address
}

resource "cloudflare_record" "vps-shme-v6" {
  zone_id = local.this_zone_id
  name    = "vps"
  type    = "AAAA"
  proxied = "false"
  value   = data.digitalocean_droplet.vps-shme.ipv6_address
}

As you can see, this uses data sources to get information from Digital Ocean and Hetzner, and then applies the IPv4/IPv6 addresses as values in cloudflare_record resources.

I also create DNS records for instances within Google Cloud Platform and Oracle Cloud Infrastructure, but with these I use Terraform Remote State. The reason for this is that both of these require extra configuration and files (GPG keys for OCI, a JSON document for GCP) that add extra steps and complication to the pipeline. While this does mean that I am reliant on the Terraform Remote State being up to date (rather than a direct data source like for Digital Ocean or Hetzner), I accept this trade off to avoid the additional complexity.

The relevant Terraform for the GCP and OCI records are shown below: -

data.tf

data "terraform_remote_state" "oracle-ork" {
  backend = "consul"
  config = {
    address = "consul.noisepalace.co.uk"
    scheme  = "https"
    path    = "terraform/oraclecloud/oci"
  }
}

data "terraform_remote_state" "yetiops-goggle" {
  backend = "consul"
  config = {
    address = "consul.noisepalace.co.uk"
    scheme  = "https"
    path    = "terraform/gcp/yetiops-goggle"
  }
}

main.tf

resource "cloudflare_record" "vps-ork-01" {
  zone_id = local.this_zone_id
  name    = "ork-01"
  type    = "A"
  proxied = "false"
  value   = data.terraform_remote_state.oracle-ork.outputs.instance_public_ip
}

resource "cloudflare_record" "vps-gog-01" {
  zone_id = local.this_zone_id
  name    = "gog-01"
  type    = "A"
  proxied = "false"
  value   = data.terraform_remote_state.yetiops-goggle.outputs.gog-01-ipv4
}

These use Consul as the source for Remote State. More details on this are in this section.

The Drone step will run a plan against these resources (and all other defined resources), and show what changes need to be made (if any).

Steps - Terraform Apply

  - name: Terraform Apply
    image: jmccann/drone-terraform:latest
    settings:
      actions:
        - validate
        - plan
        - apply
    environment:
      DIGITALOCEAN_TOKEN:
        from_secret: digitalocean_token 
      HCLOUD_TOKEN:
        from_secret: hcloud_token
      CLOUDFLARE_API_TOKEN:
        from_secret: cloudflare_api_token
    when:
      branch:
      - main
      event:
      - push
      - tag

This step is almost identical to the Plan stage. The only difference is that it uses the apply action as well. This runs a validate and a plan action (to ensure that the code is still valid when merged with the main branch) and then applies the changes.

This is all controlled via Gitea pull requests, meaning that any PRs raised will go through all the validation and planning steps. Changes are then applied without any human interaction with the Terraform CLI itself.

For most Terraform code, these are all the steps you will need.

Additional steps

As noted above, most of my Terraform code doesn’t need anything more than the steps covered already. The secrets to expose may differ (based upon the infrastructure used), otherwise everything else is the same.

However for some Terraform code, I need to make files available (e.g. SSH keys, GPG keys, configuration objects). The following step covers how to do this: -

steps:
  - name: Place SSH keys - PR
    image: alpine
    environment: 
      SSH_PRIV:
        from_secret: drone_ssh_priv
      SSH_PUB:
        from_secret: drone_ssh_pub
      OCI_PRIVATE_KEY:
        from_secret: oci_private_key
    volumes:
      - name: cache
        path: /ssh
    commands:
      - echo -e "$SSH_PRIV" | tee /ssh/id_rsa
      - echo -e "$SSH_PUB" | tee /ssh/id_rsa.pub
      - echo -e "$OCI_PRIVATE_KEY" | tee /ssh/oci_private_key.pem
      - chmod 644 /ssh/*
    when:
      event:
      - pull_request

[...]

volumes:
  - name: cache
    temp: {}

This step takes the contents of some secrets, and then places them into files. In the above, these are my Drone SSH keys, and also the OCI GPG private key. These are placed in a volume. A volume is a shared directory/path that can be made available to other steps within a pipeline.

For Oracle Cloud Infrastructure, the GPG key is required to authenticate against the API, whereas the SSH keys (specifically the public key) are used in the cloud-init/user data to bootstrap the instances. By default they do not allow password-based authentication to login via SSH, so they need some form of SSH key to allow you to login.

An example of using this is below: -

- name: Terraform Plan
  image: jmccann/drone-terraform:latest
  settings:
    actions:
      - validate
      - plan
    vars:
      ssh_public_key: "/ssh/id_rsa.pub"
  environment:
      TF_VAR_tenancy_ocid: 
        from_secret: oci_tenancy_ocid
      TF_VAR_user_ocid: 
        from_secret: oci_user_ocid
      TF_VAR_compartment_ocid: 
        from_secret: oci_compartment_ocid
      TF_VAR_fingerprint: 
        from_secret: oci_fingerprint
      TF_VAR_private_key_path: "/ssh/oci_private_key.pem"
  volumes:
    - name: cache
      path: /ssh
  when:
    event:
    - pull_request

As you can see, the SSH key path is supplied as a variable. We also supply the OCI private key path as a TF_VAR environment variable. The most important part here is the volumes section, as without this the keys created in the previous step would not be available.

Volumes could be used in many different ways (e.g. cached dependencies, creating build artifacts and pushing them). In this case it make files available between steps, allowing authentication to certain providers.

The same step is used in my Lab Terraform when creating Proxmox virtual machines. This is because cloud-init in Proxmox requires adding files directly to the Proxmox host’s file system (rather than being stored in an API). In my case, I use SSH/SCP within Terraform to transfer the files to the Proxmox host(s), using the Drone SSH keys for authentication.

Remote State Backend with Consul

While not specific to Drone, using some form of Remote State Backend within Terraform allows the following: -

• A central place to source state between Drone jobs and local Terraform testing
• Using Remote State as a data source in Terraform code
• State Locking - avoids multiple jobs/people applying changes at the same time (and potentially breaking each others changes)
  • Not all backends support this, and some require a separate backend for locking (e.g. AWS S3 for state storage, and DynamoDB for state locking)
• Most importantly, it means Drone does not need to write to a terraform.tfstate file in the repository and push it back to the repository after changes are made

The last point is especially pertinent, as without this the following steps would be required: -

• Create a cache/state volume in the pipeline
• Make the Terraform Drone plugin place state files in the volume
• Run a step after each Terraform Apply stage to add, commit and push changes back to the code repository
  • This also means giving Drone access to push (and not just pull/clone rights) access to the repository

Instead, using a Remote State Backend means none of these extra steps are required. Many different options for Backends are available (e.g. AWS S3, Azure Blob Storage, etcd, Postgres), but I chose Consul as I already run Consul for Prometheus Service Discovery.

Setting up this backend looks like the below: -

terraform {
  backend "consul" {
    address = "consul.noisepalace.co.uk"
    scheme  = "https"
    path    = "terraform/oraclecloud/oci"
  }
}

The path structure is arbitrary. I decided to use a $APPLICATION/$PROVIDER/$PURPOSE structure, but you could name each one after characters in Star Wars or Transformers if you wished! This stores state in the Consul KV store, as shown below: -

Demonstration

The below video is a demonstration of making a change to the Terraform repository and committing it. In this step I am going to create a DNS record in Cloudflare: -

Summary

In this, we have seen how to can make use of Drone to apply Terraform configuration.

This has also shown how we can make use of Consul to provide a consistent Backend for Terraform, reducing the need for additional steps to manage Terraform state files with Drone.

In the next post, I’m going to cover using Drone to build Go releases.

Previous posts in the series are: -

• Introduction
• Ansible Improvements
• SaltStack Improvements
• Introduction to Drone CI
• Using Drone with Ansible

As we have already covered how to use set up Drone CI, as well as a working pipeline with Ansible, it’s time to show it working with Salt.

Background

As mentioned before, I use Ansible to bootstrap nodes, set up monitoring of devices that can’t be managed by Salt, build my DNS and DHCP configuration and some other minor.

For most other tasks though, I use Salt to manage my machines. This covers everything from setting up Prometheus and exporters, NFS, Promtail/Loki, Samba, Wireguard, Gitea and more.

Differences

Because Salt (unlike Ansible) uses agents and a central Salt server to apply changes to nodes, Drone can’t run Salt directly from within a container. I could configure Salt Masterless, but this requires a lot of extra work (including cloning the entire Salt configuration to each managed node), removing the speed benefits that Salt has over Ansible.

With this being the case, Drone needs some way of access the Salt central server (known as a “master”) to check, test and apply changes.

In the Introduction to Drone CI post, I referred to different kinds of runners (Docker, Exec, SSH) that perform different functions. In this case, Drone will use the Exec Runner. The Exec Runner can run commands directly on a machine (rather than inside of a Docker container), which fits with how to manage Salt.

Full Drone Pipeline

---
kind: pipeline
name: pr
type: exec

clone:
  disable: true

node:
  salt: server

trigger:
  branch:
  - main
  event:
  - pull_request

steps:
- name: Check all minions are up 
  commands:
    - "sudo salt --force-color '*' test.ping"
  when:
    event:
    - pull_request

- name: Salt Lint
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "git checkout ${DRONE_SOURCE_BRANCH}" 
    - "find /srv/salt -type f -name '*.sls' -print0 | xargs -0 --no-run-if-empty salt-lint -x 204 -x 203"
  when:
    event:
    - pull_request

- name: Highstate test
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "git checkout ${DRONE_SOURCE_BRANCH}" 
    - "sudo salt --force-color '*' saltutil.pillar_refresh"
    - "sudo salt --force-color '*' state.highstate --state-verbose=False test=True"
  when:
    event:
    - pull_request

---
kind: pipeline
name: slack-pr
type: docker

clone:
  disable: true

depends_on:
- pr

trigger:
  branch:
  - main
  event:
  - pull_request
  status:
  - success
  - failure

steps:
- name: slack-pr
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{#success build.status}}
        {{repo.name}} PR build passed. 
        Merge in to apply.
        PR: https://git.noisepalace.co.uk/YetiOps/{{repo.name}}/pulls/{{build.pull}}
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{else}}
        {{repo.name}} PR build failed. 
        Please investigate. 
        PR: https://git.noisepalace.co.uk/YetiOps/{{repo.name}}/pulls/{{build.pull}}
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{/success}}      
  when:
    status:
    - failure
    - success
    event:
      - pull_request

---
kind: pipeline
name: slack-push-start
type: docker

trigger:
  branch:
  - main
  event:
  - push
  - tag

clone:
  disable: true

steps:
- name: slack-push-start
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{repo.name}} build is starting.
      Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
  when:
    branch:
    - main
    event:
    - push
    - tag

---
kind: pipeline
name: push
type: exec

node:
  salt: server

trigger:
  branch:
  - main
  event:
  - push
  - tag

depends_on:
  - slack-push-start

clone:
  disable: true

steps:
- name: Salt Lint
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "find /srv/salt -type f -name '*.sls' -print0 | xargs -0 --no-run-if-empty salt-lint -x 204 -x 203"
  when:
    branch:
    - main
    event:
    - push
    - tag

- name: Salt Highstate 
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "sudo salt --force-color '*' --state-verbose=False state.highstate"
  when:
    branch:
    - main
    event:
    - push
    - tag

---
kind: pipeline
name: slack-push
type: docker

trigger:
  branch:
  - main
  event:
  - push
  - tag
  status:
  - success
  - failure

depends_on:
- push

clone:
  disable: true

steps:
- name: slack-push
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{#success build.status}}
        {{repo.name}} build passed.
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{else}}
        {{repo.name}} build {{build.number}} failed. Please investigate. 
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{/success}}      
  when:
    status:
    - failure
    - success
    branch:
    - main
    event:
    - push
    - tag

Compared to the Ansible pipeline, there are many similarities but also a few key differences.

The first is that the Drone file contains multiple pipelines. Each separator (the ---) is the start of a new pipeline. The reason I need to do this is because each pipeline has a type. I already have a working Slack notification step, so it makes sense to continue using this. However the Slack notification step uses Docker. As noted already, I am using the Exec runner for commands on the Salt server, which requires a use of the exec type. To use them all in the same “build” (i.e. a run through of all the steps in the Drone file), each “type” of build needs to be separated.

Also, you’ll notice the disabling of clones (i.e. performing a Git clone). Drone by default will run an implicit Clone step in every pipeline, to make code available within a container. However this doesn’t work for the Exec runners, at which point this step will fail. I also don’t need to clone any code for Slack notifications, so I disable it in every Pipeline defined in the Drone file.

Another difference is the node selector. Node selectors in Drone use key-value pairs, which are defined in the Exec Runner configuration as a label - DRONE_RUNNER_LABELS=salt:server. You can add more labels with commas, and these can also be set as environment variables rather than in a configuration file. What this means is that the Pipeline will run on nodes that match the key-value pair and nowhere else. This ensures that Drone doesn’t run the Pipeline on a server other than that which is running the Salt server.

Finally, there is the depends_on field. By default, Drone will run pipelines defined in a file in parallel. This is really useful when deploying to multiple locations, or running concurrent (parallel) steps or pipelines. However in this Drone file, sending Slack notifications at the same time as running Salt would notify before the stage is finished. Using depends_on ensures each Pipeline runs sequentially, with the status of each Pipeline affecting any Pipelines that depend on it.

Pipeline Steps

As the Slack notification steps are identical to the steps in the Ansible pipeline, I’ll cover the Salt specific steps.

Check all minions are up

- name: Check all minions are up 
  commands:
    - "sudo salt --force-color '*' test.ping"
  when:
    event:
    - pull_request

As this step (and others in the Drone file) use the Exec runner, there is no need to specify a Docker Image to use as part of the process. Instead, I supply the commands to run on the machine.

The test.ping command is used to ensure all the managed minions are up and return a response. If this fails, it means an agent is not reachable, and therefore configuration cannot be applied to all nodes at once. This ensures Salt is working everywhere before trying to apply changes.

The --force-color argument is used because if a real terminal (i.e. SSH or someone physically logged in to a machine) is not present, Salt will not display colours. Red is quite an easy colour to spot for problems, rather than the output being same colour whether a command passed or failed!

Salt Lint

- name: Salt Lint
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "git checkout ${DRONE_SOURCE_BRANCH}" 
    - "find /srv/salt -type f -name '*.sls' -print0 | xargs -0 --no-run-if-empty salt-lint -x 204 -x 203"
  when:
    event:
    - pull_request

As you can see in this, the commands used are the same as those that a user would need to run directly on the machine. This is where the Exec runner can become unwieldy compared to running in Docker, as you aren’t guaranteed to be running in a clean state (i.e. no hanging commits/changes, the code is cloned fresh).

In this, the Exec runner will change directory to the Salt base path (/srv/salt in this case), checkout the main branch, run a Git pull (to bring in all previously merged changes and committed branches), and then checkout the branch opened for this Pull Request.

Finally, Drone runs salt-lint over the Salt files to ensure there are no glaring mistakes or bad practices. For example, it will catch if file permissions are not being set as part of a file creation task, or modal permissions are valid, and much more.

I ignore two rules, which are 203 - Most files should not contain tabs and 204 - Lines should be no longer than 160 chars. The first is because of setting some file contents in a file on OpenBSD that requires tabs, and the second is because I use a lot of URLs that would become really hard to read (and reuse for other states) if I started to split them down. Most other Salt Lint rules are valid for my codebase, but these two would require making changes that I don’t feel are all that beneficial.

Highstate Test

- name: Highstate test
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "git checkout ${DRONE_SOURCE_BRANCH}" 
    - "sudo salt --force-color '*' saltutil.pillar_refresh"
    - "sudo salt --force-color '*' state.highstate --state-verbose=False test=True"
  when:
    event:
    - pull_request

In this, the commands are similar to those in the previous steps. The main difference here is running a “pillar refresh”, and a “highstate” in test mode.

The pillar refresh is used to ensure that each host that Salt manages has the latest pillar data (analogous to host/group variables in Ansible, or custom facts in Puppet). I have had issues in the past where states haven’t applied correctly because the pillar data is not up to date on each node being managed. This makes sure that states don’t fail erroneously because the agents haven’t pulled in the latest data yet.

The highstate all states for each host that they assigned to. The test=True flag makes sure that changes are not applied (i.e. a dry run). The state-verbose=False flag tells Salt to only output tasks that have changed, rather than outputting the status of every task that ran on a node.

Highstate

- name: Salt Highstate 
  commands:
    - "cd /srv/salt"
    - "git checkout main"
    - "sudo -u stuh84 git pull"
    - "sudo salt --force-color '*' --state-verbose=False state.highstate"
  when:
    branch:
    - main
    event:
    - push
    - tag

This state performs a very similar function to the previous step. The main differences are: -

• Checking out the main branch, pulling in the changes, and NOT checking out another branch (i.e. running from the primary/default branch)
• No pillar refresh, as this happens in the previous step, at which point it is very unlikely to be out of date

This is where changes are actually applied, based upon all the new code being merged in. Other than Slack notifications, this is the final step that Drone will run.

Demonstration

The below video is a demonstration of making a change to the Salt repository and committing it. In this case I had no actual changes I needed to make, so this pipeline is based upon removing a line from the README file. The same process is used when making actual changes within Salt: -

Summary

In this, we have now seen how we can make use of the Exec runners and how to use them in the same pipelines as Docker runners.

This has also shown why it is preferred to use Docker runners, as Docker runners clone the repository directly and can run commands as if the repository is the base path. Exec runners are effectively like running a script on a server, meaning that you need to think about paths, Git flow (i.e. checking out code, making sure you have the most recent branch etc) whereas this isn’t a consideration for the Docker runners.

In the next post, I’m going to cover using Drone with Terraform. This will also involve using Hashicorp’s Consul for a remote state backend.

Previous posts in the series are: -

• Introduction
• Ansible Improvements
• SaltStack Improvements
• Introduction to Drone CI

Now that we have a better Ansible and Salt codebase, and have Drone working, we can start to look at some Drone pipelines! In this post, I will cover using Ansible with Drone.

Background

As mentioned in previous posts, I currently use Ansible to manage bootstrapping machines in my home infrastructure, as well as managing DNS, DHCP, IP address management, and adding monitoring/Prometheus checks for hosts not managed by Salt.

In addition, I have also recently started managing my lab environments (usually for Proof-of-Concepts, or even building infrastructure to use in posts on this site) using Ansible too.

To manage all of this before, it required manual updates, running multiple playbooks from the command line, all while logged in to a single server.

By creating pipelines in Drone, I can now commit changes directly to Git (from any machine, or even a phone/tablet) and the changes will roll out automatically.

Steps

The flow I want to use for making changes is as follows: -

1. Create a branch in Git
2. Make the relevant changes
3. Commit the branch to Git
4. Raise a Pull Request
5. Before merging, have the Ansible playbooks go through a syntax check and linting, as well as a dry run
6. Send a notification of success or failure
7. Show the results of the dry run
8. Merge the Pull Request
9. Apply the changes
10. Send a notification of success or failure

This seems like a lot of steps, but this flow is how I tend to work anyway (personally and professionally). Drone adds the automated testing and applying the changes.

Full Drone Pipeline

The below is the full pipeline: -

kind: pipeline
name: default
type: docker

trigger:
  branch:
    - main

steps:
- name: Syntax Check 
  image: plugins/ansible
  settings:
    become: true
    playbook: playbook.yml
    inventory: inventory
    private_key:
      from_secret: drone_ssh_priv
    ssh_extra_args: "-o StrictHostKeyChecking=no"
    requirements: requirements.txt
    galaxy: requirements.yml
    syntax_check: true
  when:
    event:
    - pull_request

- name: Lint
  image: cytopia/ansible-lint
  commands:
    - ansible-lint playbook.yml --force-color
  when:
    event:
    - pull_request

- name: Show Diff and Check
  image: plugins/ansible
  settings:
    become: true
    playbook: playbook.yml
    inventory: inventory
    private_key:
      from_secret: drone_ssh_priv
    diff: true
    check: true
    ssh_extra_args: "-o StrictHostKeyChecking=no"
    requirements: requirements.txt
    galaxy: requirements.yml
  when:
    event:
    - pull_request

- name: slack-pr
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{#success build.status}}
        {{repo.name}} PR build passed. 
        Merge in to apply.
        PR: https://git.noisepalace.co.uk/YetiOps/{{repo.name}}/pulls/{{build.pull}}
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{else}}
        {{repo.name}} PR build failed. 
        Please investigate. 
        PR: https://git.noisepalace.co.uk/YetiOps/{{repo.name}}/pulls/{{build.pull}}
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{/success}}      
  when:
    status:
    - failure
    - success
    event:
      - pull_request

- name: slack-push-start
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{repo.name}} build is starting.
      Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
  when:
    branch:
    - main
    event:
    - push
    - tag

- name: Apply Playbook 
  image: plugins/ansible
  settings:
    become: true
    playbook: playbook.yml
    inventory: inventory
    private_key:
      from_secret: drone_ssh_priv
    ssh_extra_args: "-o StrictHostKeyChecking=no"
    requirements: requirements.txt
    galaxy: requirements.yml
    diff: true
  when:
    branch:
    - main
    event:
    - push
    - tag

- name: slack-push
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{#success build.status}}
        {{repo.name}} build passed.
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{else}}
        {{repo.name}} build {{build.number}} failed. Please investigate. 
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{/success}}      
  when:
    status:
    - failure
    - success
    branch:
    - main
    event:
    - push
    - tag

To cover a few of the terms: -

• kind - This defines the kind of Drone execution. This is almost always going to be pipeline (although template is another option)
• type - This is the kind of execution. This could be docker, exec, ssh, kubernetes and many more
• trigger - This defines what conditions need to be met for a pipeline to start
• steps - These are the steps that the pipeline will take

The trigger step in this pipeline specifies that it will only run if the destination branch of the Pull Request is main, or commits directly to main. Pull Requests between other branches (e.g. dev to prod) would not be covered by this.

Pipeline Steps

I’ll now go through each step to show what they do.

Syntax Check

steps:
- name: Syntax Check 
  image: plugins/ansible
  settings:
    become: true
    playbook: playbook.yml
    inventory: inventory
    private_key:
      from_secret: drone_ssh_priv
    ssh_extra_args: "-o StrictHostKeyChecking=no"
    requirements: requirements.txt
    galaxy: requirements.yml
    syntax_check: true
  when:
    event:
    - pull_request

This step details the following: -

• The name of the step
• The Plugin (a.k.a. Docker Image) used to execute the step
• Any settings specific to the plugin
• When to execute the step

All available Plugins for Drone can be found here. In our case, we use a plugin (Docker Image) that will run Ansible. We also specify a few settings: -

• become: true - This states that we will use privilege escalation (e.g. sudo, doas) for tasks
  • This is often a requirement when installing packages and/or updating configuration of system daemons
• playbook: playbook.yml - This specifies the path for the playbook. In this repository, it is in the base path
• inventory: inventory - This specifies the inventory file for Ansible, which in this case is a file in the base path called inventory
• ssh_extra_args - This specifies any extra arguments to the Ansible SSH client process
• requirements: requirements.txt - This specifies any Python dependencies required for the pipeline to run
• galaxy: requirements.yml - This specifies any Ansible Galaxy roles (and now Collections) to install before running the Playbooks
• private_key - This section specifies the SSH private key used by Ansible/Drone to connect to machines in the inventory
• syntax_check: true - This tells Ansible to only check that the syntax of the roles and playbooks are correct, not actual running any steps

SSH Extra Arguments

The extra argument supplied here is -o StrictHostKeyChecking=no. This tells Ansible to ignore if the host SSH key is unknown.

In most cases, you don’t want to enable this as a host key changing may indicate a machine has been compromised. However because the Docker containers that apply the changes are brand new on every run, they would have no previous knowledge of any hosts. For Ansible to run from a container, we must ignore strict host key checking.

Private Key

The private key is created on another machine using ssh-keygen -t rsa. We add the public key as an authorized keys on all nodes we want to manage. In my case, this is done using an Ansible role (that Drone also controls the execution of).

The contents of the private key are then added to a secret within Drone.

Drone Secrets can be repository specific, or common across a Git organisation. The below shows this: -

We can then source the contents of this secret within a Drone pipeline.

When clause

The when field says when this action will take place. In this task, it says that it will take action on Pull Requests. Combined with the trigger on the main branch set across all steps in the Pipeline, this means that this will only take effect on Pull Requests into the main branch.

Lint

The lint step is below: -

- name: Lint
  image: cytopia/ansible-lint
  commands:
    - ansible-lint playbook.yml --force-color
  when:
    event:
    - pull_request

This uses the ansible-lint package to check that we have used the correct syntax, loops are defined correctly, variables are defined correctly and many other rules too.

Rather than supplying configuration settings (like in the previous step), this just runs a command in a Docker image against our Playbook.

Show Diff and Check

- name: Show Diff and Check
  image: plugins/ansible
  settings:
    become: true
    playbook: playbook.yml
    inventory: inventory
    private_key:
      from_secret: drone_ssh_priv
    diff: true
    check: true
    ssh_extra_args: "-o StrictHostKeyChecking=no"
    requirements: requirements.txt
    galaxy: requirements.yml
  when:
    event:
    - pull_request

This step is almost identical to the Syntax Check stage, but with some minor differences: -

• We do not set syntax_check to true
• We add the diff: true and check: true flags

This will do a Dry Run of the playbook (i.e. shows what actions would take place) and will also show a diff (i.e. the changes) between what already exists and what would change.

This allows us to see what actions would take place, without them actually being applied. We can then make a judgement call on whether the changes are correct before merging and applying the code.

Slack PR

- name: slack-pr
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{#success build.status}}
        {{repo.name}} PR build passed. 
        Merge in to apply.
        PR: https://git.noisepalace.co.uk/YetiOps/{{repo.name}}/pulls/{{build.pull}}
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{else}}
        {{repo.name}} PR build failed. 
        Please investigate. 
        PR: https://git.noisepalace.co.uk/YetiOps/{{repo.name}}/pulls/{{build.pull}}
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{/success}}            
  when:
    status:
    - failure
    - success
    event:
      - pull_request

This task is used to send notifications to a Slack instance. We have a template which includes relevant details (i.e. links to the build job and the pull request), and also shows a slightly different message based upon whether the job was successful or not.

One point to note here is that in the when section, we match on a status of success or failure. Most pipeline steps will not execute if a previous step has failed, which would mean we wouldn’t receive notifications on failure (only success). Adding this condition means it will execute even if a previous step failed.

Slack Push Start

- name: slack-push-start
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{repo.name}} build is starting.
      Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}      
  when:
    branch:
    - main
    event:
    - push
    - tag

This step is almost identical to the previous step, except that it matches upon a push to the main branch, or a tag being applied to a commit in the main branch.

This allows us to notify that a build has started.

Apply Playbook

- name: Apply Playbook 
  image: plugins/ansible
  settings:
    become: true
    playbook: playbook.yml
    inventory: inventory
    private_key:
      from_secret: drone_ssh_priv
    ssh_extra_args: "-o StrictHostKeyChecking=no"
    requirements: requirements.txt
    galaxy: requirements.yml
    diff: true
  when:
    branch:
    - main
    event:
    - push
    - tag

This step is identical to the Show Diff and Check step, except that we do not use the check field. This will apply all the changes, as well as showing a diff of all changes (rather than only showing that tasks were executed).

We have also changed when this will run, which again is based upon a push to the main branch, or a tagged commit.

Slack Push

- name: slack-push
  image: plugins/slack
  settings:
    webhook:
      from_secret: drone_builds_slack_webhook 
    channel: builds
    template: >
      {{#success build.status}}
        {{repo.name}} build passed.
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{else}}
        {{repo.name}} build {{build.number}} failed. Please investigate. 
        Build: https://drone.noisepalace.co.uk/YetiOps/{{repo.name}}/{{build.number}}
      {{/success}}            
  when:
    status:
    - failure
    - success
    branch:
    - main
    event:
    - push
    - tag

This final step sends a notification that the Playbook apply succeeded or failed. Again, this is based upon a push to the main branch (i.e. a merged pull request, or a direct push to the branch) or a tagged commit.

Demonstration

The below video is a demonstration of making a simple change (adding a CNAME to DNS), and seeing Gitea and Drone work together to apply the changes: -

Summary

This has demonstrated how to use Ansible with Drone. I currently use this for multiple repositories, and will probably add more to it in future as well.

The next post will cover how I use Drone CI with SaltStack, which involves the Docker runner and the Exec runner.

In the previous posts I covered the improvements to my Ansible Playbooks (i.e. a complete overhaul of them) and the improvements to my Saltstack environment. Both of these were pre-requisites to being able to implement a Continuous Integration environment on my infrastructure.

This post is going to cover the choices of CI environments I could use, and why I chose Drone.

The choices

In terms of continuous integration, there are a number of different solutions to fit every kind of infrastructure, budget and need. For example, if all of your code is on GitHub, an obvious choice would be GitHub Actions. The same can be said of the Gitlab CI offering as well.

If you are more geared towards Windows deployment, you may choose something like TeamCity. If you are in AWS you may choose CodePipelines and CodeDeploy. If you are in Azure, you may choose Azure DevOps. The options are endless.

With this being the case, I needed to set out what I wanted in a CI tool: -

• Self-hosted
  • I have a homelab, so something I can run on it is preferable
• Resource usage isn’t huge
  • My homelab is a 2012 Mac Mini and a 6th generation NUC, so resources are enough but not unlimited
• It integrates with my chosen Git repository provider
• The pipelines are written in a simple language/format

Before I found Drone, the two main contenders were Gitlab CI and Jenkins.

Jenkins?

I only briefly considered Jenkins. Jenkins has it’s place, but it isn’t what I was after.

I’m not a huge fan of the Groovy language for writing pipelines. Also the resource usage can be quite significant, especially if one of the plugins is poorly written.

I have supported Jenkins at multiple businesses, and always run into issues with the plugin system. Upgrading one can break others, or upgrading one core plugin means every other plugin needs updating too.

Due to this, it was easy for me to rule out Jenkins.

Gitlab CI?

Gitlab CI seemed like the obvious choice at this point. It also helps that I have integrated Gitlab CI before at one of my previous employers, so I already had a lot of experience with it.

However, after running just self-hosted Gitlab on a virtual machine on my infrastructure for a while, I found the resource usage to be a little too much. Even at idle, the machine used a lot of memory and CPU (at least compared to the other machines on my infrastructure).

I subsequently removed Gitlab from my personal infrastructure. With this being the case, I needed to move all my repositories from my self-hosted Gitlab instance into another Git system. Thankfully,

I already have a Gitea instance running on my infrastructure, so after migrating all the repositories to it, I began the search for a CI system that works with Gitea

What works with Gitea?

Gitea is a self-hosted Git service, providing a web interface, API and standard Git forge utilities (i.e. issues, pull requests, webhooks to external systems etc). However Gitea does not (at the time of writing) have an integrated CI system (unlike GitHub or Gitlab).

After a little research, I found the Awesome-Gitea repository. This contains a list of useful utilities that can integrate with Gitea, or augment the functionality. In this list were a few CI systems that integrate natively with Gitea.

The one that caught my eye was Drone. It is built in Go (so had a good chance of low resource usage), the pipelines are written in YAML, it is self-hosted, and integrates natively with Gitea.

I decided to give Drone a try.

Setting up the Drone Server

Drone has two versions, the Community edition and the Enterprise edition. I chose the Community edition as I didn’t need any of the extra features that the Enterprise edition provides (like clustering, ephemeral agents/runners, autoscaling).

To install the server, Drone provide documentation for integration with GitHub, Gitlab, Gogs (what Gitea is forked from), Gitea, Bitbucket and Gitee.

The installation uses Docker, so you need to have a system that supports containers (Podman will probably work, although I haven’t tried it). You also to generate some secrets and an OAuth Application settings up in Gitea.

Creating the OAuth Application in Gitea

To create an OAuth application in Gitea, click on your user icon in the top right hand corner, and click settings: -

From here, choose Applications: -

You should now see the option to Manage OAuth2 Applications. In the fields provided, add a name (e.g. drone-ci), and the URL Drone will run on followed by /login. In my home infrastructure, this is https://drone.noisepalace.co.uk, but in my demo lab, this is http://192.168.203.10/login: -

Click create application, save the Client ID and Client Secret provided, and we can now install Drone!

Installing Drone

To install Drone, we need a machine that is running Docker. I am running this on a Debian Bullseye machine (Docker Installation docs here), but any machine that can run Docker, Podman, or other OCI-compatible runtimes will probably be fine.

As per the Drone documentation, you need to create a shared secret that will be used for communication between the Drone server and the runners. The documentation suggests to use OpenSSL like so: -

$ openssl rand -hex 16
bea26a2221fd8090ea38720fc445eca6

This isn’t a requirement though. If you want to use pwgen, a password store, or making up your own secret, all are fine.

The documentation then says to pull down the Drone image and then run a Docker command directly to install it.

Personally, I prefer using something like Docker Compose for this so that I always have the container declarations ready to use.

If you want to run this on Kubernetes you can, although I won’t detail that in this post (maybe in the future if and when I move my infrastructure to Kubernetes). My docker-compose.yml file looks like the below: -

version: '3.4'
services:
    drone:
        ports:
            - '80:80'
            - '443:443'
        restart: always
        container_name: drone
        volumes:
            - drone-storage:/data
        environment:
          - DRONE_GITEA_CLIENT_ID=###CLIENT_ID###
          - DRONE_GITEA_CLIENT_SECRET=###CLIENT_SECRET###
          - DRONE_GITEA_SERVER=http://192.168.203.10:3000
          - DRONE_RPC_SECRET=###RPC_SECRET###
          - DRONE_SERVER_HOST=192.168.203.10
          - DRONE_SERVER_PROTO=http
          - DRONE_USER_CREATE=username:stuh84,admin:true
        image: drone/drone:2
        networks:
          - drone
networks:
  drone:
    external: false

volumes:
  drone-storage:
    external: false

In a production scenario, you would want to replace the DRONE_GITEA_SERVER variable with a URL (e.g. https://git.noisepalace.co.uk), and the same for the DRONE_SERVER_HOST (e.g. https://drone.noisepalace.co.uk).

Place this docker-compose.yaml file in a directory (I have it in a directory called drone), run docker-compose up -d, and you should then have a functioning Drone server: -

The nice part about setting up a Gitea OAuth2 application is that you can then login with your Gitea credentials.

Now that we have the Drone server setup, we need some runners to run our CI jobs.

Setting up Drone Runners

There are a few different options for runners with Drone. Some are infrastructure specific (e.g. Digital Ocean, AWS, MacStadium), whereas others are more generic. I currently use three different kinds of runners: -

• Docker Runner - This runner interacts with the Docker daemon on a host to run containers that will execute the actions specified in the CI jobs
  • This is my most used runner, with most tasks being suited to running in a container
• Exec Runner - This runner will run commands directly on a host rather than via containers
  • I use this for my Salt workflow, for reasons I will cover later
• SSH Runner - This runner will use SSH to login to a host and run commands on it
  • I use this for some OpenBSD-specific tasks, as I haven’t been able to get the Exec Runner working on OpenBSD

Docker Runner

As before, the Drone documentation uses Docker commands directly to create a runner. I again use Docker Compose to achieve this, so that everything is declared and stored in a Git repository for when I need it again in future.

The docker-compose.yaml looks like the below: -

version: '3.4'
services:
    drone-runner:
        ports:
            - '3000:3000'
        restart: always
        container_name: drone-runner
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
        environment:
          - DRONE_RPC_SECRET=###RPC_SECRET###
          - DRONE_RPC_HOST=192.168.203.10
          - DRONE_RPC_PROTO=http
          - DRONE_RUNNER_CAPACITY=2
          - DRONE_RUNNER_NAME=docker-runner
        image: drone/drone-runner-docker:1
        networks:
          - drone

networks:
  drone:
    external: false

Drone recommend that the runner runs on a different host from the Drone server itself.

At this point, we can run docker-compose up -d and see if it registers with the Drone server: -

$ docker-compose up -d
Creating network "drone-runner_drone" with the default driver
Pulling drone-runner (drone/drone-runner-docker:1)...
1: Pulling from drone/drone-runner-docker
97518928ae5f: Pull complete
4af047b71fe6: Pull complete
de9b7e52f3f6: Pull complete
679d1ec5770c: Pull complete
Digest: sha256:70da970bb76a62567edbea1ac8002d9484664267f4cbb49fbd7c87a753d02260
Status: Downloaded newer image for drone/drone-runner-docker:1
Creating drone-runner ... done

$ docker logs drone-runner
time="2022-02-04T16:19:46Z" level=info msg="starting the server" addr=":3000"
time="2022-02-04T16:19:46Z" level=info msg="successfully pinged the remote server"
time="2022-02-04T16:19:46Z" level=info msg="polling the remote server" arch=amd64 capacity=2 endpoint="http://192.168.203.10" kind=pipeline os=linux type=docker

Looks good!

Exec Runner

The Exec Runner runs directly on the host, so first we need to get the Exec Runner binary: -

$ curl -L https://github.com/drone-runners/drone-runner-exec/releases/latest/download/drone_runner_exec_linux_amd64.tar.gz | tar zx
$ sudo mv drone_runner_exec /usr/local/bin

Other architectures and operating systems are available from here.

Next, we need to create a configuration file for Drone. I place this in /etc/drone-runner-exec/config: -

DRONE_RPC_SECRET=###RPC_SECRET###
DRONE_RPC_HOST=192.168.203.10
DRONE_RPC_PROTO=http
DRONE_LOG_FILE=/var/log/drone-runner-exec/log.txt
DRONE_RUNNER_LABELS=exec:test

The configuration is similar to what we used in the Docker runner section. The only difference here is that we also specify DRONE_RUNNER_LABELS. These aren’t strictly necessary, but you can use these labels to restrict jobs to execute on certain Runners.

For example, if I wanted to target my Salt Master server, I could use a label of salt:server, and then refer to that in the Drone CI jobs.

Once this is created, we can then create a service file. You can do this yourself, or the Exec Runner can do this for you by running: -

$ drone-runner-exec service install
$ drone-runner-exec service start

We can then check whether this is communicating with the Drone server like so: -

$ sudo systemctl status drone-runner-exec
● drone-runner-exec.service - Drone Exec Runner
     Loaded: loaded (/etc/systemd/system/drone-runner-exec.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-02-04 11:33:55 EST; 6s ago
   Main PID: 5181 (drone-runner-ex)
      Tasks: 7 (limit: 2303)
     Memory: 2.5M
        CPU: 11ms
     CGroup: /system.slice/drone-runner-exec.service
             └─5181 /usr/local/bin/drone-runner-exec service run --config /etc/drone-runner-exec/config

Feb 04 11:33:55 drone-runner systemd[1]: Started Drone Exec Runner.
Feb 04 11:33:55 drone-runner drone-runner-exec[5181]: time="2022-02-04T11:33:55-05:00" level=info msg="starting the server" addr=":3000"
Feb 04 11:33:55 drone-runner drone-runner-exec[5181]: time="2022-02-04T11:33:55-05:00" level=info msg="successfully pinged the remote server"
Feb 04 11:33:55 drone-runner drone-runner-exec[5181]: time="2022-02-04T11:33:55-05:00" level=info msg="polling the remote server" capacity=2 endpoint="http://192.168.203.10" kind=pipeline type=exec

Perfect.

SSH Runner

The SSH runner is for hosts that can be reached via SSH, but can’t necessarily run the Exec Runners themselves. Unfortunately I could not get the Exec Runner to run reliably on an OpenBSD machine, so I use the SSH runner for OpenBSD tasks instead.

Jobs that use the SSH runner require either supplying a user and password, or an SSH key pair to access the machine in question. I created a key pair for this with: -

$ ssh-keygen -t ed25519 -f drone_ssh_key

This creates the drone_ssh_key.pub and drone_ssh_key files which can then be used with the SSH runner. You just need to add the public key as an authorized key for whatever machines you want to access, at which point jobs can specify to use these keys to access the machines in question.

To setup the SSH runner, you again use Docker. The instructions refer to using Docker directly, and again I use Docker Compose for this, like so: -

version: '3.4'
services:
    drone-runner-ssh:
        ports:
            - '3100:3000'
        restart: always
        container_name: ssh-runner
        environment:
          - DRONE_RPC_SECRET=###RPC_SECRET###
          - DRONE_RPC_HOST=192.168.203.10
          - DRONE_RPC_PROTO=http
        image: drone/drone-runner-ssh

We run this as before, placing a docker-compose.yaml file in a directory of your choice (I chose drone-ssh-runner) and then run docker-compose up -d. You should see the runner connect to the Drone server: -

$ docker-compose up -d
Creating network "drone-ssh-runner_default" with the default driver
Pulling drone-runner-ssh (drone/drone-runner-ssh:)...
latest: Pulling from drone/drone-runner-ssh
5a3ea8efae5d: Pull complete
db93b31631ef: Pull complete
5dfa4717c03d: Pull complete
Digest: sha256:cfab2ecc93abb9132f37ef01edc29ac80143e5217423095bb979c310804d3513
Status: Downloaded newer image for drone/drone-runner-ssh:latest
Creating ssh-runner ... done

$ docker logs ssh-runner
time="2022-02-06T15:20:29Z" level=info msg="starting the server" addr=":3000"
time="2022-02-06T15:20:29Z" level=info msg="successfully pinged the remote server"
time="2022-02-06T15:20:29Z" level=info msg="polling the remote server" capacity=10 endpoint="http://192.168.203.10" kind=pipeline type=ssh

Now that we have our runners, we can run some jobs!

Running Drone Pipelines

Drone pipelines are configured using YAML, in a file called .drone.yml that sits in the base path of a repository. The format of this file is like the below: -

kind: pipeline
type: docker
name: default

steps:
- name: testing
  image: alpine 
  commands:
  - echo "Hello from Drone" 

The format is quite similar to Gitlab CI, GitHub Actions, and other CI systems that use YAML to define the steps. We need to take a couple of prerequisite steps before we can run a pipeline in Drone though.

Create a repository in Gitea

If you currently have no repositories in Gitea, you will need to create one: -

Click on the plus sign, and follow the steps to create your first repository.

Sync Drone

If Drone is not synchronized yet, you won’t see any repositories (or at least no new repositories if you already had some in your Gitea instance). Click on the Sync button in the top right hand corner, and Drone will get a list of all repositories that your user has access to in Gitea: -

After this, you should now see your repository appear: -

Configure repository

Click on the repository in Drone, and then click Activate Repository: -

You can now change settings like timeouts on builds (in case a build goes on too long), disabling builds that come from Pull Requests (may be useful in public repositories to stop your resources being used by drive-by commits!) and more. However to just start builds, you don’t need to do anything else.

Add the .drone.yml file to the repo

You can now add the Drone pipeline file to the repository. We’ll use our example repository here: -

$ git clone ssh://git@192.168.203.10:222/stuh84/drone-docker-runner
Cloning into 'drone-docker-runner'...
warning: You appear to have cloned an empty repository.

$ cd drone-docker-runner

$ cat > .drone.yml <<- EOF
kind: pipeline
type: docker
name: default

steps:
- name: testing
  image: alpine
  commands:
  - echo "Hello from Drone"
EOF

$ git add .drone.yml

$ git commit -m "Added .drone.yml"
[master (root-commit) 65509e6] Added .drone.yml
 1 file changed, 9 insertions(+)
 create mode 100644 .drone.yml

$ git push
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 322 bytes | 322.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote: . Processing 1 references
remote: Processed 1 references in total
To ssh://192.168.203.10:222/stuh84/drone-docker-runner.git
 * [new branch]      main -> main

Now every file you add to the repository should now trigger a pipeline in Drone.

Testing the pipeline

Lets commit a random file and see what happens: -

$ echo "Test" > test

$ git add .

$ git commit -m "Test"
[main ab343da] Test build
 1 file changed, 1 insertion(+)
 create mode 100644 test

$ git push
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 279 bytes | 279.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote: . Processing 1 references
remote: Processed 1 references in total
To ssh://192.168.203.10:222/stuh84/drone-docker-runner.git
   65509e6..ab343da  main -> main

We should now see a build kick off in Drone: -

We can see the steps of the build too: -

Summary

At this point, we now have a running Drone server, as well as different kinds of runners that will be able to execute pipelines. The Drone documentation does a good job of showing how to create pipelines, different conditions to run pipelines based upon, as well as how to run pipelines across multiple different kinds of runners (whether different runners, different architectures or different operating systems).

Rather than going into too much detail in this post, I’m going to cover each kind of runner and pipeline that uses it in future posts.

Future posts

The next four posts are going to cover how to create Drone pipelines for: -

• Ansible - Using the Docker Runner for all tasks
• Saltstack - Using the Exec Runner for most tasks, and the Docker Runner for notifications
• Go building and releasing - Using the Docker Runner on x86_64 and arm64 Linux, as well as using the SSH runner for OpenBSD builds and releases
• Hugo - Using the Docker Runner

As we go through of these posts, we’ll see how create pipelines, how to trigger them, how to take different steps based upon Pull Requests or merges to the main branch, and how through all of this Git now becomes the main method of defining and triggering changes on my infrastructure rather than manual intervention!

As mentioned in the last post, I decided to split my use of Ansible and Salt into: -

Ansible

• Bootstrapping nodes with the Saltstack agent and my SSH keys
• Taking a nodes SSH key and adding it to Gitea/Gitlab/GitHub
• Managing DNS and DHCP for hosts in my network (pools, static reservations, DNS entries etc)
• Adding certain hosts to Cloudflare DNS so that I can use Lets Encrypt on my applications
• Adding IPs/descriptions to Netbox for IPAM

Salt

• Pretty much everything else!

Ansible is great as an early-stage/node preparation tool, as it doesn’t rely on any Agents being applied. Once this is all done, Saltstack can takes over and manage what runs on nodes that have agents installed, what Prometheus exporters (i.e. monitoring agents) run alongside the applications, and more.

This series goes through what has changed in my home infrastructure, and more importantly why it needed to change. While my Ansible setup was rewritten from scratch mostly (new repository, new roles, different variable structures etc), my Salt environment didn’t need quite as much work.

If you already use Salt, or want to, hopefully some of this will help you in improving your Salt environment.

What needed to change?

When I started using Saltstack on my home infrastructure, I was already using it in a professional capacity too. While this doesn’t guarantee that everything will be perfect, I had a solid grounding to build upon.

However I wasn’t entirely happy with what I had put together. The main issues were: -

• Very little was managed with Salt
  • I only had Consul installation and configuration, some base packages and installing/enabling LLDP
• Consul pillar data (i.e. variables that applied to the hosts) was required to be defined up front for every host
• Services would restart whenever Salt ran, even if nothing changed
• The Pillars that defined what exporters Consul would add to it’s service list was manually defined
• Consul binaries were stored in the Git repository directly, rather than sourced from Hashicorp’s repositories or releases
• States across different operating systems (e.g. Ubuntu/Debian Linux, Alpine and OpenBSD) differed slightly, and yet were repeated in their entirety

Why did I want to change this? I wanted to move to continuous integration for my infrastructure. Also, I wanted Git to be a source of truth for how my applications were defined. If every run restarts services, and every new host needs additional configuration, it makes the process less declarative and more reactive (and impactful).

Also, as I was migrating services from Ansible to Salt, I didn’t want to inherit some of the same issues I had with the Ansible playbooks (non-repeatable tasks, locally-stored binaries, dependency issues). If I could get Salt into a good state, I would have a solid blueprint for adding more services to it.

The issues

First I’ll outline all the issues, and then I’ll go through how I approached them.

Host-specific pillars

My pillar top.sls file (the file which defines what pillars are tied to what host) had the following definition: -

base:
  '*':
    - consul.{{ grains['nodename'] }}

The asterisk (as you might be able to guess) means that this applies to every host. The {{ grains['nodename'] }} placeholder that refers to the name of the host. The below shows what the nodename is for some of my hosts: -

# salt '*' grains.item nodename
archspire.noisepalace.home:
    ----------
    nodename:
        archspire
git-01.noisepalace.home:
    ----------
    nodename:
        git-01
vpn-01.noisepalace.home:
    ----------
    nodename:
        vpn-01
ns-03:
    ----------
    nodename:
        ns-03
pinkfloyd.noisepalace.home:
    ----------
    nodename:
        pinkfloyd
dev-02.noisepalace.home:
    ----------
    nodename:
        dev-02

What this means is that Salt will look for a file in the directory consul called $NODENAME.sls (e.g. vpn-01.sls, dev-02.sls). If a file does not exist for every node, Salt will throw an error, because the Pillar top.sls says that every host requires one.

This is an issue, as it means that when a brand new host is added, a file must be created that contains all the variables it needs. The point in configuration management is so that many hosts can be configured and prepared en masse. Defining a file for every single host goes against the principles of automation!

Why did I set up the pillars like this in the first place? The reason is because I run Consul on my VPSs (for Prometheus exporter service discovery). As I don’t want to make my Consul servers publicly available, I needed to ensure the interface it would listen on (in my case, Wireguard tunnel interfaces). There are many ways to do this, but I chose the “easiest” rather than the most scalable. Hindsight is a wonderful thing!

Hard-coding data in Pillars

If we take a look at the old Pillar data, we can see the data Consul would use for binding to a specific interface: -

# vps-shme.sls
consul:
  bind_int: wg0
  prometheus_services:
    - node_exporter
    - wireguard
    - iptables
    - apache
    - mysql
    - icmp

# ns-03.sls
consul:
  bind_int: enp6s0
  prometheus_services:
    - node_exporter
    - named
    - pihole
    - dhcp
    - icmp

The variable bind_int was used in the Consul configuration template when deploying Consul to a node. It ensured Consul would bind to the correct interface. This required defining the pillar before we can run Salt, otherwise Salt would fail to run.

If we take a look at the Consul configuration template itself, we can see where this Pillar data was used: -

datacenter = "noisepalace"
data_dir = "/opt/consul"
encrypt = "$TOKEN"
retry_join = ["192.168.0.6"]
{%- if pillar['consul'] is defined %}
{%- if pillar['consul']['bind_int'] is defined %}
{%- set interface = pillar['consul']['bind_int'] %}
bind_addr = "{{ grains['ip4_interfaces'][interface][0] }}"
{%- endif %}
{%- endif %}

node_meta {
  env = "noisepalace" 
}

performance {
  raft_multiplier = 1
}

This takes the interface we supply in our Pillar data, and then derives the IP address of this interface from the host’s grains (the host facts).

We also have a hard-coded list of what prometheus_services run on this machine. If one of the exporters is removed, it also needs to be removed from here. If I forgot to add the exporter to the list, it would never be monitored. This variable was used in the Consul State file (states/consul/init.sls) like so: -

{% if 'prometheus_services' in pillar['consul'] %}
{% for service in pillar['consul']['prometheus_services'] %}
/etc/consul.d/{{ service }}.json:
  file.managed:
    - source: salt://consul/files/{{ service }}.json
    - user: consul
    - group: consul
    - mode: 0640
    - template: jinja
{% endfor %}
{% endif %}
{% endif %}

consul_reload:
  cmd.run:
    - name: consul reload

For every service in the list, we would add the JSON file into the correct Consul directory, and then at the end we would reload Consul. While this task statement is okay, it is reliant on hard-coded data.

Needless restarting/reloading of services

As I didn’t use Salt for many things, or run it often, the odd service restart wasn’t an issue. However the more I moved to it, the more applications would restart without good reason. If every minor configuration change required restarting almost every application on a machine, the machines would have huge CPU/memory/disk usage spikes, as well as taking the services down.

As an example, the below reloads Consul, whether anything has changed or not: -

consul_reload:
  cmd.run:
    - name: consul reload

Also a lot of the Ansible Playbooks I wanted to migrate (particularly exporter installation) would restart services, Prometheus, exporters and more every time they ran, e.g.: -

- name: Daemon Reload and run prometheus
  systemd:
    state: restarted
    enabled: yes
    daemon_reload: yes
    name: prometheus

Binaries stored in the repository

Storing binaries in Git (unless you are using something like Git LFS is not recommended. The problem with having binaries stored in Git (or other version control repository) are: -

• Git is geared towards showing changes, which is difficult and not very useful on binary files
• The repository size is significantly larger compared to storing text/configuration/code
• The version of the binary is static until you download a new version and commit it to the repository

Bear in mind that Git stores all versions of objects committed. If over time you have multiple versions of a binary (even if they have the same name) in the history, a clone of the repository in future would include all versions of that binary too.

The following is how Consul was installed previously: -

/usr/local/bin/consul:
  file.managed:
    {% if grains['os'] != 'Alpine' and grains['os'] != 'OpenBSD' and grains['cpuarch'] == 'x86_64' %}
    - source: salt://consul/files/consul_amd64
    {% elif grains['os'] != 'Alpine' and grains['os'] != 'OpenBSD' and 'arm' in grains['cpuarch'] %}
    - source: salt://consul/files/consul_arm
    {% endif %}
    - user: root
    - group: root
    - mode: 755

Here there is a version of Consul for x86-64 machines, and one for arm, both stored in the repository.

Similar tasks repeated for different systems

In my network I run a mixture of Debian, Ubuntu, Alpine, OpenBSD and a single Arch Linux machine (mainly for development/testing).

Debian and Ubuntu are similar enough that most states for Debian work with Ubuntu. However Arch, Alpine, and OpenBSD especially, differ significantly. Arch uses a different package manager (and some different locations for some binaries), Alpine uses a different init system and service manager, and OpenBSD is an entirely different operating system altogether.

To cater to these, many states were repeated but with minor differences. For example: -

{% if grains['os'] != 'OpenBSD' %}
/opt/consul:
  file.directory:
    - user: consul
    - group: consul
    - mode: 755
    - makedirs: True

/etc/consul.d:
  file.directory:
    - user: consul
    - group: consul
    - mode: 755
    - makedirs: True

[...]
{% if grains['os'] == 'OpenBSD' %}
/opt/consul:
  file.directory:
    - user: _consul
    - group: _consul
    - mode: 755
    - makedirs: True

/etc/consul.d:
  file.directory:
    - user: _consul
    - group: _consul
    - mode: 755
    - makedirs: True
[...]

The only difference in the above is that the user and group are prefixed by an underscore (_) on OpenBSD, and yet we have entirely separate tasks to create directories and add files.

If I need to change the directory that Consul stores data in or it’s configuration, I would have had to change it in multiple places. This often leads to forgetting to change it in all places and breaking at least one of the systems used.

The Changes

While the above issues were significant, they didn’t require a complete refactor (unlike my Ansible repository).

Host-specific pillars

I wanted to move away from host-specific pillars for every machine. This doesn’t mean that hosts cannot have pillars that are specific to them, just that by default they are only required if they differ from the defaults. Having sensible defaults rather than no defaults can dramatically reduce complexity and manual configuration work.

I approached this by making better use of custom grains.

Grains are like facts in Ansible. Salt derives the facts from the host, rather than defining them for the host. For example, the CPU architecture of a machine is derived from the host, as is what OS it is running.

As mentioned already, I use Ansible to bootstrap hosts and to deploy the Salt Minion application. A few minor changes in my Ansible code would help with solving this issue.

Previously, the Salt Minion configuration template looked like the below: -

master: salt-master.noisepalace.home
id: {{ ansible_fqdn }}
nodename: {{ ansible_hostname }}
grains:
  nodename: {{ ansible_hostname }}

We had a single custom grain here (which overrides the discovered nodename grain), but nothing more. The below shows the changes I made to the template: -

master: salt-master.noisepalace.home
id: {{ ansible_fqdn }}
nodename: {{ ansible_hostname }}
grains:
  nodename: {{ ansible_hostname }}
{% if 'vps' in ansible_hostname %}
  bind_int: wg0
{% else %}
  bind_int: {{ ansible_default_ipv4['interface'] }}
{% endif %}
  groups:
{% for group in group_names %}
    - {{ group }}
{% endfor -%}

I added the bind_int grain, and also the groups grain. The bind_int grain will contain the primary interface on a machine (i.e. the one which has a default route) unless it is one of my VPSs. If it is a VPS, the bind_int grain will always be the Wireguard tunnel interface (wg0).

I then updated the Consul configuration template to remove the reliance on Pillar data: -

datacenter = "noisepalace"
data_dir = "/opt/consul"
encrypt = "$ENCRYPTION_TOKEN"
node_name = "{{ grains['nodename'] }}"
retry_join = ["192.168.0.6"]
{%- set interface = grains['bind_int'] %}
bind_addr = "{{ grains['ip4_interfaces'][interface][0] }}"

node_meta {
  env = "noisepalace"
}

performance {
  raft_multiplier = 1
}

The Consul pillar data for each host is no longer required for it to be installed and configured correctly.

We still need to do something about the Prometheus services that are registered with Consul, but I will cover that in a later section.

In addition, I added the Ansible groups that each host is in as grains. This means that I can define a host as having a role, and then install packages/states/configuration based upon the role. A example of this is below: -

base:
[...]
  'groups:vmh':
    - match: grain
    - vmh
    - exporters.libvirt_exporter
    - exporters.syncthing_exporter
    - docker
    - syncthing

The above says that if a host is in the vmh group, we will install the vmh states (which installed KVM/QEMU and related services), Docker, Syncthing (for syncing ISOs between hosts) and related exporters for monitoring.

I also use this to say whether the qemu-guest-agent package should be installed on a machine, based upon it being in the vmh-guest group: -

base_packages:
  pkg.installed:
  - pkgs:
    - lldpd
    - jq
    - rsync
    - wget
[...]
{% if 'vmh-guests' in grains['groups'] %}
    - qemu-guest-agent
{% endif %}
[...]

This means the qemu-guest-agent package is not installed on the hypervisors and VPSs, only those that need it (i.e. my virtual machines).

Hard-coding data in pillars

I am no longer hard-coding which interface Consul will bind to (using custom grains). The next improvement is to remove the hard-coded list of exporters to register with Consul (for Prometheus to discover and monitor).

Part of the issue previously is that because all of the exporters were installed by Ansible, Salt did not know what was already installed on a machine. However now the exporters are deployed with Salt, we can also include the relevant file to register them with Consul: -

## Wireguard exporter 
[...]
/etc/consul.d/wireguard.json:
  file.managed:
    - source: salt://consul/files/wireguard.json
    - user: consul
    - group: consul
    - mode: 0640
    - template: jinja

consul_reload_wireguard:
  cmd.run:
    - name: consul reload
    - onchanges:
      - file: /etc/consul.d/wireguard.json

## Pihole exporter 
[...]
/etc/consul.d/pihole.json:
  file.managed:
    - source: salt://consul/files/pihole.json
    - user: consul
    - group: consul
    - mode: 0640
    - template: jinja

consul_reload_pihole:
  cmd.run:
    - name: consul reload
    - onchanges:
      - file: /etc/consul.d/pihole.json

These types of tasks are included in the State file for each exporter. If an exporter is installed, it gets registered with Consul automatically. This removes the reliance Pillar data for services Prometheus needs to monitor, meaning no Consul pillar needs to be defined for every host.

This isn’t to say that no Consul pillar data exists at all though. Some applications already have a metrics/monitoring endpoint (i.e. no exporters are required) that exposes metrics in the correct Prometheus format. In this case, we still supply a Consul pillar like so: -

## git-01.sls
consul:
  prometheus_services:
    - gitea

## pinkfloyd.sls
consul:
  prometheus_services:
    - traefik
    - alertmanager

However unlike before, this doesn’t cover every possible Prometheus service, just additional services. If I show what services are running on pinkfloyd for example: -

$ consul catalog services -node pinkfloyd
alertmanager
cadvisor
consul
icmp
libvirt
mikrotik_exporter
nextcloud
nginx
node_exporter
pihole
redis
speedtest
syncthing
traefik
wireguard

All but traefik and alertmanager are not dependent upon Pillar data.

Restarting/reloading of services

To avoid restarting/reloading services when nothing has changed, I used three different directives in Salt: -

• onchanges - This will run a task only if a prerequisite task has changed
• watch - This is very similar to onchanges, except that for certain kinds of tasks additional actions may be taken
• require - This only requires that a task completed successfully, as opposed to a previous task completing and making changes

A good example of require is ensuring a package has been installed before trying to configure it.

If a package fails to installed, then the files it would install (e.g. service files, configuration, directories) will not exist. If a subsequent task tries to put additional files into the directory that the package installation would have created, it will also fail. With the require keyword, we ensure that tasks with hard dependencies do not run without their prerequisites completing first.

The onchanges directive means that the task will only run if a prerequisite task made any changes. This is useful for things like reloading Consul if a file was added/changed in it’s configuration directory, or performing a systemctl daemon-reload if the SystemD unit file changes.

A good example with both require and onchanges is below: -

test_prometheus_config:
  cmd.run:
    - name: "/usr/local/bin/promtool check config /etc/prometheus/prometheus.yml"
    - onchanges:
      - file: /etc/prometheus/prometheus.yml
      - file: add_prometheus_alerts

prometheus_reload:
  cmd.run:
    - name: curl -X POST localhost:9090/-/reload
    - onchanges:
      - file: /etc/prometheus/prometheus.yml
      - file: add_prometheus_alerts
    - require:
      - test_prometheus_config

In the above, we use promtool (a Prometheus utility CLI tool) to check that our Prometheus configuration is correct. The promtool command will also check Alertmanager rules by default. This uses onchanges to make sure it only runs if there have been updates to the /etc/prometheus/prometheus.yml task or the add_prometheus_alerts task.

If the test_prometheus_config task runs without errors, then we trigger a reload of Prometheus via the API. If the test_prometheus_config task fails, Prometheus does not reload. If neither the Prometheus configuration or Alertmanager rules have changed, no reload takes place.

This not only ensures that Prometheus only reloads when it’s configuration files changes, but has added benefit of making sure they are syntactically correct.

A quick point to the note is that Salt tasks can have custom names (e.g. add_prometheus_alerts) or the name can be the file/service being changed (e.g. /etc/prometheus/prometheus.yml). The onchanges directive must refer to the name of the task, whether that is a custom name, or matches the file/service being acted upon.

watch versus onchanges

The onchanges directive instructs Salt to run the task if a preceding task changed. In the above we can see that this runs commands on the machines in question to test/reload Prometheus.

Take the below as an example: -

prometheus_service:
  service.running:
    - name: prometheus
    - enable: True
    - onchanges:
      - file: /etc/systemd/system/prometheus.service

When Prometheus is installed, this will ensure the service is running and is enabled. If the service file changes though, the task doesn’t do anything because the service is still running (unless another process/user stopped it) and is still enabled.

What watch does is for certain types of tasks, it will perform additional actions. If we change the above to this: -

prometheus_service:
  service.running:
    - name: prometheus
    - enable: True
    - watch:
      - file: /etc/systemd/system/prometheus.service

Salt will now perform the additional action of restarting the service if the file it is watching changes. Not every type of task has an additional watch behaviour, but in this case it helps to cut down on the amount of tasks we need to define.

Binaries stored in the repository

This is a pretty straightforward change to make, especially for Consul. Hashicorp now make a lot of their products available via Apt and RPM repositories. This means all my Debian and Ubuntu machines can now use the repositories, and get updates when they are available (not when I remember to update the binary in my Git repository).

This doesn’t help with my OpenBSD, Alpine and Arch machines. For them, I use what is in their package repositories. All three do keep relatively up-to-date packages for Consul, whereas the versions in the Debian/Ubuntu archives are usually months/years old.

To move to this approach, we use: -

{% if grains['os_family'] == 'Debian' %}
consul_repo:
  pkgrepo.managed:
    - humanname: Hashicorp
    - name: "deb https://apt.releases.hashicorp.com {{ grains['oscodename'] }} main"
    - dist: {{ grains['oscodename'] }}
    - file: /etc/apt/sources.list.d/hashicorp.list
    - gpgcheck: 1
    - key_url: https://apt.releases.hashicorp.com/gpg
{% endif %}

{% if grains['os_family'] == 'RedHat'  and grains['os'] != 'Fedora' %}
consul_repo:
  pkgrepo.managed:
    - humanname: Hashicorp
    - file: /etc/yum.repos.d/hashicorp.repo
    - baseurl: https://rpm.releases.hashicorp.com/RHEL/$releasever/$basearch/stable
    - gpgcheck: 1
    - gpgkey: https://rpm.releases.hashicorp.com/gpg
{% endif %}

{% if grains['os_family'] == 'RedHat' and grains['os'] == 'Fedora' %}
consul_repo:
  pkgrepo.managed:
    - humanname: Hashicorp
    - file: /etc/yum.repos.d/hashicorp.repo
    - baseurl: https://rpm.releases.hashicorp.com/fedora/$releasever/$basearch/stable
    - gpgcheck: 1
    - gpgkey: https://rpm.releases.hashicorp.com/gpg
{% endif %}

{% if grains['os_family'] == 'Suse' %}
Virtualization_containers:
  pkgrepo.managed:
    - humanname: "Virtualization:containers (openSUSE_Tumbleweed_and_d_l_g)"
    - baseurl: https://download.opensuse.org/repositories/Virtualization:/containers/openSUSE_Tumbleweed_and_d_l_g/
    - gpgcheck: 1
    - gpgkey: https://download.opensuse.org/repositories/Virtualization:/containers/openSUSE_Tumbleweed_and_d_l_g/repodata/repomd.xml.key
    - autorefresh: 0
    - gpgautoimport: True
{% endif %}

consul_package:
  pkg.installed:
  - pkgs:
    - consul
  - refresh: True

[...]

As you can see, I also have declarations for RHEL-based machines and SuSE. While I don’t currently run them on my infrastructure, it is worth having these for if and when I do need them. I wanted to be agnostic as I can be to what systems I need in my infrastructure, allowing me to choose the best systems for each role.

The last task referenced above installs Consul, and refreshes the package repositories so that it will use the latest package available. This ensures that we install from the Hashicorp repository after it is added (to the systems that repositories exist for). This will also use the native package installer on each system, whether that is Apt, DNF, Pacman, Zypper, APK, pkg_add or anything else.

In the cases where official package repositories are not available (e.g. most exporters, Prometheus and more), we can use this approach instead: -

{% if 'x86_64' in grains['cpuarch'] %}
retrieve_bind_exporter:
  cmd.run:
    - name: wget -O /tmp/bind_exporter.tar.gz https://github.com/prometheus-community/bind_exporter/releases/download/v{{ pillar['bind_exporter']['version'] }}/bind_exporter-{{ pillar['bind_exporter']['version'] }}.linux-amd64.tar.gz
{% endif %}
{% if 'aarch64' in grains['cpuarch'] %}
retrieve_bind_exporter:
  cmd.run:
    - name: wget -O /tmp/bind_exporter.tar.gz https://github.com/prometheus-community/bind_exporter/releases/download/v{{ pillar['bind_exporter']['version'] }}/bind_exporter-{{ pillar['bind_exporter']['version'] }}.linux-arm64.tar.gz
{% endif %}

extract_bind_exporter:
  archive.extracted:
    - name: /tmp
    - enforce_toplevel: false
    - source: /tmp/bind_exporter.tar.gz
    - archive_format: tar
    - user: root
    - group: root

{% if 'x86_64' in grains['cpuarch'] %}
/usr/local/bin/bind_exporter:
  file.rename:
    - name: /usr/local/bin/bind_exporter
    - source: /tmp/bind_exporter-{{ pillar['bind_exporter']['version'] }}.linux-amd64/bind_exporter

delete_bind_exporter_dir:
  file.absent:
    - name: /tmp/bind_exporter-{{ pillar['bind_exporter']['version'] }}.linux-amd64
{% endif %}

{% if 'aarch64' in grains['cpuarch'] %}
/usr/local/bin/bind_exporter:
  file.rename:
    - name: /usr/local/bin/bind_exporter
    - source: /tmp/bind_exporter-{{ pillar['bind_exporter']['version'] }}.linux-arm64/bind_exporter

delete_bind_exporter_dir:
  file.absent:
    - name: /tmp/bind_exporter-{{ pillar['bind_exporter']['version'] }}.linux-arm64
{% endif %}

delete_bind_exporter_files:
  file.absent:
    - name: /tmp/bind_exporter.tar.gz
{% endif %}

This does pin the exporters to a version rather than having them on the latest, but it does at least ensure that I am not storing an old version in a repository and using that.

Similar tasks repeated for different systems

One of the issues I found with having tasks that are almost identical but with one field/option different (e.g. a different username/group) is that I would often make updates to one and not the other, and then wonder why my changes were not taking place.

Unlike Ansible, Salt generates the state files on the fly, allowing you to use Jinja2 templating syntax inside of them. How does this help us? Lets see: -

{% if grains['os'] == 'FreeBSD' %}
/usr/local/etc/consul.d/consul.hcl:
{% else %}
/etc/consul.d/consul.hcl:
{% endif %}
  file.managed:
    - source: salt://consul/files/consul.hcl
{% if grains['os'] == 'OpenBSD' %}
    - user: _consul
    - group: _consul
{% else %}
    - user: consul
    - group: consul
{% endif %}
    - mode: 0640
    - template: jinja

The first conditional evaluates based upon whether the machine running this task is a FreeBSD machine or not. FreeBSD uses a different location for user-installed binaries and configuration files (usually prefixed with /usr/local). With the conditional in place, we can use a different filename based upon the system.

After this, we also have a conditional to say that if we are on OpenBSD, the user and group for Consul is _consul. For any other system, the user and group would be consul.

Rather than potentially 3 different tasks (a standard task, FreeBSD with a differing filename, OpenBSD with different user/groups), we have a single task with some conditionals in place. Updates to this task will now apply to all machines, not just the ones I remember to update!

What else?

So after mitigating all these issue, what else needed to be done? Move as much to Salt as possible!

As mentioned, previously I only installed and managed Consul with Salt. Now, I manage the following: -

I have also integrated Hashicorp’s Vault for secret/token management. This means I can avoid committing sensitive values to Git repositories. I will go into more information on this in a future post, including how to use it with Ansible, Salt, Drone and using the Go SDK.

What still needs work?

There are still a few things I want to tackle in my Salt setup, some minor, some quite major.

Some of my state files could be improved. For example, the states I use to deploy the Prometheus node_exporter need work. These separate files (with very similar tasks) for Linux systems using SystemD, Alpine and OpenBSD. As shown already, these can be consolidated quite easily. However I essentially used the same code as laid out in my series on deploying Prometheus and Consul with Saltstack.

Not all applications I use are managed by Salt yet. Most are, but I haven’t tackled FreeRADIUS, monitoring OpenBSD package upgrades, and a few other small pieces. These aren’t huge, but they do need to be managed.

Finally, I want to make more use of Custom Grains. The grains I have now are fine, but the more I have, the less static host definitions/wildcard matches are required in my top.sls files. Examples could include deploying the Salt master, deploying Print servers, what requires Wireguard tunnels and more. This would allow me to treat all my machines as having roles instead. This simplifies configuration, and makes it easy to add new hosts with similar/the same roles.

Bonus Section: Why not Salt Masterless instead of Ansible?

When I released the last post, I got a question on the Admin Admin Podcast Telegram Channel as to why I don’t use Salt Masterless instead of Ansible. This a valid question, especially as it would mean only managing one type of codebase. I have a few reasons for this.

Firstly, the process to use Salt Masterless still requires bootstrapping Salt onto a machine.

We also need to add additional configuration to Salt on a machine to tell it to use Masterless mode (see the referenced documentation for what needs to be done)

Additionally, the Salt configuration must be available to the machine (by cloning it locally, or defining a Git source in the Salt configuration).

Finally, Salt Masterless works the opposite way to Ansible. Rather than Ansible telling multiple machines what to do, Salt Masterless requires running Salt commands directly on each host to update their configuration. If you are familiar with Puppet (and running puppet agent -t in Cron/service files periodically), this is the same sort of model.

To make this even less desirable, the machines would then need their Salt configuration replaced at the end of the bootstrap process to now use Salt in non-Masterless mode.

If anything, the work to use Masterless mode requires more steps than using non-Masterless!

Masterless mode makes a huge amount of sense in a Cloud-like environment, using something like Cloud-Init/Cloud-Config to install Salt on the machines, and then point Salt to Git as a source of it’s state files. I wouldn’t hesitate to use it in that sort of scenario.

With my infrastructure though, Ansible makes more sense. Short of adding a user with the correct privileges to the machine so that Ansible can run the tasks it needs, I don’t need to do anything else. It also fits much better into using continuous integration, as Salt Masterless would require logging to every host and running commands. Ansible manages all the host interaction, it just needs a valid user and credentials/SSH key available.

Summary

The updates made to Salt, and turning it into (mostly) the source of truth of what runs on my infrastructure (that isn’t in a container) has made managing my personal infrastructure a lot nicer.

My applications are up to date, the configuration is version controlled for the applications, and now I can trust what is in Git is also what is running.

It is also now in a state that if no changes are made in my Salt code, no changes are seen when running Salt. Services aren’t restarted every time, configuration is repeatable, everything runs clean! This is exactly what I was aiming for, paving the way to using a Continuous Integration system to manage and deploy future changes.

Next post

The next post in this series will cover getting started with Drone, the different kinds of “runners”, and the Drone files that contain the actions to take. Once this is done, there will be subsequent posts on how I integrated Ansible and Salt with Drone, how I build Go binaries, and how I deploy this website that you’re reading this post on using Drone too!

I had a collection (pun not intended) of single-use playbooks that performed tasks like: -

• Updating DHCP (ISC DHCP) server configuration
• Updating DNS (BIND9) server configuration
• Updating Prometheus configuration (including Alertmanager and associated alert rules)
• Installing Prometheus exporters on new machines
• Installing the salt-minion (for Saltstack configuration management) package on new machines

To set up a new machine, I had to update a file that contained the machine’s intended IP address, MAC address (for static DHCP allocations) and then run each playbook against it.

This isn’t particularly scalable, and it made moving to a continuous integration system really difficult.

What is a role?

An Ansible Role is a set of tasks that are related to each other, e.g. they configure/install a certain application. More often than not you would create a role to install and configure a single service, e.g. bind for DNS or isc-dhcp-server.

Roles can also be shared (either in your organisation, or publicly). Rather than duplicating code, you can use your own (or other peoples) roles again when the need arises. The Ansible Role Documentation gives a more in-depth overview of what a role is and how to use it.

A Role has a certain directory structure, like so: -

-- roles
    |-- base
    |   |-- defaults
    |   |-- files
    |   |-- handlers
    |   |-- meta
    |   |-- tasks
    |   |-- templates
    |   |-- tests
    |   |-- vars
    |-- dns
    |   |-- defaults
    |   |-- files
    |   |-- handlers
    |   |-- meta
    |   |-- tasks
    |   |-- templates
    |   |-- tests
    |   |-- vars

You can create the directory structure yourself, or you can use ansible-galaxy init $ROLENAME. The latter will instantiate a directory with the correct structure, as well as some basic README files, tests and base YAML files to add your tasks/handlers/tests to. Ansible Galaxy is the central repository of Ansible Roles that are shared publicly, but you can use the CLI tool to create local roles without a requirement to share them.

Typically in an Ansible playbook if you want to use a template, a file, handler or anything else you need to give the full path (or at least the relative path) to find them. However within a role, Ansible will look for files in one of the role directories first. For example, if you had a task like the following: -

- name: Copy file to server
  copy:
    src: myfile.txt
    dest: /tmp/myfile.txt
    owner: myuser
    group: mygroup
    mode: 0644

If you place the file myfile.txt in the files directory of your role, Ansible will look there first, meaning you do not need to specify the full/relative path. This allows you to keep all related files, templates and tasks in the same role, making it much easier to share with others. It also helps when you looking at the role again a few months down the line.

How can we use a role?

To use a role, create a directory called roles, add the role in there, and then create a playbook in the top-level directory that references it. The below is a short example of using a role: -

- hosts: all
  become: true
  roles:
    - role: base
      tags:
        - base

In this, we have included a role called base, as well as adding a tag to it. Tags can be used to run only certain roles/tasks. If you run ansible-playbook --tags base playbook.yml, the only tasks/roles that will run are those tagged with base. This is especially helpful when creating your roles and testing them.

You can also use public roles (from Ansible Galaxy) like so: -

- hosts: all
  become: true
  roles:
    - geerlingguy.docker

If you then add the following to a requirements.yaml file at the base of your repository: -

roles:
  - name: geerlingguy.docker

By default this will look for the role in Ansible Galaxy before trying to run against your hosts. You can also refer to roles directly from Git repositories using: -

roles:
  - name: nginx_role
    src: https://github.com/bennojoy/nginx
    version: main

This is how you can share roles in your organisation without necessarily making them available for public consumption.

To specify multiple roles for different hosts in a single playbook, you can do something like the following: -

- hosts: archspire.noisepalace.home
  become: true
  roles:
    - role: blackbox_checks
      tags:
        - blackbox
        - prometheus

- hosts: vmh
  become: true
  roles:
    - role: cockpit
      tags:
        - cockpit

- hosts: ns-04.noisepalace.home,ns-03.noisepalace.home
  become: true
  roles:
    - role: dns
      tags:
        - dns
        - bind9
    - role: dhcp
      tags:
        - dhcp

What we can see here is that I am running different roles against different hosts, all from the same place. This means that when you run the single playbook, all machines will get the changes they need for every role applicable to them.

Working example: DNS

The best way to show what has changed is to go through a previous example of how I used to update DNS and compare it with what I use now. DNS is a good example because it uses some tasks, installs packages and uses files and templates.

Old Playbook

The previous playbook looked like the below: -

---
- hosts: 192.168.10.1,192.168.10.2
  become: yes
  tasks:
  - name: Install bind9
    package:
      state: present
      name: bind9
      update_cache: yes

- hosts: 192.168.10.1
  become: yes
  tasks:
  - name: Pick up vars file
    include_vars:
      file: ../dhcp/ip-hosts.yaml
      name: home_hosts
    tags:
    - configuration

  - name: Pick up net vars file
    include_vars:
      file: ../netvars.yaml
    tags:
    - configuration

  - name: Generate reverse lookup file
    template:
      src: named/db.168.192.in-addr.arpa.j2
      dest: /etc/bind/db.168.192.in-addr.arpa
    tags:
    - configuration

  - name: Generate forward lookup file
    template:
      src: named/db.noisepalace.home.j2
      dest: /etc/bind/db.noisepalace.home
    tags:
    - configuration

  - name: Named Local
    copy:
      src: named/named.conf.local.primary
      dest: /etc/bind/named.conf.local
    tags:
    - configuration

  - name: Named Options
    template:
      src: named/named.conf.options.primary
      dest: /etc/bind/named.conf.options
    tags:
    - configuration

- hosts: 192.168.10.2
  become: yes
  tasks:
  - name: Pick up net vars file
    include_vars:
      file: ../netvars.yaml
    tags:
    - configuration

  - name: Named Local
    copy:
      src: named/named.conf.local.secondary
      dest: /etc/bind/named.conf.local
    tags:
    - configuration

  - name: Named Options
    template:
      src: named/named.conf.options.secondary
      dest: /etc/bind/named.conf.options
    tags:
    - configuration


- hosts: 192.168.10.1,192.168.10.2
  become: yes
  tasks:

  - name: Stats directory
    file:
      path: /var/bind
      owner: bind
      group: bind
      mode: '0755'
      state: directory
    tags:
    - configuration

  - name: Reload bind9
    systemd:
      name: bind9
      state: reloaded
      enabled: yes
    tags:
    - configuration

There are a few issues with this playbook that I wanted to address: -

• Statically defined hosts
  • This means that if I wanted to re-use this with other hosts, I have to update this playbook (and all the host declarations inside of it)
• The same/similar tasks repeated
• BIND9 will be reloaded at the end of this playbook, even if nothing changed
• We refer to two different sets of variables files, including one that is in a different “roles” directory
  • As this didn’t use true Ansible Roles before, I refer to a role here as a different directory of Playbooks
• One of the sets of variables is imported with a prefix of home_hosts on every variable

Some of this is down to my understanding when creating the Playbooks, but some of it was down to just wanting to get it to work and then change later (which I expected to happen much sooner than 2 years later!). For those who know how often a temporary fix becomes semi-permanent, this shouldn’t come as too much of a surprise 😄.

An example of the variables are below: -

ip-hosts.yaml

home_hosts:
   router:
      mac: c4:ad:34:55:cb:03
      ip_addr: 192.168.0.1
   VanHalen:
      mac: 02:95:09:c0:d1:9a
      ip_addr: 192.168.0.3
   archspire:
      mac: f4:4d:30:64:4d:bf
      ip_addr: 192.168.0.5
   pinkfloyd:
      mac: a8:20:66:34:f0:2b
      ip_addr: 192.168.0.6
[...]
   behemoth:
      cname: behemoth-10g
   pihole:
      cname: netutil-01
   grafana:
      cname: pinkfloyd
   prometheus:
      cname: pinkfloyd

netvars.yaml

netvars:
  ip_ranges:
    - 192.168.0.0/24
    - 192.168.2.0/24
    - 192.168.3.0/24
    - 192.168.4.0/24
    - 192.168.5.0/24
    - 192.168.10.0/24
    - 192.168.20.0/24
    - 192.168.99.0/24
    - 192.168.100.0/24
    - 192.168.255.0/24

Old templates

The previous templates used looked like the below: -

db.noisepalace.home.j2

;
; BIND data file for local loopback interface
;
$TTL    604800
@       IN      SOA     noisepalace.home. root.noisepalace.home. (
                              2         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@               	IN      NS      ns4.noisepalace.home.
@               	IN      A       192.168.10.1
ns3             	IN      A       192.168.10.2
ns4             	IN      A       192.168.10.1
{% for name in home_hosts.home_hosts %}
{% if "cname" in home_hosts.home_hosts[name] %}
{{ name|lower }} IN      CNAME  {{ home_hosts.home_hosts[name]['cname'] }}
{% else %}
{{ name|lower }} IN      A      {{ home_hosts.home_hosts[name]['ip_addr'] }}
{% endif %}
{% endfor %}

While this is mostly okay, we could clean up the conditional check on CNAMEs and A records. Also having every variable prefixed with home_hosts is redundant.

db.168.192.in-addr.arpa.j2

;
; BIND reverse data file for the Noise Palace
;
$TTL    604800
@       IN      SOA     noisepalace.home. root.noisepalace.home. (
                              1         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@		IN      NS      noisepalace.home.
{% for name in home_hosts.home_hosts %}
{% if home_hosts.home_hosts[name]['ip_addr'] is defined %}
{% set octet = home_hosts.home_hosts[name]['ip_addr'].split('.') %}
{{ octet[3] }}.{{ octet[2] }}   		IN      PTR     {{ name | lower }}.noisepalace.home.
{% endif %}
{% endfor %}

Other than the home_hosts prefix on this, this is okay. It does hard-code the domain name, but I only use one domain in my home infrastructure anyway, so I’m okay with this trade-off for now.

Other templates

The other templates are standard BIND9 configuration defining my zones, what I forward to for recursive DNS and other standard features. The only interesting part I have is adding my subnets in as “goodclients” (i.e. those that are allowed to query DNS from BIND9): -

acl goodclients {
  {% for network in netvars['ip_ranges'] %}{{ network }};
  {% endfor -%}
  localhost;
};

New Role: Outline

As noted, the previous approach had a few limitations that I wanted to address. The below is a quick summary of what I want to fix in the tasks: -

• Statically defined hosts
• Repetition of tasks
• Only reload/restart bind9 if a change has been made
• Consolidate the variables into a single file, available globally across all roles and hosts
• Remove redundant prefixing/namespacing of variables

For the templates, I wanted them to be simpler with less conditional logic.

Finally, I wanted to remove the primary/secondary roles from the servers, and just have each server answer with what it has.

I have found a few issues with records not replicating correctly between my two instances of bind9. Given both are managed with Ansible anyway, I don’t need to have the primary inform the secondary of what records exist.

New Role: Initialisation

To initialise the role, I ran ansible-galaxy init dns in the roles directory of my Ansible repository. I also added it to my overall Playbook like so: -

- hosts: ns-04.noisepalace.home,ns-03.noisepalace.home
  become: true
  roles:
    - role: dns
      tags:
        - dns
        - bind9

In the above, I have applied the role to my ns-04 and ns-03 hosts (two virtual machines on my network).

This is because the role is solving the Statically defined hosts issue. If I wanted to add additional servers, I can add them to the hosts list in the overall Playbook, rather than updating the role itself.

As an aside, why do I have ns-04 and ns-03 rather than ns-01 and ns-02? The simple reason is that when I moved from running everything on two Raspberry Pis and onto my two home servers, I didn’t want to end up breaking DNS in my house.

I had records for ns-01 and ns-02 pointing to the Raspberry Pis, so I could add the new servers, move all other machines/DHCP pools/anything else reliant on DNS to use the new machines, and then decommission the Raspberry Pis. If I had simply tried to replace them and it turned out I’d made a configuration error, I would have broken DNS (including access to the internet to help me fix it!).

New Role: Tasks

The tasks Playbook in roles/dns/tasks/main.yml looks like the following: -

---
# tasks file for dns
- name: Generate reverse lookup file
  template:
    src: db.168.192.in-addr.arpa.j2
    dest: /etc/bind/db.168.192.in-addr.arpa
    owner: root
    group: bind
    mode: 0644
  notify:
    - Reload bind9
  tags:
  - configuration

- name: Generate forward lookup file
  template:
    src: db.noisepalace.home.j2
    dest: /etc/bind/db.noisepalace.home
    owner: root
    group: bind
    mode: 0644
  notify:
    - Reload bind9
  tags:
  - configuration

- name: Named Local
  template:
    src: named.conf.local.j2
    dest: /etc/bind/named.conf.local
    owner: root
    group: bind
    mode: 0644
  notify:
    - Reload bind9
  tags:
  - configuration

- name: Named Options
  template:
    src: named.conf.options.j2
    dest: /etc/bind/named.conf.options
    owner: root
    group: bind
    mode: 0644
  notify:
    - Reload bind9
  tags:
  - configuration

- name: Stats directory
  file:
    path: /var/bind
    owner: bind
    group: bind
    mode: '0755'
    state: directory
  tags:
  - configuration

We have fewer tasks compared to the original Playbook This is partly due to reducing repetition and partly due to moving away from primary/secondary replication.

We also use something called notify. The purpose of notify is to trigger a Handler. When using templates and files, if a change has occurred, the handler is triggered. If no changes were required to the files on the servers, the handler is not triggered.

One interesting advantage of using handlers is that if multiple tasks refer to the same Handler, Ansible will wait until the last time it is called before triggering. This means that we don’t restart the service multiple times during the Playbook.

The handler is defined in roles/dns/handlers/main.yml as such: -

---
# handlers file for dns
- name: Reload bind9
  systemd:
    name: bind9
    state: reloaded
    enabled: yes
  tags:
  - configuration

This achieves the aims of reducing repetition of tasks, and only restarting/reloading bind9 when needed.

New Role: Templates

In terms of templates, I was able to remove a few (due to getting rid of Primary/Secondary roles). For the ones left, I made a couple of changes to make them easier to work with, and to provide more comments/help when reading the output files at a later date.

roles/dns/templates/db.noisepalace.home.j2

;
; BIND data file for local loopback interface
;
$TTL    604800
@       IN      SOA     noisepalace.home. root.noisepalace.home. (
                              2         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@               	IN      NS      ns4.noisepalace.home.
@               	IN      A       192.168.10.1
ns3             	IN      A       192.168.10.2
ns4             	IN      A       192.168.10.1
{% for network in networks %}
; Subnet: {{ network['subnet'] }}
; Purpose: {{ network['purpose'] }}
{% for host in network['hosts'] %}
{{ host['name']|lower }} IN      A      {{ host['ip_addr'] }} ; {{ host['description'] }}
{% endfor %}
{% endfor %}
{% for domain in cnames %}
{% for record in domain['records'] %}
{{ record['name']|lower }} IN      CNAME  {{ record['cname'] }}
{% endfor %}
{% endfor %} 

The redundant home_hosts prefix when looping through variables has been removed. The conditional check on whether this is a CNAME or an A record has also been removed (as CNAMEs are now in their own section of the variables). We also add some comments in to make the resulting file easier to navigate.

I will go through what has changed in the variables file soon so that you can see where the information is sourced from.

roles/dns/templates/db.168.192.in-addr.arpa.j2

;
; BIND reverse data file for the Noise Palace
;
$TTL    604800
@       IN      SOA     noisepalace.home. root.noisepalace.home. (
                              1         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@		IN      NS      noisepalace.home.
{% for network in networks %}
; Subnet: {{ network['subnet'] }}
; Purpose: {{ network['purpose'] }}
{% for host in network['hosts'] %}
{% set octet = host['ip_addr'].split('.') %}
{{ octet[3] }}.{{ octet[2] }}   		IN      PTR     {{ host['name']|lower }}.noisepalace.home.
{% endfor %}
{% endfor %}

This doesn’t differ much from the original template, except for some additional comments for each subnet. Still, we have removed the redundant home_hosts prefix on all the variables, so that is a win!

New Role: Variables

This section isn’t specific to this role. However it will help to show the difference between using local/directory-specific variables and using global variables usable by all hosts and roles.

Previously the variables for this role were found in other directories. We had to have specific tasks to include them in the Playbook (and any others that required it). The same would be true for any other application/Playbook I had created, as there was no global definition of variables.

Now in the overall Playbook, we have the following: -

- hosts: all,localhost
  become: false
  pre_tasks:
  - name: Include Global vars
    include_vars: vars.yml
    tags:
      - vars

This includes a global variables file that is available to all hosts and for all roles. Any changes in the global variables file are then available to all roles that require them.

I also changed the format of the variables to look like the below: -

[...]
networks:
  - subnet: 192.168.0.0/24
    purpose: Management
    domain: noisepalace.home
    hosts:
    - name: router
      mac: c4:ad:34:55:cb:03
      ip_addr: 192.168.0.1
      description: "Core Router - MikroTik 4011 Core"
      monitor:
        icmp: yes
        mikrotik_snmp: true
    - name: archspire
      mac: f4:4d:30:64:4d:bf
      ip_addr: 192.168.0.5
      description: "Archspire - Intel NUC VMH"
    - name: pinkfloyd
      mac: a8:20:66:34:f0:2b
      ip_addr: 192.168.0.6
      description: "PinkFloyd - Mac Mini VMH"
[...]
cnames:
  - domain: noisepalace.home
    records:
    - name: behemoth
      cname: behemoth-10g
    - name: pihole
      cname: netutil-01
    - name: grafana
      cname: pinkfloyd
    - name: prometheus
      cname: pinkfloyd
[...]

Before everything was a dictionary (i.e. keys and values). Now I have moved to using lists for the variables I want to iterate over. I find lists slightly easier to use in Jinja2 templates, and I also find it helps at first glance to see where one host starts and one host ends.

You’ll also notice that each subnet has a purpose (i.e. the kind of hosts that exist in it) and also each host has a description.

We saw in the templates that each DNS record has a description next to it (as the hostname alone may not always show enough information). Also all the records in a subnet will be prefixed by a comment that has the CIDR range (e.g. 192.168.0.0/24, 192.168.10.0/24) as well as the purpose of the subnet.

New Role: Summary

At this point, the role can now be used against multiple hosts (i.e. not hardcoded to two hosts), we’ve reduced repetition, made the variables available to all roles and we only perform a reload/restart on bind9 when required. The templates have been simplified (while also giving more information in the output files) and the primary/secondary DNS roles are gone.

I went through a similar exercise against other Playbooks, combining all variables that would be useful to other hosts into the global variables files, reducing repetition, and using handlers in all places I could.

What other roles do I use?

As well as the aforementioned DNS role, I have a number of other roles defined in my overall Playbook: -

base

This is the base role for anything I might want against all hosts.

Currently, this just installs gnupg2 on any Debian machines. Without it, adding Debian repositories fails because of a lack of GPG-key management software.

sshkeys

This takes my public SSH keys from my workstations/laptops/desktops/terminal apps on phones/tablets and adds them to the authorized_keys files for my primary user on each provisioned machine.

authkeys

This takes the same keys as above, but adds them to my Gitea, Gitlab and GitHub user so that I can clone my repositories using SSH.

Rather than using a native Ansible module for this, it uses a small Go tool I built to interact with the Gitea/Gitlab/GitHub APIs. I created the tool in part help learning Go, and it also makes adding other Git/VCS providers quite straightforward in future.

saltstack

This role adds the Saltstack repositories to any Debian or RHEL-based machine. These are the only official repositories available for distributions I use (i.e. Debian and Ubuntu primarily, with Alma/Rocky for testing sometimes).

It will then install the salt-minion package (from the official repository if available, from the distribution packages if not), update the configuration file to point to my Salt Master server, add a few useful grains and then restart the salt-minion daemon (to force it to register with the Salt Master).

blackbox_checks

For any hosts not managed by Salt (i.e. routers, switches, access points, IoT devices), this role will add them to my Prometheus checks so that I can monitor and alert on them. This uses the Prometheus blackbox_exporter that I have detailed in other posts.

cockpit

This install Cockpit on my hypervisors. This is only used for Virtual Machine consoles when I am not on a machine with virt-manager available, but it is useful to have nonetheless.

dhcp

This installs isc-dhcp-server, configures the pools for each of my subnets (i.e. one per VLAN in my network), any static IP reservations, and then reloads isc-dhcp-server when required.

drone

This installs the drone-exec-runner on two of my machines (one arm64 based, one that is running the Salt Master). I will go into much more detail on this in a later post.

cloudflaredns

For a number of DNS records, I add them to Cloudflare (who I use for DNS). The reason for this is so that I can use one of my purchased domains with Lets Encrypt to get valid certificates for the hosts on my network.

Quite a few applications now either warn when HTTPS isn’t used or uses a self-signed certificate. Some browsers now make you type thisisunsafe to accept a self-signed certificate. Other applications won’t even work without HTTPS.

I have been running HTTP only on any application in my network for years now, but I decided it was time to start using HTTPS properly.

As Cloudflare is my public DNS provider, I will toggle what hosts I want in my variables file using cf_add: true, and then they will be added to the zone for my purchased domain.

ipam

I run an instance of Netbox on my network, but I have been updating it manually. This role updates IP entries automatically, meaning I can see how much space I have left in each subnet, as well as just a general overview of what I have running.

All of the above roles make use of the same global variables files. This allows me to feed the DNS, DHCP, IPAM and Cloudflare roles with the same information.

The final overall Playbook that runs all of these roles looks like the following: -

- hosts: all,localhost
  become: false
  strategy: mitogen_free
  pre_tasks:
  - name: Include Global vars
    include_vars: vars.yml
    tags:
      - vars

- hosts: config-01.noisepalace.home
  strategy: mitogen_free
  roles:
    - role: sshkeys
      tags:
        - sshkeys
        - authkeys

- hosts: all
  strategy: mitogen_free
  become: true
  roles:
    - role: base
      tags:
        - base
    - role: authkeys
      tags:
        - authkeys
    - role: saltstack
      tags:
        - saltstack

- hosts: archspire.noisepalace.home
  strategy: mitogen_free
  become: true
  roles:
    - role: blackbox_checks
      tags:
        - blackbox
        - prometheus

- hosts: vmh
  strategy: mitogen_free
  become: true
  roles:
    - role: cockpit
      tags:
        - cockpit

- hosts: ns-04.noisepalace.home,ns-03.noisepalace.home
  strategy: mitogen_free
  become: true
  roles:
    - role: dns
      tags:
        - dns
        - bind9
    - role: dhcp
      tags:
        - dhcp

- hosts: piutil-01.noisepalace.home,config-01.noisepalace.home
  strategy: mitogen_free
  become: true
  roles:
    - role: drone
      tags:
        - drone

- connection: local
  strategy: mitogen_free
  hosts: localhost
  become: false
  roles:
    - role: cloudflaredns
      tags:
        - cloudflaredns
        - dns
    - role: ipam
      tags:
        - ipam

Mitogen?

One of the benefits of moving to Roles is making a change to a single variables file, and all the roles that use the added/changed variable (e.g. adding a new host/IP address) will roll out at the same time (so long as you don’t use ansible-playbook --tag $TAG or ansible-playbook --skip-tags $TAG).

The new Playbook and roles now runs against all of the hosts in my home and personal infrastructure. A change to add a new host will still run all the other roles too, even though not all may be relevant to it.

This means that a quick addition/fix takes much longer than it used to. There are a couple of tweaks you can do to speed up Ansible compared to the default behaviour.

General Ansible performance improvements

The first tweak is changing the “strategy”. The strategy defines the way Ansible runs tasks.

By default, Ansible uses the linear strategy. For each task in a playbook, Ansible waits for a response from each server that the task runs against before moving onto the next task.

Changing this to free means that each host will try to reach the end of the Playbook in the quickest time it can without waiting for other hosts.

Changing your default strategy to free can help with getting some hosts to complete as quickly as they can. This can help with overall Playbook execution time, and means that not all hosts are penalised by the slowest one.

The second tweak is to increase the amount of forks the Ansible process will make.

A fork is a separate process of Ansible that will run concurrently with others. If you have multiple cores on your machine(s) available, this will speed up time to process the playbooks by allowing Ansible to execute tasks concurrently. The more threads available, the less the instructions contend with each other for CPU time.

Is it enough?

Even with the above tweaks, the execution time for a Playbook with a few roles can still be significant.

Previously the single Playbooks I used would take around 1-3 minutes to execute. This depended on the amount of tasks, amount of files to manage and any loops. I may have been running multiple Playbooks, but if I only wanted to update DNS, it would only take or a minute or two.

With the move to using a number of roles against multiple hosts, the overall execution time reached around 15-20 minutes. If I had to run every single one of my old Playbooks, this may have taken a similar amount of time. However other than adding a brand new host, it was rare that I needed to run lots of Playbooks at once.

At this point, I was considering separating out the single Playbook with multiple roles into a few separate playbooks with some of the roles. I could then add in conditional checks (or even run separate repositories) so that each role didn’t need to run every time.

However this goes against the idea of moving to this approach, in that a single variable change may affect multiple Roles, so having some Playbooks run and not others could leave drift in configuration. Also, if I did need to run every role (i.e. new host being added), it would still take the full execution time anyway.

After a little research on improving Ansible performance and execution time, I came across Mitogen.

What is Mitogen?

Mitogen is a Python library that is used to write distributed programs, and to provide native Python APIs where shell commands/interaction is usually required. While Ansible does primarily use Python, it also does call out to the shell from Python for quite a few different operations.

Mitogen also provide a way to use the library with Ansible directly. It replaces (where possible) calling out to the shell with pure Python equivalents, re-uses SSH connections and more. Mitogen claims that this will speed up Playbook execution anywhere from 1.5 to 7 times, as well as reducing CPU consumption considerably. They specifically mention that any task/action that uses loops or with_* directives should see noticeable improvements.

To use Mitogen with Ansible, there are a couple of options. If you plan to run Ansible from a standard host (i.e. a bastion host, your machine), then either clone the repository from their GitHub (or download a release tarball from their releases page) and refer to the location of the repository/extracted files in your Ansible configuration.

If you intend to run Ansible from a container/image, then you need to ensure that either the image/container has Mitogen installed/included, or that it is included as part of your Ansible repository.

To include it in your repository, you can either add it as a Git submodule, or extract the contents of the release tarball into a directory in your repository directly.

Once you have done this, create/update ansible.cfg in your repository with the following: -

[defaults]
host_key_checking = False
strategy_plugins = mitogen/ansible_mitogen/plugins/strategy

I have disabled host_key_checking because if Ansible runs in an ephemeral container, it hasn’t SSH’d to any of the hosts previously. Without this, running Ansible will fail from a container, as the process for accepting a host key would require user input.

In addition to this, I add the strategy: mitogen_free to the tasks in the overall Playbook. You can set this as a default option in the ansible.cfg file.

I have found that some roles/collections may have issues with Mitogen (the netbox collection didn’t work on a release candidate of Mitogen, but does work on the latest GA release). Having the option to enable it per task/role inclusion allows me to run it against all roles it is compatible with, and revert to the in-built Ansible strategies if a role/collection has issues.

Did it make a difference?

All of this would be moot if there were no improvements in execution time. So what difference has it made?

I am skipping ahead in the series slightly in showing the results from my Drone CI jobs. They are a good benchmark for the time to execute all the roles, and all of them run all roles defined in the overall Playbook.

Only using forks

I set the amount of forks set to 15 ever since I started moving to roles, so I don’t have any benchmarks/execution times without it!

The times at the end (15:51 and 16:11) are how long the jobs (in minutes and seconds) have taken to execute. The Pull Request (using --check and --diff mode only, no actual changes applied) takes just over 16 minutes. When the PR is merged, it takes just under 16 minutes.

This is quite a long time to wait for a single DNS change or adding a host!

Using the standard free strategy

My first attempt to reduce the execution time was to change the strategy to free: -

That is quite an improvement, saving 4 and 5 minutes of execution time.

Allowing hosts to get to the end of a playbook quicker saves on execution time overall. This makes sense, as some hosts may be quicker at some tasks and slower at others. Allowing them to not be penalised by other hosts (or to not slow down other hosts) cumulatively saves time overall.

Using Mitogen

We are already using the free strategy, so what difference do we see when moving to mitogen_free?

Wow, that is a huge difference! I can now do a full Pull Request AND Merge executions in less time than it had taken to run one or the other before.

5 minutes of execution time is still more than the 1-3 minutes it used to take to apply my single Playbooks previously, but bear in mind that this is now applying 11 roles against all hosts. For me, this is an acceptable trade-off for knowing that every host has all roles and the latest changes applied.

Other changes

If you look over my list of roles I now use with Ansible, you may notice something missing.

Didn’t you say you used Ansible to configure Prometheus?

Yes, earlier in this post I mentioned that before this refactor, Ansible would: -

Update Prometheus configuration
Install Prometheus exporters on new machines

Why wasn’t it included in the “What other roles do I use?” section?

As part of this refactor, I thought about why I run both Ansible and Saltstack in my home infrastructure. Both achieve very similar goals.

One of the main differences between the two is that Ansible isn’t agent-based. It doesn’t need a machine/node to be bootstrapped/have additional packages installed to manage it, which is very useful when configuring a brand new host.

However Salt is incredibly quick in my experience. The section on performance tweaks and using Mitogen with Ansible is something you don’t necessarily need to think about too much with Salt, as it has a lot of performance advantages to begin with.

Salt uses a queuing mechanism (based upon ZeroMQ). The Salt Master doesn’t tell each node what to do directly. Instead it adds messages to the queue and then the Agents will execute the actions describe by the messages (more information on this here).

The Salt agents then execute all the commands and operations to configure the server they run on. By contrast, Ansible has to tell each host what command/execution/task are required, and controls the order/strategy to execute them.

Because of this, I decided to use Ansible for what it is (in my opinion) best at, bootstrapping nodes (i.e. installing allowed SSH keys, configuring DHCP/DNS, installing the Salt agent) and managing nodes that can’t run agents.

Ansible knows about every host on my network (in that there are variables that have the IP address/MAC address etc) for anything that needs a static allocation and needs basic monitoring (i.e. those that cannot run agents themselves), so it makes sense to add them into monitoring/DNS/DHCP with Ansible during the bootstrap/playbook run stage.

For hosts that can run Agents (i.e. all my home servers/virtual machines/VPSs etc), everything other than the initial bootstrap is managed by Salt. They get a default set of Prometheus exporters, ICMP monitoring, base packages and more. I can then include other applications for the machine to run, as well as the associated Prometheus exporter(s) that will be monitor the deployed applications (e.g. Wireguard and the wireguard_exporter, Docker and cadvisor).

To show the difference in execution time between the two, here is an example of Salt running against all hosts, applying all relevant states: -

My Salt environment has 47 different states/applications it can deploy. Even with all the tweaks I have done to my Ansible setup, Salt achieves a lot more in a lot less time!

This isn’t to say Ansible is bad, as it isn’t. I love and use Ansible in a lot of different ways, and could easily use only Ansible to manage everything on my infrastructure.

I like the option of having both Ansible and Salt. I can play to their strengths, rather than making compromises to use only one or the other.

Summary

Hopefully this post helps you in either understanding Ansible roles, how to transition from single Playbooks to Ansible Roles (and why you would want to) and also helps with reducing the execution time of your Playbooks too.

The move from single Playbooks to Roles was probably the biggest part of the refactor, but I am now able to rely solely on my continuous integration to test and implement my Playbooks. Being able to trigger a full update of my infrastructure by making updates in the Gitea mobile app (GitNex) is a wonderful thing 😁.

Next post

In the next post, I will cover the changes that have improved my Salt environment. This isn’t as big of an overhaul as Ansible was, as I already had recent professional experience of Salt when I started using it at home.

However there are a few interesting changes I made that require less manual updates to the Salt states and pillars, or any in the case of a host that I am just testing/not running extra applications on.

Previously, most of my personal infrastructure was managed by single-purpose Ansible Playbooks and a minimal set of Salt states. When I first got back into Ansible (mid-to-late 2019) after a break of about 3-4 years, I didn’t have a great understanding of Ansible Roles.

These Playbooks and States served their purpose for quite a while. After having managed DNS/DHCP manually (including SQL database manipulation with PowerDNS) and no monitoring at all, this was a massive step up and made managing my infrastructure much easier.

Why change?

So what changed? What problems did I still want to solve?

Firstly, a lot of my personal infrastructure was still managed ad-hoc (i.e. logging in and installing/updating configuration). Everything from deploying SQL databases, managing IP ranges, VPN configuration, and more.

Also despite using Ansible and Salt, most of the Playbooks and States were not very idempotent. They would work fine on the first run, but a subsequent run would fail.

Finally, after the onboard network interface on one of my home servers (Pink Floyd) started to drop out frequently, it made me paranoid that if one of my home servers failed, I would struggle to return a replacement to the previous state.

All in all, Ansible had helped in solving my biggest pain points, but I couldn’t rely on it if something went wrong.

Rethinking the approach

As happens quite often in tech, a failure usually leads to making improvements. Before it became apparent that the network interface on Pink Floyd was failing, I had assumed that I was reaching the limit of the server’s resources. I’d recently added a few virtual machines (mostly for development testing and playing with eBPF), and soon after I started seeing frequent drop outs in my monitoring.

This prompted me to make a few changes: -

• Changing the scrape_interval for some intensive/long-running checks on my infrastructure
• Remove a few virtual machines
• Move some applications/machines to my other home server (Archspire)
• Use containers more where possible

As part of this, I removed a self-hosted Gitlab instance. I originally intended to use Gitlab CI on my infrastructure, but struggled to dedicate the time to it.

I moved all the repositories from the Gitlab instance to my Gitea instance and removed the Gitlab machine. I still wanted to move to using a CI solution, but using the in-built Gitlab approach was no longer an option.

Upgrades

After a little more investigation (and discovering the network interface was at fault), I added a USB NIC to Pink Floyd and it has been running fine ever since. The USB NIC also supports 2.5Gbe, so once I buy a switch(es) that support multi-gig, I can take advantage of the extra bandwidth. For now, this just means I have a functioning interface (which is a big upgrade from an intermittently functioning interface!).

After moving a few applications and machines over to Archspire (and seeing the available memory/disk space dwindle), I decided it was time to give the machine a bit of a boost. I bought 2 16Gb of DDR4 SO-DIMM sticks of RAM and a 512Gb NVME, doubling both the memory and storage on this machine. This also helped solved some issues with a few applications/machines running out of memory (e.g. a Unifi Controller container, the Salt Master server), always a bonus!

Finally, I added two more VPSs for serving external services (i.e. this website, RSS, read-it-later, external monitoring). In addition to my existing Digital Ocean droplet, I now also run two Hetzner Hcloud instances.

Gitea and CI

For those who haven’t used Gitea, it is a self-hosted Git forge similar to GitHub, Gitlab and BitBucket. It is written in Go, and the resource usage is much lower than the self-hosted versions of GitHub, Gitlab or BitBucket. It started as a fork of Gogs but now appears to be more popular.

As already mentioned, I wanted to starting using continuous integration with my infrastructure. Now that I had removed my Gitlab instance, I couldn’t use Gitlab CI. Gitea itself doesn’t have any native continuous integration support out of the box. This is fine if all you need is Git repository hosting, but I wanted to start making use of CI because: -

• I wanted to lint and test my Ansible and Salt configuration before applying it
• I now write a lot of tools in Go. I want to generate releases I can use across all of my machines (rather than building manually and SCP/rsync-ing them around), as well as lint and test them
• I want to take the GitOps approach of updating a repository and the changes happen automatically, making Git the source of truth (and not the infrastructure)
  • It is much easier to update a line in a file and commit it than it is to login to a machine(s) to run a series of commands

It also needed to be self-hosted (because this is all running on a local Git instance, not a publicly available one), and ideally quite low in resource usage.

After some searching around (and finding the awesome-gitea repository), I found Drone.

Drone

Drone is a CI/CD platform that uses yaml to define the steps to take when code is committed into your repository. It works in a similar way to the previously mentioned Gitlab CI and GitHub Actions, while not being tied to either platform. It integrates natively with Gitea, and can run jobs in different ways: -

• In containers (e.g. Docker)
  • For most jobs this makes sense (e.g. building Go tools, running Ansible Playbooks)
• Using the exec runner to run commands directly on a machine (for when running in a container isn’t viable)
  • A good example of the use case is with Salt, as the Salt Master can tell all Minions to update
• Using the ssh runner to run commands on a machine available via SSH (for hosts that do not support containers and can’t run the exec runner).
  • Running commands on something like OpenBSD, or a network device, where the agents do not run correctly (or at all in the case of network gear)

There is an enterprise version of Drone that supports ephemeral agents (i.e. the agents spin up when required, rather than running at all times), clustering and more, I don’t require any of these features in my infrastructure, so the OSS version is more than enough for my needs.

Each repository that you want to run tasks on required a .drone.yml file, with a format quite similar to Gitlab CI/GitHub Actions/many other CI solutions: -

kind: pipeline
name: default
type: docker

trigger:
  branch:
    - main

steps:
  - name: submodules
    image: alpine/git
    commands:
    - git submodule update --init --recursive

  - name: spellcheck
    image: tmaier/markdown-spellcheck
    commands:
      - mdspell --ignore-numbers --ignore-acronyms --report "content/**/*.md"
    when:
      event:
       - pull_request
[...]

Each step will run when an event triggers it. For example, you could have a step which performs linting, but only when a pull request is raised. You could also have it deploy a version of your code to a staging/testing environment before it is merged into your main branch, and then roll out to production after it is merged.

Drone also has a number of plugins so that you don’t need to define every step yourself. For example, it already has plugins for Ansible and Hugo, as well as sending notifications to a number of platforms (Slack, Matrix, Telegram etc).

Configuration management

To ensure that all of my Ansible and Salt configuration was ready to be used with a CI platform (Drone in my case) then it needed a huge refactor. The requirements were: -

• Drone (or any other CI tool or automated process) can run them
• Ansible must use roles
• The CI job must run against all hosts and with all the necessary roles
  • This ensures all necessary changes go out at once, not when I remember to run them!
• Ansible and Salt must be idempotent (i.e. running the Playbooks/States again does not fail or make further changes)
• Tasks/actions only run when required (i.e. do not restart a service on every run, only when the configuration/version changes)

My previous posts on Salt and Ansible followed these rules, I just never applied the same rules to my own infrastructure before! (Practice What You Preach)

Containers

I have been using and working with containers for about the past 6 years, and have run a few on my personal infrastructure previously. However it was all very ad-hoc, no consistency, and most of the applications I run were still native packages and/or installed inside virtual machines. Examples of those which I have ran on VPSs/Virtual Machines which could very easily be in containers are: -

• Hugo (for hosting this blog)
• Read-it-later services
• RSS
• Netbox
• Oxidized

During this overhaul, I moved all of the above services, and added some other services too in the process.

Now the big question people may ask is am I running all of this on Kubernetes? The answer is not yet. I am comfortable with Kubernetes as a platform, having managed and deployed the platform and applications on Kubernetes in multiple companies now. However I don’t feel like my infrastructure is quite ready for it yet (having two home servers goes against the idea of quorum for a start!), and it would require refactoring everything about my personal infrastructure. Given how long it has taken me to find the time to fix all the current issues with my infrastructure, a full refactor isn’t on the cards just yet.

More to come

There is a lot to cover in what has changed, especially on the configuration management side. Rather than try and put this all into one huge post, this will be split out into: -

• Moving from basic Ansible Playbooks to using Roles
• Improving my Salt configuration files
• Managing dependencies and optional tasks/actions in both Ansible and Salt
• Setting up Drone
• Using Drone with Ansible and Salt
• Using Drone with Go to build releases
• Using Drone to run tasks on OpenBSD
• Using Drone to build this website (including some useful linting/checking)
• Managing secrets that can be used by Ansible, Salt and Drone
• Additional services that provide benefits to my home and personal infrastructure
• What else I want to do in future

A lot of these posts are not specific to my home infrastructure, and should hopefully provide benefit to anyone using Ansible, Salt, Drone, or those who just like reading about other people’s home labs (like me!).

Firstly, I started a new job in May. Any new role takes time to get up to speed, this one no different. I am finding my feet there now, freeing up more mental capacity to look at new technologies and interests.

Secondly, I have struggled somewhat with ideas for new posts. Part of this is that I am getting more into writing Go, and even learning some C (mainly for working with eBPF). As I’m still very much in the learning phase with these, I don’t feel like I have much to say on either topic yet. Everything I’m doing has many posts and resources on the wider internet already . I will probably put something together soon on all of this, but I still don’t feel comfortable just yet.

In terms of more infrastructure-related technologies, I haven’t really found anything new and interesting since discovering Pulumi. My experience is much greater on infrastructure-style projects and technologies, and where I feel most comfortable writing posts about. I just need something to come along and wow me again!

In the meantime, I am delving into other hobbies like gaming (specifically Mass Effect Legendary Edition) and building keyboards: -

It's alive!

Need to play with the layout a little as it isn't quite to my liking, but it works 😁

Wonderful PCB and kit @pauljames! pic.twitter.com/JfCdulUOqk

— Stuart Howlette (@stuh84) September 12, 2021

As far as this site goes, I’ve made a couple of minor changes.

The first is that I’ve removed comments. I have rarely found the need to read and write comments on other people’s blogs and sites, at which point I found the idea of having them myself made little sense. Other contact methods are available to reach me, so if you want to discuss something on here, get in touch with me directly!

Secondly, I have removed Google Analytics. Initially when I started this site I wanted to see the interaction and growth on my posts. However as time has passed, I have realised this isn’t a useful metric. The posts that have received the most visits are those which are either those which are too far removed from my current interests/work or for technologies I haven’t used in a couple of years (and therefore cannot provide any follow up on).

There were times that I tried to cater my posts to what I thought would give higher interaction (although I was often wrong!). I didn’t enjoy writing these posts anywhere near as much as something I just felt genuinely excited about (e.g. Pulumi, Prometheus).

Also I have, and have had for a long time, massive privacy concerns with Google and analytics in general. I already felt conflicted in using analytics when I first started this site, and over time I have grown increasingly uncomfortable with having them.

So from now on, there are no comments, no analytics, just the site and nothing else.

I will return soon with some new posts, hopefully a lot sooner than it has been between this post and the last one!

With a recent job change came the chance to learn and use Puppet. For me, the best way to learn a different tool-set is to take something you have done before, and then try and do it using the new tool. In this case, we are using Puppet to configure Consul, Prometheus and the Prometheus node_exporter to monitor Linux machines (similar to a previous series using Saltstack).

What is Puppet?

Puppet allows you to manage a fleet of servers and other devices. The benefits of this are increasing consistency in application and infrastructure configuration, as well as reducing the toil (i.e. human interaction) of managing infrastructure.

Puppet’s first release was in 2005, which predates Chef (2009), Saltstack (2011) and Ansible (2012), with only CFEngine being released beforehand. While other configuration management systems were release between CFEngine and Puppet, only CFEngine is still under active development. This means that some of the early advances in configuration management were made with Puppet, with the influence being seen in nearly all configuration management tools since.

Puppet’s syntax is based upon a subset of the Ruby language. The Puppet Agent (the client that runs on servers) is written in Ruby as well. Up until recent versions of Puppet, the server side component of Puppet was also written in Ruby, but has transitioned to being written in Clojure (which itself is based upon Java).

Puppet has the concept of manifests, which are the equivalent of Saltstack’s state files, and Ansible’s playbooks. You can also run different environments, allowing you to have common code, but parameters specific to different classes of infrastructure (e.g. staging, production, different regions etc).

A collection of manifests, along with static files and templates, is known as a Puppet module. They are separated by directory (e.g. $module_directory/my_module/manifests and $module_directory/my_other_module/manifests), with the directory structure defining how they are referenced.

Variables can be defined directly inside the manifests themselves, or using Hiera.

Hiera?

Hiera is a way of separating out the configuration itself from data. For example, if you want to install a package, the action to be taken is defined in a manifest (i.e. package installation), but the version can be specified in Hiera.

Hiera data is written in YAML, It can be environment specific, and even node specific. You could refer to different DNS servers, different package versions (e.g. a testing/beta package for your staging environment, prior to general release), or whatever else you deem necessary.

What does a manifest look like?

Puppet manifests contain classes, which are unique blocks of code that define an action (or actions) to be taken. They can be referred to directly, inherited by other classes (e.g. defining a base class, and then extending them in other classes where necessary), or you can refer to them from other classes, so that multiple “child” classes are referenced using a “parent” class.

Using Debian (i.e. apt) package installation as an example, a basic manifest looks like the below: -

class packages::qemu {
    package{'qemu-guest-agent': 
        ensure => 'installed'
    }
}

The equivalent Ansible code is below: -

- name: Install qemu-guest-agent
  package:
    name: qemu-guest-agent
    state: present

The equivalent Saltstack code would be: -

# Shorthand
qemu-guest-agent:
  pkg.installed

#Longhand
install_qemu_guest_agent:
  pkg.installed:
    - pkgs: 
      - qemu-guest-agent

The directory structure in Puppet defines how you refer to classes as well. Each class must be in one of your modules directory, and must be in a directory that is the same as your class name. For example, our class packages::qemu must be in the $module_directory/packages/manifests directory. If it was in the mypackages or qemupackages directory instead, Puppet would throw an error. Your directory structure matters more than in other configuration management systems, with Ansible and Saltstack not requiring named tasks/actions/blocks of code.

Roles and Profiles

Roles and Profiles is a way of structuring your Puppet code that adds abstractions which lead to more generic modules (rather than environment or node-specific code). This method was referenced in this post back in 2012, and has since become a semi-official way of defining Puppet code.

The idea is that rather than attaching classes (i.e. the code blocks previously mentioned) directly to a node or nodes, you create a node role and associate profiles with this node. For example, you could create a base role that includes standard packages, monitoring agents and other common configuration, but then create a separate role that is only used by the monitoring servers (in our case, Prometheus).

Roles can be extensions of other roles, meaning that if you choose to assign say, a prometheus role to the prometheus server, it will also inherit the base role too, which includes all common configuration.

A role can contain multiple profiles, which can be application specific, refer to an application and its dependencies, or profiles that align with how your business deploys applications. For example you could require that all applications must register a service with Consul, a set of firewall rules and SELinux policies, which would all be referred to in a single profile (or multiple profiles and grouped by class).

This means that it is perfectly possible to create a set of base roles and profiles that all nodes must use, but then additional profiles can be referenced for installing/managing the applications running on them.

Aims

In this post, we will do the following: -

• Set up a Puppet Server
• Install multiple Linux servers (physical and virtual) with Puppet agents
  • We will have machines running Debian (plain Buster and Proxmox), Rocky Linux, Ubuntu and Alma Linux
• Run Prometheus on one of the servers
• Install and run Hashicorp’s Consul on all of the servers (for service discovery)
• Install and run the Prometheus Node Exporter on all of the servers
• Register the Prometheus node_exporter as a service in Consul so that Prometheus can discover and monitor the servers

I decided to include Rocky and Alma Linux to see how well Puppet works on both Debian-based and RHEL-based distributions, while also seeing the progress of both distributions vying to be the “replacement” for pre-Stream CentOS.

Setting up a Puppet Server

The Puppet Server for this post is running on Debian Buster, with 2 CPU cores and 2G of memory. In a production scenario you would want to use more RAM and CPU, but for the purposes of this post, this is more than enough.

Add the Apt repository

To install the Puppet Apt repository, follow the instructions here. To summarise, you install a Debian package that adds the Puppet repository to your machine, including the relevant GPG keys.

Install the Puppet Server

As the Puppet Server uses Clojure, first install a compatible Java Virtual Machine. You can install either a Java 8 or 11 compatible JVM. The easiest to get started with is the OpenJDK (install with apt install openjdk-11-jre-headless). If you want to choose an alternative (e.g. Amazon’s Corretto, the official Oracle JVM), then Puppet supports these too.

After this, install the Puppet Server using apt install puppetserver. You can also install the Puppet Agent (using apt install puppet-agent) so that the configuration of the server is also managed by Puppet.

Configure the server

After installing the server we then need to configure it. It has mostly sane defaults, but we do need to make a couple of changes. If you are using less than 4Gb of memory, or you are using significantly more (and therefore have more resources that Puppet could use), update the memory assigned the JVM in /etc/default/puppetserver: -

# Modify this if you'd like to change the memory allocation, enable JMX, etc
JAVA_ARGS="-Xms1g -Xmx1g -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger"

The -Xms (initial memory assigned) and -Xmx (maximum amount of memory) are 2g (i.e. 2Gb) of memory by default. With the machine I am running, this would take up all available memory, so I reduced this to 1g (i.e. 1Gb) instead. If your Puppet Server has more memory available (say 8Gb, 16Gb or more), then raise this so that the server has more resources available to process more agents.

I also update the basemodulepath variable in /etc/puppet/puppet.conf to point to multiple directories: -

basemodulepath = $codedir/forge:$codedir/modules:$codedir/custom:modules

The $codedir can be changed, but by default is /etc/puppetlabs/code (i.e. where all of your manifest and modules reside). The purpose of each directory is: -

• module - Standard directory for modules
• forge - Modules downloaded from Puppet Forge
• custom - Modules that I have created

Puppet Forge is a place where users can submit modules they have created, which allows you to make use of other peoples code in your Puppet environment. This can be everything from package management to firewall rules. Using modules from Puppet Forge allows you to focus on what is unique to your environment, with common tasks using tried and tested modules.

Add Puppet to $PATH

By default, Puppet is installed in the /opt/puppetlabs directory when on Linux. This is fine, except that if you want to interact with the Puppet Server or Puppet Agent, you have to refer to /opt/puppetlabs/bin/puppet or /opt/puppetlabs/bin/puppetserver.

The easiest way to change this is to add this to your $PATH variable (usually defined in ~/.bashrc, ~/.zshrc, ~/.profile, or other shell configuration file). However Puppet usually requires privileged access (i.e. sudo or root). To avoid running commands as root, you can update an option in your sudo configuration called the secure_path. This is the $PATH variable that sudo uses, which makes it available to all users with sudo access.

This variable is in /etc/sudoers on both Debian-based and RHEL-based distributions. For example, on Debian, you would update it to this: -

Defaults	secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/puppetlabs/bin"

At this point, you can run sudo puppet or sudo puppetserver directly.

Setting up machines as Puppet Agents

The next task is to install the agents on our nodes. In this post, I have installed the agent on: -

• A Debian Buster virtual machine with 2Gb of RAM and 2 CPU cores, running the Puppet server
• A Debian Buster virtual machine with 2Gb of RAM and 2 CPU cores, to run Prometheus
• An Ubuntu virtual machine with 1 CPU core and 2Gb of RAM
• A Rocky Linux virtual machine with 2 CPU cores and 2Gb of RAM
• An Alma Linux virtual machine with 2 CPU cores and 2Gb of RAM
• Two Proxmox (based upon Debian Buster) hypervisors that run all of the aforementioned machines
  • One is a Lenovo X220 with an Intel i5-2420m and 16Gb of RAM
  • One is a HP Elitebook 9470m with an Intel i7-3687u and 16Gb of RAM
  • Both are running in a Proxmox cluster

Installing the repositories

As with the Puppet Server installation, when on Debian or Debian-like systems (e.g. Ubuntu), download the Debian package that enables the repository. This is the same repository as used for the server. If you are on Debian, do the following: -

# Debian Buster
$ wget https://apt.puppet.com/puppet7-release-buster.deb

# Ubuntu Focal Fossa (20.04 LTS)
$ wget https://apt.puppet.com/puppet7-release-focal.deb

You then install the repository using dpkg -i puppet7-release-$DISTRO.deb.

On RHEL-based distributions, you can install the RPMs directly using: –

# RHEL7-based (eg CentOS 7, Red Hat Enterprise Linux 7, Oracle Linux 7)
$ rpm -Uvh https://yum.puppet.com/puppet7-release-el-7.noarch.rpm

# RHEL8-based (eg Rocky Linux 8, Alma Linux 8, Red Hat Enterprise Linux 8)
$ rpm -Uvh https://yum.puppet.com/puppet7-release-el-8.noarch.rpm

Installing the Puppet Agent

Once the repositories are enabled, install the Puppet Agent with either apt install puppet-agent, yum install puppet-agent or dnf install puppet-agent (depending on the distribution). The Puppet Agent is mostly self contained, so you do not need to install anything like Ruby or a Java Virtual Machine to start using it.

Once the agent is installed, you can register the agents against the Puppet Server

Registering Agents

Configure the Puppet Server

To register agents, they need to know what server they are registering with. You can update the /etc/puppet/puppet.conf file, or you can run the following command: -

$ sudo puppet config set server $DNS_OF_PUPPETSERVER --section main

Replace $DNS_OF_PUPPETSERVER with the DNS hostname of your Puppet Server. For me, this is puppetserver.meshuggah.yetiops.lab. This updates the Puppet configuration file to refer to the correct Puppet server to look like the below: -

[main]
server = puppetserver.meshuggah.yetiops.lab

SSL Bootstrap

Puppet Agents register against the server by creating a certificate signing request. This must be signed by the server to confirm that the Agent is authorized to run manifests and modules retrieved from.

To do this on a machine running the Puppet agent, run puppet ssl bootstrap: -

$ sudo puppet ssl bootstrap
Info: Creating a new RSA SSL key for prometheus.meshuggah.yetiops.lab
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for prometheus.meshuggah.yetiops.lab
Info: Certificate Request fingerprint (SHA256): 50:2A:55:A1:99:72:9F:A0:43:0C:A0:76:CD:01:24:9E:A0:63:19:EC:61:2E:50:1B:22:44:8A:4D:CF:0A:A4:3D
Info: Certificate for prometheus.meshuggah.yetiops.lab has not been signed yet
Couldn't fetch certificate from CA server; you might still need to sign this agent's certificate (prometheus.meshuggah.yetiops.lab).
Info: Will try again in 120 seconds.

Login to Puppet Server and sign the certificate with puppetserver ca sign --certname prometheus.meshuggah.yetiops.lab: -

$ sudo puppetserver ca sign --certname prometheus.meshuggah.yetiops.lab
Successfully signed certificate request for prometheus.meshuggah.yetiops.lab

The Puppet Agent will then check if the request has been signed within 2 minutes (120 seconds), and then confirm that the process is complete: -

Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for prometheus.meshuggah.yetiops.lab
Info: Certificate Request fingerprint (SHA256): 50:2A:55:A1:99:72:9F:A0:43:0C:A0:76:CD:01:24:9E:A0:63:19:EC:61:2E:50:1B:22:44:8A:4D:CF:0A:A4:3D
Info: Downloaded certificate for prometheus.meshuggah.yetiops.lab from https://puppetserver.meshuggah.yetiops.lab:8140/puppet-ca/v1
Notice: Completed SSL initialization

Other tasks

Autosigning

If you manage a lot of infrastructure, the process of signing each individual server can become quite unwieldy and time consuming. Instead you can use autosigning. This comes with a caveat in that, this potentially makes your Puppet estate less secure, as the rules can be quite open for what will be autosigned.

However if you are only on a private network with no external access, or are willing to overlook the potential security implications for the gains in time and automation, you can create rules in a file called /etc/puppetlabs/puppet/autosign.conf for Agents that should have their certificate signing requests automatically approved and assigned. More information on this is available here, but at a very basic level, you can autosign anything with a certain domain name like so: -

$ cat autosign.conf
*.meshuggah.yetiops.lab

View all registered agents

At this point, you can view all registered agents on the Server with: -

$ sudo puppetserver ca list --all
Signed Certificates:
    prometheus.meshuggah.yetiops.lab         (SHA256)  27:0C:C5:79:00:20:21:36:72:90:3C:DD:C5:2A:BE:5D:27:BA:A7:A2:B5:53:11:C9:EC:F6:88:10:AF:92:EC:92	alt names: ["DNS:prometheus.meshuggah.yetiops.lab"]
    ubuntu-util.meshuggah.yetiops.lab        (SHA256)  BF:59:C4:B5:68:68:E1:2F:F1:74:D1:28:33:D3:75:00:17:63:6F:E9:2F:4D:2A:62:44:19:C0:E0:BC:3A:3A:AD	alt names: ["DNS:ubuntu-util.meshuggah.yetiops.lab"]
    almautil.meshuggah.yetiops.lab           (SHA256)  1F:C0:51:D8:BF:48:FF:F7:C8:1B:87:CA:21:A6:29:4E:F3:71:5E:AF:58:65:13:51:CD:61:4A:AA:44:FB:3B:F6	alt names: ["DNS:almautil.meshuggah.yetiops.lab"]
    pve.meshuggah.yetiops.lab                (SHA256)  04:C5:BE:D9:EE:55:71:1F:26:39:94:4C:1F:CC:9C:82:C6:2F:BD:A2:81:96:DC:5B:5B:61:0B:38:17:4E:5E:FD	alt names: ["DNS:pve.meshuggah.yetiops.lab"]
    pve2.meshuggah.yetiops.lab               (SHA256)  FD:66:5F:D7:46:AF:38:91:31:4A:26:65:76:FA:4B:AE:37:B0:FD:E8:A0:01:38:6F:3E:D5:19:E0:33:D1:B9:37	alt names: ["DNS:pve2.meshuggah.yetiops.lab"]
    rockyutil.meshuggah.yetiops.lab          (SHA256)  95:8B:C2:06:0F:D9:81:00:6D:B5:B7:DB:FA:78:74:C2:6C:54:1C:27:85:51:F8:12:D7:FF:80:69:3F:A8:B3:ED	alt names: ["DNS:rockyutil.meshuggah.yetiops.lab"]
    puppetserver.meshuggah.yetiops.lab       (SHA256)  F4:C6:B6:11:A7:CC:03:5E:0B:79:1A:9B:F9:37:7D:CA:9E:63:7A:5D:C2:DA:5D:0A:79:4F:EE:6C:93:44:4B:91	alt names: ["DNS:puppet", "DNS:puppetserver.meshuggah.yetiops.lab"]	authorization extensions: [pp_cli_auth: true]

If you run this without the --all option, it will show all unsigned requests, which is useful for if you need to determine the certname (effectively the hostname) of Agents that have registered.

Forge modules

As we don’t want to define our own code to install packages, define firewall rules and other “basic” operating system tasks, we add Puppet modules from Puppet Forge. It is possible to install them all manually. However you may prefer to use a module manager for this.

There are multiple module managers available, including r10k (which is one of the most known and popular). For the purposes of this post, I am using one called librarian-puppet (mainly because this is what my workplace uses). It is supplied as a Ruby Gem, which is installed using gem install librarian-puppet.

Once it is installed, run librarian-puppet init from inside the /etc/puppetlabs directory. This creates a Puppetfile, which you populate with the required modules and versions. I also run librarian-puppet config path code/forge to make sure that any modules installed by librarian-puppet go into a separate directory, reducing the risk of conflicting with my own modules.

My Puppetfile looks like the below: -

#!/usr/bin/env ruby
#^syntax detection

forge 'https://forgeapi.puppetlabs.com'

mod   'puppet-archive',              '3.2.0'
mod   'puppet-boolean',              '2.0.1'
mod   'puppetlabs-apt',              '4.5.1'
mod   'puppetlabs-concat',           '4.2.1'
mod   'puppetlabs-firewall',         '1.12.0'
mod   'puppet-firewalld',            '4.4.0'

If you then run librarian-puppet install, all of the above modules with their specified versions are installed, allowing you to refer to them in your code.

Puppet Module: Setting an MOTD

The first module we create is a Message Of The Day module. This is displayed to users on login. First, create a folder called motd in your modules directory (in my case /etc/puppetlabs/code/custom/motd) and then create a manifests and templates directory.

The following is the directory structure of this module: -

/etc/puppetlabs/code$ tree custom/motd/
custom/motd/
├── manifests
│   └── env.pp
└── templates
    └── motd.erb

Manifest

The env.pp manifest looks like the below: -

class motd::env {
	$pve_host = hiera('pve_host', '')
	file { '/etc/motd':
		content => template('motd/motd.erb'),
		owner   => 'root',
		group   => 'root',
		mode	=> '644',
	}
}

The Puppet class is called motd::env. In this we: -

• Lookup a variable called pve_host that we define in hiera data
• Create an action to update a file (/etc/motd), based upon a template
• The file will be owned by root, the group root, and have rw-r--r-- permissions

This is a very basic class, but is very common as a lot of what we will do is based upon templates.

Template

The template itself looks like the below: -

Configured by Puppet
Environment: <%= @environment %>
Hostname: <%= @hostname %>
Hypervisor: <%= @pve_host %>

For those who have used Jinja2, the templating system in Puppet can take a little getting used to. However a lot of the functionality is similar, with the difference mainly being in syntax.

In the above, the <%= $VAR %> syntax is a placeholder for a variable (like {{ $VAR }} in Jinja2). The environment and hostname variables are derived from Puppet facts about the node this runs on, whereas the pve_host variable is derived from our hiera data in the previous section.

Profile

As mentioned, we are using roles and profiles to associate the actions required to nodes. This profile can be found in /etc/puppetlabs/code/environments/production/modules/profile/manifests/mymotd.pp

It is worth noting that I am not taking full advantage of the profiles approach, in that I am using a “default” classifier in the profile: -

class profile::mymotd
{
    case $facts['hostname_role'] {
        default: {
            include motd::env
        }
    }
}

We could derive different roles based upon a fact called hostname_role (which would be a created/custom fact applied to nodes). You could choose to include additional/different classes based upon matching a different hostname_role (e.g. if the hostname_role is monitoring-server, also include additional classes that update the MOTD).

In most cases in this post, we will include a single class, but there is nothing saying this is a strict requirement.

Role

The role that refers to this profile is the base role: -

class role::base
{
    include profile::mymotd
    include profile::packages
    include profile::consul
    include profile::node_exporter
}

This role can be found in /etc/puppetlabs/code/environments/production/modules/profile/role/base.pp.

As you can see, in this base profile, we are including multiple profiles, which is where the power of the Roles and Profiles abstraction method starts to come to fruition.

Now all nodes that match our base role will include the mymotd profile, packages, profile, consul profile and node_exporter profile. This is where the consistency of configuration management is achieved, as we can effectively guarantee a minimum set of applications/configuration in place for the entire Puppet-managed estate.

Matching nodes

Now we need to make sure this role is applied to nodes. This can be achieved by adding node declarations in our site.pp file (found in /etc/puppetlabs/code/environments/production/manifests/site.pp): -

node /^prometheus./     
{ 
    include role::prometheus
}

node default
{
    include role::base
}

As we can see, we have a default node selector, as well a selector that matches the prometheus role. Any node that has a hostname that doesn’t begin with prometheus. will be matched by the default role.

Puppet Module: Installing packages

The next module is for installing packages. In this, we are just installing the qemu-guest-agent, but it could be any package(s): -

class packages::qemu {
	package{'qemu-guest-agent': 
        ensure => 'installed'
    }
}

There are no templates being referenced, and we are using the distribution packages rather than adding any additional repositories.

Our profile is almost identical to the previous profile: -

class profile::packages
{
    case $facts['hostname_role'] {
        default: {
            include packages::qemu
        }
    }
}

As we saw in the previous module, we are referencing the profile from the role::base class, meaning all machines referencing this class will have the agent installed.

Puppet Module: Hashicorp Repository

The next module is one that is not referenced by a role or profile. Instead it is used by other modules. This module adds the Hashicorp Debian or RPM repositories so that we can install the latest versions of Hashicorp’s packages (e.g. Consul), rather than relying on out-of-date distribution packages.

The contents of the manifest are: -

class hashicorp::repository {
	$distro = $facts['os']['distro']['codename']
	if $facts['os']['family'] == 'Debian' {
		apt::source { 'hashicorp':
			comment  => 'Hashicorp Repository',
			location => 'https://apt.releases.hashicorp.com',
			release  => "${distro}",
			repos    => 'main',
			key      => {
				'id'=>'E8A032E094D8EB4EA189D270DA418C88A3219F7B',
				'server' => 'keyserver.ubuntu.com',
			},
			include  => {
				'deb' => true,
			},
		}
	}
	if $facts['os']['family'] ==  'RedHat' {
		yumrepo { 'hashicorp':
			enabled  => 1,
			descr    => 'Hashicorp Repository',
			baseurl  => 'https://rpm.releases.hashicorp.com/RHEL/$releasever/$basearch/stable',
			gpgkey   => 'https://rpm.releases.hashicorp.com/gpg',
			gpgcheck => 1,
			target    => '/etc/yum.repo.d/hashicorp.repo',
  		}
	}
}

In this, we are referencing facts (i.e. information about or applied to the Agent(s)) to get the codename (e.g. buster for Debian 10, focal for Ubuntu 20.04 LTS). We also run a conditional against the operating system family, which for us is either Debian-based or Red Hat-based.

If the host is Debian-based, add the Apt repository using the codename. If the host is Red Hat-based, then add the Hashicorp Yum repository. The codename is not required for the Yum repository, as this information is derived by Yum/DNF when installing packages.

I have separated this out because it is not Consul specific, and instead could be used to install Hashicorp’s Terraform, Packer or other tools. However it is only required if we want any of Hashicorp’s tools, so referring to it from the Consul module (and potentially Terraform or Packer modules) makes a lot of sense.

Puppet Module: Installing and running Consul

The Consul module has multiple manifests, allowing us to separate out the logical steps taken. This makes it easier to understand what the module is doing. The structure of the module is as follows: -

/etc/puppetlabs/code$ tree custom/consul/
custom/consul/
├── manifests
│   ├── config.pp
│   ├── firewall.pp
│   ├── init.pp
│   ├── install.pp
│   └── service.pp
└── templates
    ├── consul.client.hcl.erb
    ├── consul.prom_service.hcl.erb
    └── consul.server.hcl.erb

We use an init.pp manifest that contains other classes: -

class consul {
    contain hashicorp::repository
    contain consul::install
    contain consul::config
    contain consul::firewall
    contain consul::service

	Class['hashicorp::repository']
	~> Class['consul::install']
	~> Class['consul::config']
	~> Class['consul::firewall']
	~> Class['consul::service']
}

What we see here is that not only are we referencing all of our Consul manifests, but we also reference the Hashicorp module (specifically the hashicorp::repository class) so that we can install Consul directly from Hashicorp.

In our profile, rather than referencing each class in this module, we can just reference class consul to include all of the manifests.

Also, we can see that there are a number of Class['$CLASS_NAME'] definitions. This is to guarantee the order in which each Class is executed. The reason behind this is that task execution in Puppet is ran in parallel rather than serial (i.e. one after the other). We wouldn’t be able to start the Consul service until Consul is actually installed, so we need to ensure the step that controls the service does not trigger too early.

In my experience so far, the RHEL-based distributions installed Consul in a short enough time that the task execution still succeeded. Debian-based distributions however hadn’t finished installing Consul by the time we want to configure Consul and start the service.

Installing Consul

Manifest

The install.pp manifest looks like the below: -

class consul::install {
	group { 'consul':
		name => 'consul',
	}

	user { 'consul':
		name   => 'consul',
		groups => 'consul',
		shell  => '/bin/false',
	}

	if $facts['os']['family'] == 'Debian' {
		package{'consul':
			ensure => 'latest',
			require => Class['apt::update']
		}
	}

	if $facts['os']['family'] == 'RedHat' {
		package{'consul':
			ensure => 'latest',
		}
	}

	file { '/var/lib/consul':
		ensure  => 'directory',
		owner   => 'consul',
		group   => 'consul'
	}
}

To explain the steps here, we are: -

• Creating a consul user and group
• Installing the latest consul package, as well as triggering an update of apt if running on a Debian-based system
  • Without the update, Debian-based distributions will not use the repository, instead using whatever is in their package archive (usually a very out of date version)
• Creating the /var/lib/consul directory for the consul data, owned by the consul user and group

At this point Consul is installed, but is not running or configured.

Configuring Consul

Manifest

The next manifest is the config.pp manifest: -

class consul::config {
	$consul_dc = lookup('consul_vars.dc')
	$consul_enc = lookup('consul_vars.enckey')
	$servers = lookup('consul_vars.servers')
	$is_server = lookup('consul.server')

	if $is_server {
		file { '/etc/consul.d/consul.hcl':
			content => template('consul/consul.server.hcl.erb'),
			owner   => 'root',
			group   => 'root',
			mode	=> '644',
		}
	} else {
		file { '/etc/consul.d/consul.hcl':
			content => template('consul/consul.client.hcl.erb'),
			owner   => 'root',
			group   => 'root',
			mode	=> '644',
		}
	}
}

In the above we are: -

• Using Hiera lookups to retrieve our Consul Datacentre (consul_vars.dc), encryption key (consul_vars.enc), the list of Consul servers (consul_vars.servers) and whether the host is a Consul server itself (consul.server)
• If the host has consul.server set to true, we use our consul.server.hcl.erb template
• If the host has consul.server set to false, we use our consul.client.hcl.erb template

Hiera Data

The Hiera data being referenced here can be found in code/environments/production/data. We have a common.yaml file which applies to all nodes in the production environment, and a node-specific nodes/puppetserver.meshuggah.yetiops.lab.yaml that overrides the common.yaml file for that node only: -

common.yaml

consul:
    server: false
consul_vars:
    dc: meshuggah
    servers:
    - puppetserver.meshuggah.yetiops.lab
    enckey: "fUPA97Ga7IDLl/HST3H3xw=="

nodes/puppetserver.meshuggah.yetiops.lab.yaml

consul:
    server: true

What this means is that almost all Consul values are the same. The only exception is that the puppetserver will be a Consul server as well, rather than just a client.

Templates

The templates that we reference are for the server and the client: -

consul.server.hcl.erb

{
<%- @networking['interfaces'].each do |interface| -%>
  <%- if interface[1]['ip'] =~ /^10.15.40/ -%>
  "advertise_addr": "<%= interface[1]['ip'] %>",
  "bind_addr": "<%= interface[1]['ip'] %>",
  <%- end -%>
<%- end -%>
  "bootstrap_expect": 1,
  "client_addr": "0.0.0.0",
  "data_dir": "/var/lib/consul",
  "datacenter": "<%= @consul_dc %>",
  "encrypt": "<%= @consul_enc %>",
  "node_name": "<%= @hostname %>",
  "retry_join": [
  <% @servers.each do |server| -%>
    "<%= server %>",
  <%- end -%>
  ],
  "server": true,
  "autopilot": {
    "cleanup_dead_servers": true,
    "last_contact_threshold": "200ms",
    "max_trailing_logs": 250,
    "server_stabilization_time": "10s",
  },
  "ui": true,

consul.client.hcl.erb

{
<%- @networking['interfaces'].each do |interface| -%>
  <%- if interface[1]['ip'] =~ /^10.15.40/ -%>
  "advertise_addr": "<%= interface[1]['ip'] %>",
  "bind_addr": "<%= interface[1]['ip'] %>",
  <%- end -%>
<%- end -%>
  "data_dir": "/var/lib/consul",
  "datacenter": "<%= @consul_dc %>",
  "encrypt": "<%= @consul_enc %>",
  "node_name": "<%= @hostname %>",
  "retry_join": [
  <% @servers.each do |server| -%>
    "<%= server %>",
  <%- end -%>
  ],
  "server": false,
}

In these templates, we see our first use of loops and conditionals in Puppet/ERB (Extended Ruby Template) syntax. If we look at the server loop, and compare to what we would use in Jinja2, you can start to see some similarities: -

ERB

<% @servers.each do |server| -%>
  "<%= server %>",
<%- end -%>

Jinja2

{% for server in servers -%}
  "{{ server }}",
{%- endfor -%}

There are differences, but not to the point of making it hard to understand. In the ERB example, the variable we will reference is inside pipes (|), and the variable to run the loop against has a .each method attached to it. In Jinja2, the language describes the action (i.e. for var in list_of_vars).

The conditional logic is also very similar: -

ERB

<%- if interface[1]['ip'] =~ /^10.15.40/ -%>
  [...]
<% end %>

Jinja2

{%- if (interface[1]['ip'] | regex_search("^10.15.40")) -%}
  [...]
{% endif %}

If anything, the ERB syntax is much less verbose.

To explain what is happening in the templates themselves: -

• The network interface facts are checked
  • If one of the interfaces has an IP starting with 10.15.40 (my lab range), then use that for the Consul advertise_addr and bind_addr values
  • This ensures Consul is listening on the correct interface
• We define the directory Consul will store it’s data (/var/lib/consul)
• We reference our Consul Datacentre, Encryption Key and the machine’s hostname
• We list all of the Consul Servers so that when a Consul Client or Server starts, it attempts to reach an existing Consul Server first
• In addition, if the machine is running as a Consul Server, the template: -
  • Sets the bootstrap_expect value to 1 (i.e. don’t wait for quorum, a single server is enough)
  • Set server to true
  • Set some timeouts/thresholds for Consul
  • Enable the UI

Configuring the firewall

This task is only for Red Hat-based distributions, as Rocky and Alma Linux both have firewalld enabled by default. This isn’t required for Debian, although in a production scenario it is advisable to run some form of firewall.

Manifests

The firewall.pp manifest looks like the below: -

class consul::firewall {
	if $facts['os']['family'] == 'RedHat' {
		firewalld::custom_service{'consul':
			short       => 'consul',
			description => 'Consul',
			port        => [
			  {
			      'port'     => '8301',
			      'protocol' => 'tcp',
			  },
			  {
			      'port'     => '8500',
			      'protocol' => 'tcp',
			  },
			  {
			      'port'     => '8600',
			      'protocol' => 'tcp',
			  },
			  {
			      'port'     => '8301',
			      'protocol' => 'udp',
			  },
			  {
			      'port'     => '8600',
			      'protocol' => 'udp',
			  },
			]
		}
		firewalld_service { 'Consul':
			ensure  => 'present',
			service => 'consul',
			zone    => 'public',
		}
	}
}

The above creates a firewalld custom service for Consul, and then attaches it to the public zone. This will allow Consul to communicate with other hosts.

As noted, we only apply this if the os.family fact contains RedHat (i.e. the machine is running a Red Hat-based distribution).

Enabling the service

Finally, we need to enable the Consul service. All distributions in this lab run systemd, so we use the same manifest and template for Debian-based and Red Hat-based distributions.

Manifests

The service.pp manifest looks like the below: -

class consul::service {
    service {'consul':
        ensure => 'running',
        enable => true,
        restart => "consul reload",
    }
}

When installed via the Hashicorp repositories, Consul includes a systemd unit file. The only interesting part here is the restart => "consul reload" section. What this says is that rather than restarting the entire service, if another class/manifest triggers a reload/refresh, use the consul reload command rather than restarting the entire service.

In most cases, consul reload is all we need to use. For example, if we add or take away a service (using a Consul HCL or JSON file), a consul reload will discover the changes.. A full restart has the potential to break the Consul cluster, or at least break the connection from Prometheus to Consul for service discovery. Triggering a reload is much safer.

Puppet Module: Install the node_exporter

Like the consul module, the node_exporter module has multiple manifests and templates too. The structure is as follows: -

/etc/puppetlabs/code$ tree custom/node_exporter/
custom/node_exporter/
├── manifests
│   ├── consul.pp
│   ├── firewall.pp
│   ├── init.pp
│   ├── install.pp
│   ├── prereqs.pp
│   └── service.pp
└── templates
    └── systemd.erb

Again, we use the init.pp manifest to include other classes: -

class node_exporter {
	contain node_exporter::prereqs
	contain node_exporter::install
	contain node_exporter::service
	contain node_exporter::firewall
	contain node_exporter::consul
}

Prerequisites

As the latest node_exporter isn’t packaged for Debian or Red Hat directly, we must create the users, groups and required directories before we attempt to install the node_exporter: -

class node_exporter::prereqs {
	group { 'node_exporter':
		name => 'node_exporter',
	}

	user { 'node_exporter':
		name   => 'node_exporter',
		groups => 'node_exporter',
		shell  => '/sbin/nologin',
	}

	file { [ '/opt/node_exporter', '/opt/node_exporter/exporters', '/opt/node_exporter/exporters/dist', '/opt/node_exporter/exporters/dist/textfile' ]:
		ensure => 'directory',
		owner  => 'node_exporter',
		group  => 'node_exporter',
	}
}

The first action is to create the node_exporter group. After this, we create the node_exporter user. Finally, we create the directory structure for the textfile_collector (i.e. where node_exporter will ingest metrics from files in a certain directory).

What you’ll notice here is that we are creating each directory in the tree. This is because Puppet does not have a native way of creating a directory and it’s parent directories if they do not already exist. To achieve this in Saltstack, you would use: -

/opt/exporters/dist/textfile:
  file.directory:
    - user: node_exporter
    - group: node_exporter
    - mode: 755
    - makedirs: True

The makedirs option is what creates the parent directories. In Ansible, we would use something like: -

- name: Create a directory if it does not exist
  ansible.builtin.file:
    path: /opt/exporters/dist/textfile 
    state: directory
    mode: '0755'

Ansible natively creates the parent directories, so nothing changes here.

At this point, we can now install the node_exporter.

Install the node_exporter

As no Debian/RPM package is available of the latest node_exporter version, we retrieve the node_exporter tarball file from GitHub: -

class node_exporter::install {
    $install_path   = "/usr/local/bin"
    $package_org    = "prometheus"
    $package_name   = "node_exporter"
    $package_type   = "linux-amd64"
    $package_ensure = lookup("node_exporter.version")
    $archive_name   = "${package_name}-${package_ensure}.${package_type}.tar.gz"
    $repository_url = "https://github.com/${package_org}/${package_name}/releases/download"
    $package_source = "${repository_url}/v${package_ensure}/${archive_name}"
    
    archive { $archive_name:
        path         => "/tmp/${archive_name}",
        source       => $package_source,
        extract      => true,
        extract_command => 'tar xfz %s --strip-components=1',
        extract_path => $install_path,
        creates      => "${install_path}/${package_name}",
        cleanup      => true,
    }
}

What we can see here is that rather than specifying the full paths and URLs directly in the action, we use variables. This allows us to change the package type (e.g. use a different architecture), the package name, location to download from and more in one place. In the above, the package_name variable is reference three times, so being able to change it once place avoids typos or forgetting to change it everywhere.

The Puppet archive action is able to use a URL as the source directly, rather than needing to download the file with another action first.

One point to note is the extract_command field. This is supplied because otherwise the destination directory in /usr/local/bin would be /usr/local/bin/node_exporter-v1.1.2.linux-amd64/node_exporter. Using the strip-components flag means that the top-level directory inside of the archive is “stripped”, with the contents of said directory being placed directly in /usr/local/bin instead.

The package version itself is sourced from hiera data. In this lab, it looks like this: -

node_exporter:
    version: 1.1.2

Defining and enabling the service

We use systemd to run the node_exporter, which requires us to create a systemd unit file. The manifest for this is below: -

class node_exporter::service {
	file { '/etc/systemd/system/node_exporter.service':
		content => template('node_exporter/systemd.erb'),
		owner   => 'root',
		group   => 'root',
		mode	=> '644',
	}

	service {'node_exporter':
        ensure => 'running',
        enable => true,
	}
}

This is similar to Consul, except we just need to create the unit file first. The referenced template looks like the below: -

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.systemd --collector.wifi --collector.textfile --collector.textfile.directory=/opt/exporters/dist/textfile

[Install]
WantedBy=multi-user.target

There are no template variables in this, but it does give us the option to add them if required in future.

Firewall rules

As with Consul, this is for Red Hat-based distributions only, as they come with firewalld enabled by default: -

class node_exporter::firewall {
	if $facts['os']['family'] == 'RedHat' {
		firewalld::custom_service{'node_exporter':
			short       => 'node_exporter',
			description => 'Prometheus Node Expoter',
			port        => [
			  {
			      'port'     => '9100',
			      'protocol' => 'tcp',
			  }
			]
		}
		firewalld_service { 'Allow Node Exporter':
			ensure  => 'present',
			service => 'node_exporter',
			zone    => 'public',
		}
	}
}

The node_exporter runs on fewer ports than Consul does, so we only need to allow TCP/9100 for Prometheus to be able to scrape metrics from it.

Registering the service with Consul

Now that the node_exporter is running, we need to register a service with Consul. This service tells Consul what port the node_exporter is listening on and associated tags. This means that rather than needing to tell Prometheus what machines are running node_exporter, Prometheus discovers from Consul what each machine is running. Adding additional exporters and services requires no change on the Prometheus side.

The manifest looks like the below: -

class node_exporter::consul {
	$consul_service_name = "node_exporter"
	$consul_service_port = 9100

	file { "/etc/consul.d/${consul_service_name}.hcl":
		content => template('consul/consul.prom_service.hcl.erb'),
		owner   => 'root',
		group   => 'root',
		mode	=> '644',
	}

	exec {'consul_reload':
		command => '/usr/bin/consul reload',
		subscribe => [
			File["/etc/consul.d/${consul_service_name}.hcl"],
		],
		refreshonly => true,
    }
}

In this, we are referencing a template in the Consul module. This is so that rather than needing to have this template exist in every exporter module, we can use a common Consul template instead.

We also have an exec action, which is how arbitrary commands are ran on the machine by Puppet. This has the subscribe option, which says that we will only reload Consul if there is a change in the Consul service file.

The template itself looks like the below: -

{"service":
  {
    "name": "<%= @consul_service_name %>",
    "tags": ["<%= @consul_service_name %>", "prometheus"],
    "port": <%= @consul_service_port %>,
  }
}

As you can see, we are using the Consul service name and port as placeholders in this template. In this case, they are replaced with node_exporter and 9100, producing the below: -

{"service":
  {
    "name": "node_exporter",
    "tags": ["node_exporter", "prometheus"],
    "port": 9100, 
  }
}

After this runs, the Consul client will register this service with the Consul server. This allows the Prometheus server (and anything else that can use Consul as a source) to discover all available services by only needing to contact the Consul server(s).

Puppet Module: Prometheus

This module is only applied to any node with a hostname that starts prometheus, as seen in the role, profile and site.pp manifests below: -

site.pp

node /^prometheus./		{ 
    include role::prometheus
}

Role - prometheus.pp

class role::prometheus inherits role::base
{
    include profile::prometheus
}

Note here that we are inheriting the base role as well, meaning we also include the MOTD, base package, Consul and node_exporter profiles too.

Profile - prometheus.pp

class profile::prometheus
{
    case $facts['hostname_role'] {
        default: {
            include prometheus
        }
    }
}

The structure of the module itself is below: -

$ tree custom/prometheus/
custom/prometheus/
├── manifests
│   ├── config.pp
│   ├── init.pp
│   ├── install.pp
│   ├── prereqs.pp
│   └── service.pp
└── templates
    ├── prometheus.yml.erb
    └── systemd.erb

The structure is very similar to the node_exporter module, except we are not registering any services with Consul.

Again, our init.pp references the other classes and manifests: -

class prometheus {
	contain prometheus::prereqs
	contain prometheus::install
	contain prometheus::config
	contain prometheus::service
}

Prerequisites

The following manifest is very similar to the node_exporter prerequisites manifest: -

class prometheus::prereqs {
	group { 'prometheus':
		name => 'prometheus',
	}

	user { 'prometheus':
		name   => 'prometheus',
		groups => 'prometheus',
		shell  => '/sbin/nologin',
	}

	file { [ '/etc/prometheus', '/etc/prometheus/alerts', '/etc/prometheus/rules', '/var/lib/prometheus' ]:
		ensure => 'directory',
		owner  => 'prometheus',
		group  => 'prometheus',
	}
}

Again, we create our group and user, and also create some directories for use with Prometheus. The first three directories contain Prometheus’s configuration, as well as locations for alerts (for use with Alertmanager) and recording rules. The last directory is where Prometheus stores it’s data.

Install Prometheus

Again, this manifest is very similar to the node_exporter module: -

class prometheus::install {
    $install_path   = "/usr/local/bin"
    $package_name   = "prometheus"
    $package_type   = "linux-amd64"
    $package_ensure = lookup("prometheus.version")
    $archive_name   = "${package_name}-${package_ensure}.${package_type}.tar.gz"
    $repository_url = "https://github.com/${package_name}/${package_name}/releases/download"
    $package_source = "${repository_url}/v${package_ensure}/${archive_name}"

	archive { $archive_name:
        path         => "/tmp/${archive_name}",
        source       => $package_source,
        extract      => true,
        extract_command => 'tar xfz %s --strip-components=1',
        extract_path => $install_path,
        creates      => "${install_path}/${package_name}",
        cleanup      => true,
	}
}

The only difference here is that the package_name and package_org would be the same, so I have eschewed it in favour of using the same variable twice. If in future the organisation changes, then it is quite straightforward to add package_org variable in the repository_url instead.

Prometheus Configuration

Unlike the node_exporter, Prometheus requires a configuration file to run, so we use the following manifest to create one: -

class prometheus::config {
	$consul_servers = lookup('consul_vars.servers')
	file { '/etc/prometheus/prometheus.yml':
		notify  => Service['prometheus'],
		content => template('prometheus/prometheus.yml.erb'),
		owner   => 'prometheus',
		group   => 'prometheus',
		mode	=> '644',
	}
}

In this, we also reference the Consul servers, because we will be using them for service discovery. We also use the notify property to tell Prometheus to reload if the configuration changes.

The configuration template looks like the below: -

global:
  scrape_interval:     15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093

rule_files:
  - 'alerts/*.yml'
  - 'rules/*.yml'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets:
      - 'localhost:9090'
  - job_name: 'consul'
    consul_sd_configs:
<% @consul_servers.each do |server| -%>
      - server: '<%= server %>:8500'
<%- end -%>
    relabel_configs:
      - action: labelmap
        regex: __meta_consul_metadata_(.+)
      - source_labels: [__meta_consul_tags]
        regex: .*,prometheus,.*
        action: keep
      - source_labels: [__meta_consul_service]
        target_label: job

The two jobs are for Prometheus itself, and consul_sd_configs, which is the job used to discover services from Consul.

Defining and enabling the service

As with the node_exporter, we need to define the systemd service and run it, as per below: -

class prometheus::service {
	file { '/etc/systemd/system/prometheus.service':
		content => template('prometheus/systemd.erb'),
		owner   => 'root',
		group   => 'root',
		mode	=> '644',
	}

	service {'prometheus':
		ensure => 'running',
                enable => true,
	}
}

This references the below template: -

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.enable-lifecycle \
    --web.external-url=http://prometheus.noisepalace.home \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

As with the node_exporter, we aren’t supplying any template variables, but we have the option to if required in future.

Running Puppet

At this point, the only action we need to take on each machine running the Puppet agent is: -

$ sudo puppet agent -t

This will tell Puppet to contact the Puppet server, and run any modules that apply to it. To summarise, for our machines, this is what would apply: -

By the end of the run on all of these, you would have: -

• 8 servers running Consul client and the Prometheus node_exporter
• The Puppet server running as a Consul server as well
• The Prometheus server also running Prometheus

To show some of this in action, I have put together a video demonstration of running Puppet on all of the “util” servers: -

This doesn’t show Prometheus server or Puppet server being configured, but the steps to allow Puppet to manage them are identical to what is shown in the video.

Comparisons with Ansible and Saltstack

As noted, I have used Saltstack and Ansible quite extensively in the past, whereas using Puppet is still relatively new for me. However already I have noticed some interesting differences (as well as similarities) with Ansible and Saltstack.

Ansible

One of the reasons Ansible was created was so that configuration management could be achieved without needing an agent. Puppet, by contrast, relies heavily on the use of agents. This has positives and negatives.

The setup process to get started with Ansible is minimal, in that you install Ansible (from package managers or pip) and then start managing machines straight away. If you have SSH, WinRM or other required access to machines (even API access is enough for some tasks), you can run your Playbooks against them. With Puppet, you need to setup the server, ensure working DNS (otherwise you might encounter issues), install agents on every machine, and then start applying your modules and manifests.

However when it comes to the speed of tasks/events, Puppet is almost always quicker to apply changes in my experience. As you need to run a Puppet server, this will often be placed much closer to your infrastructure than where you are. This means that latency and bandwidth of your connection becomes much less of an issue.

There are caveats to both of the above points, in that there is nothing stopping you from running Ansible playbooks from a server(s) much closer to your infrastructure (or using something like Ansible Tower/AWX), and also Puppet does have an agentless mode (called Bolt), but in “traditional” usage of both tools, the above still applies.

The syntax differences are massive, in that Ansible aims to be very human-readable, relying on a YAML-based configuration. Puppet by contrast has some idiosyncrasies to the syntax and approach that do take a while to understand in my experience. As mentioned, if you come from a Ruby background, Puppet will feel quite natural.

Finally, the Ansible ecosystem around managing other systems like BSD (OpenBSD, FreeBSD etc), networking equipment and more is very mature and well supported (the lack of agents being an advantage here). By contrast, Puppet do not package agents directly for the BSDs, and the requirement for running agents does make managing networking hardware much more difficult.

One major advantage of Puppet is it’s longevity. There are articles from the early 2000s on some Puppet concepts which are still relevant. Also, as a symptom of nearly two decades of Puppet usage, the reliability of the product is stellar in my experience. The only time I have found Puppet to have issues is due to outside factors. Good examples would be non-synchronised clock sources (meaning certificate validation fails), failing DNS resolution or the hardware is simply not powerful enough to run it. From an ongoing usage and maintenance perspective, I have yet to see any issues with Puppet.

Saltstack

As Saltstack uses agents in a similar way to Puppet, there are fewer major differences between it and Puppet.

One of the biggest differences I have noticed is that there is no requirement for Puppet agents to maintain ongoing communication with the Puppet server. This has the upside in that hosts which are not always available are well supported. Saltstack expects hosts to always be available, meaning that for ad-hoc/hosts intermittently online it will timeout when applying configuration. This can make applying configuration changes quite slow if some of the hosts are timing out, or at least throw a lot of errors when running tasks from the Saltstack Master.

However this also means that because the bulk of the communication in Puppet is controlled by the agent, as far as I am aware there is no mechanism to run ad-hoc commands against all managed hosts, or retrieve data from all managed hosts in real time. The closest you can get with Puppet is viewing facts/catalogs from the last time Puppet made contact with the server. By contrast, running a command against every single Salt-managed machine is as simple as salt '*' cmd.run 'sudo iptables -vnL', and you can get node-specific facts using salt '*' grains.items.

As with Ansible, the syntax differences between Saltstack and Puppet are quite large. Saltstack (like Ansible) uses YAML-based configuration, but with the added ability to dynamically generate tasks with Jinja2, rather than just using Jinja2 inside of templates. For example, you can build a task in Saltstack like the below: -

{% if pillar['consul'] is defined %}
{% if pillar['consul']['prometheus_services'] is defined %}
{% for service in pillar['consul']['prometheus_services'] %}
/etc/consul.d/{{ service }}.hcl:
  file.managed:
    - source: salt://consul/services/files/{{ service }}.hcl
    - user: consul
    - group: consul
    - mode: 0640
    - template: jinja

consul_reload_{{ service }}:
  cmd.run:
    - name: consul reload
    - watch:
      - file: /etc/consul.d/{{ service }}.hcl
{% endfor %}
{% endif %}
{% endif %}

This means that we have a dynamically-generated task based upon pillar data (pillar data is very similar to Hiera, for those more familiar with Puppet), meaning that there is less repetition in your Saltstack state files.

As far as I am aware, this is not possible with Puppet. However it is also not possible with Ansible either, so this is more of a general Saltstack advantage rather than specifically a difference to Puppet.

Again, like Puppet, Saltstack can be ran without an agent. However the “traditional” installation approach tends to use a central Salt server (called a master) with agents (minions) registered with it.

The support across operating systems with Saltstack is also quite good, and it has also become increasingly popular in the networking community as well (see Mircea Ulnic’s blog for more on this).

Summary

Puppet is a great configuration management tool, and given it has been around for nearly two decades as well, it is also incredibly mature at this stage too.

If you are in the process of choosing a configuration management tool, Puppet is definitely worth your consideration. While the syntax is very different from tools like Ansible and Saltstack, this doesn’t necessarily make it more difficult, just different.

In future posts, I will show how to integrate Puppet with Cloud-Init for managing cloud-based infrastructure, as well as managing operating systems that are not Linux too!

For all the Puppet code in this blog, see my Puppet Lab repository.

Around 18 months ago, I put together a post on providing anycast services with BGP. It covered what anycast is, how to provide anycast-based services with BGP, and also how to conditionally advertise your anycast IPs using ExaBGP.

This post will take it a step further, by using Pulumi to provision servers and BGP sessions on Equinix Metal, and Saltstack that will configure the servers to advertise routes via BGP to Equinix.

Equinix Metal

As mentioned in my previous post using Pulumi, Equinix Metal are a provider of bare metal servers and infrastructure. For those who need dedicated machines with no contention over CPU/memory/storage on the server, they are able to provision them with the same tooling that others would use to create a VPS/instance on other cloud providers (e.g. AWS, Digital Ocean, Linode).

I recently guested on a stream with David McKay/Rawkode, a developer advocate at Equinix Metal going through some of the networking options available when using their infrastructure. While we didn’t get into configuring BGP, you can at least get an idea of what Equinix Metal is like as a cloud/infrastructure provider: -

A big point to take away from this is that we are able to configure BGP within Equinix Metal, and advertise routes from the servers we create. This option isn’t always available in other cloud/infrastructure providers, or at minimum requiring your own IP space.

Anycast with ExaBGP

As mentioned in the introduction, I have covered using ExaBGP for Anycast before. The aim of this post is very similar to the previous Anycast post, except that there will be no user interaction involved (other than typing pulumi up).

We won’t go into great depth about how ExaBGP works, as this is already covered here.

Saltstack

In this post we will cover the Saltstack states and pillars to configure our servers, and the use of grains to retrieve information from the servers that we provision.

For the impatient among you (I’m one of them!), you can view all of these in this repository.

As a brief refresher, or for those who are used to other configuration management systems, the following terms are used in Saltstack: -

• states - They contain the tasks that need to run. Similar to Ansible Playbooks, Puppet Manifests or Chef Cookbooks
• pillars - These are variables which can be used in our states and templates. These are similar to Ansible’s host variables
• grains - These are per-node facts, that are derived from the hosts themselves (e.g. host operating system, IP addresses etc)
  • You can also set custom grains which will can then be referenced in your state or pillars
• minions - Hosts running the Salt Minion agent, that register with the master

States

In our states directory, we have the following top file (i.e. the file which determines what states apply to what nodes): -

base:
  'G@os:Ubuntu or G@os:Debian':
    - match: compound
    - base

  'bgp-*':
    - pip
    - network
    - exabgp
    - nginx

The first section installs some base packages on any minion running Ubuntu or Debian. None of the packages are relevant to the anycast deployment (other than tcpdump, if you want to run packet captures to prove where traffic is being forwarded).

The second section matches nodes with a nodename (often the same as the hostname, but not a requirement) that starts with bgp. If a node is called bgp-paris or bgp-01 or bgp-meshuggah, they would all be matched by this. However a node called more-bgp-01 wouldn’t match.

The pip state

python3-pip:
 pkg.installed

As you can see, this just installs the python3-pip package. Without this, we cannot install ExaBGP (as it is packaged as a Python PIP module).

The network state

{% for interface in pillar['network']['interfaces'] %}
{{ interface['name'] }}:
  network.managed:
    - enabled: True
    {%- if interface['type'] is defined %}
    - type: {{ interface['type'] }}
    {%- endif %}
    {%- if interface['proto'] is defined %}
    - proto: {{ interface['proto'] }}
    {%- endif %}
    - ipaddr: {{ interface['ipv4']['address'] }}
    - netmask: {{ interface['ipv4']['mask'] }}
    {%- if interface['ipv4']['gateway'] is defined %}
    - gateway: {{ interface['ipv4']['gateway'] }}
    {%- endif %}
    {%- if interface['ipv4']['nameservers'] is defined %}
    - dns:
      {% for ns in interface['ipv4']['nameservers'] %}
      - {{ ns }}
      {% endfor %}
    {%- endif %}
{%- if interface['routes'] is defined %}
routes_{{ interface['name'] }}:
  network.routes:
    - name: {{ interface['name'] }}
    - routes:
       {%- for route in interface['routes'] %}
       - name: {{ route['name'] }}
         ipaddr: {{ route['prefix'] }}
         netmask: {{ route['mask'] }}
         gateway: {{ route['gateway'] }}
       {%- endfor %}
{%- endif %}
{% endfor %} 

While this initially looks quite complex, it is mostly if statements on whether to supply a value for each field in the the Saltstack network.managed module to create network interfaces. Not every field needs to be defined, or should (e.g. a loopback interface doesn’t need a default gateway or a nameserver).

We can add static routes as well, if we want traffic to take a different path than the default gateway.

The exabgp state

exabgp:
  pip.installed

/etc/exabgp:
  file.directory:
    - user: root
    - group: root
    - mode: 755
    - makedirs: True

/etc/exabgp/scripts:
  file.directory:
    - user: root
    - group: root
    - mode: 755
    - makedirs: True

{% if not salt['file.is_fifo']('/var/run/exabgp.in') %}
exabgp_in_fifo:
  cmd.run:
    - name: mkfifo /var/run/exabgp.in
{% endif %}

{% if not salt['file.is_fifo']('/var/run/exabgp.out') %}
exabgp_out_fifo:
  cmd.run:
    - name: mkfifo /var/run/exabgp.out
{% endif %}

/etc/systemd/system/exabgp.service:
  file.managed:
    - source: salt://exabgp/files/exabgp.service.j2
    - user: root
    - group: root
    - mode: 0644
    - template: jinja

/etc/exabgp/exabgp.conf:
  file.managed:
    - source: salt://exabgp/files/exabgp.conf.j2
    - user: root
    - group: root
    - mode: 0644
    - template: jinja

exabgp_scripts:
  file.recurse:
    - name: /etc/exabgp/scripts
    - source: salt://exabgp/scripts
    - user: root
    - group: root
    - file_mode: 0755
    - template: jinja

exabgp_systemd_reload:
  cmd.run:
    - name: systemctl daemon-reload
    - onchanges:
      - file: /etc/systemd/system/exabgp.service

exabgp.service:
  service.running:
  - enable: True
  - reload: True
  - restart: True
  - watch:
      - file: /etc/exabgp/exabgp.conf
      - file: /etc/systemd/system/exabgp.service
      - file: exabgp_scripts

There are a few parts to this: -

• We install ExaBGP using Python’s PIP
• We create the /etc/exabgp and /etc/exabgp/scripts directory
  • The base directory contains global ExaBGP configuration
  • The scripts directory contains check scripts that test if applications are functioning correctly
• We create named pipes using mkfifo that will be used by ExaBGP itself for communication
• We create a SystemD unit file to manage the startup/shutdown of ExaBGP
• We create the exabgp.conf file with all of the necessary ExaBGP configuration (including BGP peers)
• We make sure to run a daemon-reload on the ExaBGP service if the ExaBGP service file changes (ensuring we are running on the most recent version of the service)
• Finally, we make sure the service is enabled, reloaded and restarted if any of the ExaBGP files change (i.e. service, configuration or scripts)

Unfortunately we must perform a restart on ExaBGP if the scripts or configuration changes, as it doesn’t pick up the changes automatically.

The SystemD unit file is found in files/exabgp.service.j2, and looks like the below: -

[Unit]
Description=ExaBGP
After=network.target
ConditionPathExists=/etc/exabgp/exabgp.conf

[Service]
Environment=exabgp_daemon_daemonize=false
Environment=ETC=/etc
ExecStart=/usr/local/bin/exabgp /etc/exabgp/exabgp.conf
ExecReload=/bin/kill -USR1 $MAINPID

[Install]
WantedBy=multi-user.target

The files/exabgp.conf.j2 file looks like the below: -

process announce-routes {
  run /etc/exabgp/scripts/{{ pillar['exabgp']['check_script']['name'] }};
  encoder {{ pillar['exabgp']['check_script']['encoder'] }};
}

{% for peer in pillar['exabgp']['peers'] %}
neighbor {{ peer['ip'] }} {
    local-address {{ grains['bgp']['localip'] }};
    local-as {{ pillar['exabgp']['asn'] }};
    peer-as {{ peer['asn'] }}; 

    api {
        processes [ announce-routes ];
   }

}
{% endfor %}

In this, we define the process (announce-routes) and which script controls the route announcements and withdrawals. We also specify the encoder. This is either text or json, depending on the format of the message returned by the script.

The last section creates the BGP peering sessions. It loops over a list of peers in the pillars for the node, creating a neighbor statement with the local autonomous system number, Equinix’s autonomous system number, and also the local address. This is sourced from the node’s grains, which will covered in more detail in the Pulumi section.

We also have our scripts, which look like the below: -

scripts/nginx-check.sh

#!/bin/bash

while true; do
  curl -s localhost:80 > /dev/null;
  if [[ $? != 0 ]]; then
          {%- for ip in grains['anycast']['ipv4'] %}
          echo "withdraw route {{ ip['address'] }} next-hop {{ grains['bgp']['localip'] }}\n"
          {%- endfor %}
  else
          {%- for ip in grains['anycast']['ipv4'] %}
          echo "announce route {{ ip['address'] }} next-hop {{ grains['bgp']['localip'] }}\n"
          {%- endfor %}
  fi
  sleep 5
done

scripts/dns-check.sh

#!/bin/bash

while true; do
  /usr/bin/dig yetiops.net @127.0.0.1 > /dev/null;
  if [[ $? != 0 ]]; then
	  {%- for ip in grains['anycast']['ipv4'] %}
          echo "withdraw route {{ ip['address'] }} next-hop {{ grains['bgp']['localip'] }}\n"
	  {%- endfor %}
  else
	  {%- for ip in grains['anycast']['ipv4'] %}
          echo "announce route {{ ip['address'] }} next-hop {{ grains['bgp']['localip'] }}\n"
	  {%- endfor %}
  fi
  sleep 5
done

Both are very similar, in that they check the status of a command (either doing a cURL request against the local webserver, or a dig request against the local DNS server). The commands run inside of a while loop, making sure they are continuously evaluated. On every iteration, we check to see if the exit code of the command is not 0 (i.e. did the command fail).

If the command succeeded (e.g. the cURL request returned a page, or the DNS request returned an A record) then we return the text announce route $ROUTE next-hop $LOCAL-IP for each route we want to announce. If the command fails (e.g. the cURL request was refused, or the DNS request returned no result) then we return the text withdraw route $ROUTE next-hop $LOCAL-IP for each route to want to stop announcing.

Finally, we sleep for 5 seconds between each check, otherwise we may risk overloading ExaBGP with route announcements/withdrawals.

The key point here is that we are using grains rather than pillars. We will see in the Pulumi section why this is relevant, but in summary this is so that the code is reusable and not tied to specific IPs, and doesn’t require updating Salt pillars manually after provisioning our machines.

The nginx state

Finally, we have the nginx state: -

nginx:
  pkg.installed

/etc/nginx/nginx.conf:
  file.managed:
    - source: salt://nginx/files/nginx.conf.j2
    - user: root
    - group: root
    - mode: '0644'
    - template: jinja

/etc/nginx/sites-available/default:
  file.managed:
    - source: salt://nginx/files/default.j2
    - user: root
    - group: root
    - mode: '0644'
    - template: jinja

nginx.service:
  service.running:
    - watch:
      - file: /etc/nginx/nginx.conf
      - file: /etc/nginx/sites-available/default

/var/www/html/index.nginx-debian.html:
  file.absent

/var/www/html/index.html:
  file.managed:
    - source: salt://nginx/files/index.html.j2
    - user: root
    - group: root
    - mode: '0644'
    - template: jinja

In this state, we do the following: -

• We install nginx
• We update the nginx.conf configuration file using a template
• We update the default server configuration using a template
• We make sure nginx is running
  • We also watch for changes in the configuration file or default server file, reloading if they do change
• We remove the default Debian nginx index file
• We place our own index.html file using a template

The files/nginx.conf.j2 file looks like the below: -

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
	# multi_accept on;
}

http {

	##
	# Basic Settings
	##

	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;
	# server_tokens off;

	# server_names_hash_bucket_size 64;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}

The files/default.j2 file looks like the below: -

server {
        listen 127.0.0.1:80;
        {%- for ip in grains['anycast']['ipv4'] %}
	listen {{ ip['address'] }}:80;
        {%- endfor %}
	listen [::]:80 default_server;

	root /var/www/html;

	server_name _;

	location / {
		# First attempt to serve request as file, then
		# as directory, then fall back to displaying a 404.
		try_files $uri $uri/ =404;
	}
}

In this, we ensure that the default site listens on 127.0.0.1:80 (localhost, for our ExaBGP check script) and also on our anycast IPs.

Last, we have our files/index.html.j2 file: -

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
	<h1>Welcome to nginx on {{ grains['nodename'] }}!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working serving from {{ grains['nodename'] }}. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

This is the standard nginx index page, but with our nodename displayed. This allows us to see which node our traffic to the anycast IP is forwarded to.

Pillars

Now that we have our states defined, we can cover our pillars.

The top.sls file for our pillars looks like the below: -

base:
  'bgp-*':
    - exabgp.bgp
    - bind.bgp
    - network.bgp

While we aren’t using bind in this post, the states and pillars exist for it in our repository so that you can change between serving nginx and running anycast DNS.

The exabgp pillar

In the exabgp/bgp.sls file, we have the following: -

exabgp:
  asn: 65000
  check_script:
    name: "nginx-check.sh"
    encoder: text
  peers:
    - ip: 169.254.255.1 
      asn: 65530 
    - ip: 169.254.255.2 
      asn: 65530 

Why are we hardcoding the peer values? This is because when you use Equinix Metal, you always peer with the same IPs no matter where you provision your servers. Therefore we do not need to derive what IPs we will peer with at runtime, as it is always the same.

Also, when using the Local BGP option with Equinix Metal, you always use the autonomous system number 65000 for your BGP process, and the Equinix Metal BGP sessions will always use the autonomous system number 65530.

This makes it quite straightforward in terms of what values we need to derive when creating our BGP sessions, as we already know most of them before they are even provisioned.

We also define our check_script, in this case the nginx-check.sh script, as we are using nginx to prove the anycast functionality.

The network pillar

In the network/bgp.sls file, we have the following: -

network:
  interfaces:
{% for ip in grains['anycast']['ipv4'] %}
    - name: "lo:{{ loop.index }}"
      type: eth
      proto: static
      ipv4:
        address: {{ ip['address'] }}
        mask: {{ ip['mask'] }}
{% endfor %}

This is used to discover values from the node grains. We could also do directly in the state files (i.e. referring to the grains in templates/state files). However using the above allows us to set additional interfaces not derived from grains, and the states don’t need to care whether the values are originally grains or manually set in pillars.

Pulumi

Now that we have our Salt states and pillars defined, we can create the following: -

• Two servers that will serve nginx, and advertise anycast IPs to Equinix
• A single Salt Master for the above servers to register with
  • The Salt Master will then configure the other two servers with ExaBGP and nginx

If you want to see the Pulumi code first, go here.

If you haven’t used Pulumi before, or Equinix Metal, then see my Pulumi Introduction post and the Equinix Metal section of my Pulumi with other providers post.

Steps required

As a brief summary, we need to do the following with our Pulumi code: -

• Create an Equinix Metal project, and enable BGP on the project
• Add an SSH key to the project
• Create a server running as a Salt Master, using cloud-init for first-time boot configuration
• Reserve a single Equinix-owned IPv4 address that we can use as our anycast IP
  • I don’t own any IPv4 or IPv6 blocks, hence the need to use Equinix’s IP space
• Create two servers that will serve nginx, and advertise our anycast IPv4 address back to Equinix (so they know which server(s) to send it to)
• Create Equinix IPv4 BGP sessions to each server, otherwise we have nothing to peer with!

Equinix Metal doesn’t provide firewalling or anything similar, meaning that if you want to secure your systems you can either: -

• Run host-based firewalling
• Provision Equinix Metal machines that serve as the security boundary to your other machines (effectively acting as a firewall(s) for the other servers)

As we have already covered the basics of Pulumi and using it with Equinix Metal in previous posts, I won’t cover how to set up a stack, a Pulumi project or how to generate credentials to use with Equinix Metal.

The Equinix Metal project

As I have been doing a lot with Equinix Metal recently, it makes sense to have an overall/“root” project, rather than projects that are created and deleted every time I run Pulumi or destroy resources in Pulumi.

To do this, we use something like the below: -

package main

import (
	metal "github.com/pulumi/pulumi-equinix-metal/sdk/v2/go/equinix"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		// Create an Equinix Metal resource (Project)
		project, err := metal.NewProject(ctx, "pulumi-yetiops", &metal.ProjectArgs{
			Name: pulumi.String("pulumi-yetiops"),
			BgpConfig: metal.ProjectBgpConfigArgs{
				Asn:            pulumi.Int(65000),
				DeploymentType: pulumi.String("local"),
			},
		})
		if err != nil {
			return err
		}

		// Export the name of the project
		ctx.Export("projectName", project.Name)
		ctx.Export("projectId", project.ID())
		return nil
	})
}

The above creates a project called pulumi-yetiops and enables BGP, using an autonomous system number of 65000 and a BGP type of local. Local means we are going to use Equinix’s own IP addresses to advertise back to them, rather than any we may own.

An important point to note is that this project is created in it’s own Pulumi stack, which we can then reference in other stacks. If you are familiar with Terraform, this is similar to referencing remote states.

Referencing the project

After creating a stack specific to the anycast BGP deployment, we reference the project we created above: -

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		conf := config.New(ctx, "")
		commonName := conf.Require("common_name")

		rootStack, err := pulumi.NewStackReference(ctx, "yetiops/equinix-metal-yetiops/staging", nil)

		if err != nil {
			return err
		}

As in the previous posts, we have our commonName variable that we can use across our resources. We can now focus on the pulumi.NewStackReference function.

In this, we pass in the context (standard for Pulumi with Go). We then reference the stack name for our overall Equinix Metal project. From herein, we can now refer to the Export values/Outputs from the root project. An example of this is below: -

projectId := rootStack.GetStringOutput(pulumi.String("projectId"))

In the above, we get a string output of the root project’s projectId value (which we created using ctx.Export("projectId", project.ID()) in the root project). Now if the root project ID changes, this stack will be aware of the change.

Supplying a Project SSH key

Next, we add an SSH key to our project that will allow us to login to the servers (and also their serial consoles): -

		user, err := user.Current()

		if err != nil {
			return err
		}

		sshkey_path := fmt.Sprintf("%v/.ssh/id_rsa.pub", user.HomeDir)

		sshkey_file, err := ioutil.ReadFile(sshkey_path)
		if err != nil {
			return err
		}

		sshkey_contents := string(sshkey_file)

		sshkey, err := metal.NewProjectSshKey(ctx, commonName, &metal.ProjectSshKeyArgs{
			Name:      pulumi.String(commonName),
			PublicKey: pulumi.String(sshkey_contents),
			ProjectId: projectId,
		})

As in the previous Pulumi posts, this gets the home directory of the current user running Pulumi, sources their RSA-based SSH key, and then adds it to the project.

Using cloud-init

In previous posts on Pulumi, I provided UserData (i.e. first-boot configuration) using either strings containing a small amount of cloud-config YAML or referenced a YAML file with the required cloud-config. However this can be a bit limiting in terms of how we provide variables, templated values and conditional logic.

Instead I am going to use the Juju Cloud-Init module so that generating the UserData is more flexible, and keeps our Pulumi stack entirely in Go.

The inspiration for this is the Tinkerbell Testing Infrastructure code created by David McKay/Rawkode (who also happens to be a fan of Saltstack and Pulumi!).

The salt-master

The cloud-config function for the salt-master is defined as follows: -

func saltMasterCloudInitConfig() string {
	c, err := cloudinit.New("focal")

	if err != nil {
		panic(err)
	}

	c.AddRunCmd("curl -fsSL https://bootstrap.saltproject.io -o install_salt.sh")
	c.AddRunCmd("sh install_salt.sh -P -M -x python3")

	c.AddRunTextFile("/etc/salt/master.d/master.conf", `autosign_grains_dir: /etc/salt/autosign-grains
fileserver_backend:
  - roots
  - gitfs

gitfs_remotes:
  - https://gitlab.com/stuh84/salt-anycast-equinix:
    - root: states
    - base: main
    - update_interval: 120


pillar_roots:
  base:
    - /srv/salt/pillars

ext_pillar:
  - git:
    - main https://gitlab.com/stuh84/salt-anycast-equinix:
      - root: pillars
      - env: base`, 0644)

	c.AddRunTextFile("/etc/salt/minion.d/minion.conf", `autosign_grains:
- role
startup_states: highstate
grains:
  role: master`, 0644)

	c.AddRunCmd("mkdir -p /etc/salt/autosign-grains")

	c.AddRunCmd("echo -e \"master\nbgp\n\" > /etc/salt/autosign-grains/role")

	c.AddRunCmd("PRIVATE_IP=$(curl -s https://metadata.platformequinix.com/metadata | jq -r '.network.addresses | map(select(.public==false)) | first | .address')")

	c.AddRunCmd("mkdir -p /srv/salt")

	c.AddRunCmd("echo interface: ${PRIVATE_IP} > /etc/salt/master.d/private-interface.conf")
	c.AddRunCmd("echo master: ${PRIVATE_IP} > /etc/salt/minion.d/master.conf")
	c.AddRunCmd("systemctl daemon-reload")
	c.AddRunCmd("systemctl enable salt-master.service")
	c.AddRunCmd("systemctl restart --no-block salt-master.service")
	c.AddRunCmd("systemctl enable salt-minion.service")
	c.AddRunCmd("systemctl restart --no-block salt-minion.service")

	script, err := c.RenderScript()

	if err != nil {
		panic(err)
	}

	return script
}

To explain what we are doing here: -

• We create a new CloudInit definition
  • The focal reference works for both Ubuntu and Debian
• We use AddRunCmd functions (i.e. those that run only on first boot, rather than every boot) to install Salt, and make this machine a Salt Master
• We create a master.conf file to instruct the Salt Master to: -
  • Add autosigned grains based upon a node’s role (full explanation below)
  • Use the root file system, and also gitfs (i.e. mounting a Git repository as a file system)
  • Specify our Salt repository as a gitfs_remote, meaning it will be mounted and used by Salt to provide states
    • We also tell it to update every 120s, to allow changes in the repository to be reflected on the file system
  • Provide our standard pillars directory
  • Reference the pillars directory in our Salt repository as an external pillar, allowing us to update the pillars directly through Git
• We add a minion configuration file which says to: -
  • Autosign our role grain
  • When the minion is accepted by the master, automatically run a high state (i.e. apply all configuration specific to this node)
  • Add a grain of our role called master
• We create our autosign_grains_dir directory
• We place inside it a file called role, containing the lines master and bgp
• We derive our private IPv4 address from the Equinix Metal metadata service
• We specify that Salt should only accept minions on our private interface (i.e. the interface with our private IP)
• We specify to the Minion that it should communicate with the master on the same private IPv4 address

The subsequent steps run a SystemD daemon-reload to ensure we are taking into account the latest SystemD unit files, and then we enable and restart the salt-master and salt-minion services.

Finally, we render all of our steps into a cloud-init script and return it.

To explain what Grain Autosigning is, this is where a Salt Master will automatically accept any Minion that has a Grain that matches those specified in the files in the autosign-grains directory. For example, our autosign-grains/role file will look like the following: -

master
bgp

Any minion that has a grain of role with a value of master or bgp is automatically accepted. Without this, you need to manually login to the salt-master and then accept the minions using something like salt-key -a 'salt-master.example.com' or salt-key -a 'bgp-01*'.

So long as we have a grain called role, and the contents of this grain match either master or bgp, the minion will then be accepted automatically (notice in the minion.conf file creation, we add the role grain).

In a production scenario, you need to ensure that either your master/minion communication is completely private or that the accepted grain values are sufficiently unique (e.g. a UUID value) and masked/encrypted in your repository, otherwise any server running a Salt minion that has these grains could be accepted by the master. Depending on your Salt setup, this could provide anything from basic information about your infrastructure to user creation that allows connectivity to other areas of your network.

Referencing the function

To reference this function in our code, we use the following: -

saltCloudInit := saltMasterCloudInitConfig()

As the return type is a string, we can use this directly in the Equinix Device creation functions like so: -

UserData: pulumi.String(saltCloudInit)

The bgp/anycast nodes

The function we use for the anycast nodes is similar to the salt-master Cloud-Init creation function. However we do need to supply a few variables to the function for ExaBGP and the anycast IP to be configured.

We do this by creating a struct, and allowing our function to have an instance of this struct passed to it.

type BgpNodeConfig struct {
	anycastIP     string
	anycastIPMask string
	saltMasterIP  string
}

func bgpCloudInitConfig(config *BgpNodeConfig) string {
	c, err := cloudinit.New("focal")

	if err != nil {
		panic(err)
	}

	c.AddRunCmd("curl -fsSL https://bootstrap.saltproject.io -o install_salt.sh")
	c.AddRunCmd("sh install_salt.sh -P -x python3")

	c.AddRunCmd("PRIVATE_IP=$(curl -s https://metadata.platformequinix.com/metadata | jq -r '.network.addresses | map(select(.public==false)) | first | .address')")

	c.AddRunCmd(fmt.Sprintf("echo master: %s > /etc/salt/minion.d/master.conf", config.saltMasterIP))

	c.AddRunTextFile("/etc/salt/grains",
		fmt.Sprintf(
			"anycast:\n  ipv4:\n    - address: %s\n      mask: %s\n",
			config.anycastIP,
			config.anycastIPMask,
		), 0400,
	)

	c.AddRunTextFile("/etc/salt/minion.d/minion.conf", `autosign_grains:
- role
startup_states: highstate`, 0644)

	c.AddRunCmd("echo \"bgp:\n  localip: ${PRIVATE_IP}\nrole: bgp\" >> /etc/salt/grains")

	c.AddRunCmd("systemctl daemon-reload")
	c.AddRunCmd("systemctl enable salt-minion.service")
	c.AddRunCmd("systemctl restart --no-block salt-minion.service")

	script, err := c.RenderScript()

	if err != nil {
		panic(err)
	}

	return script
}

The differences between this and the saltMasterCloudInitConfig function are as follows: -

• We create a BgpNodeConfig struct, which will be used to provide the anycast IPv4 address, it’s subnet mask, and the IPv4 address of the Salt Master
• We create the bgpCloudInitConfig function, which requires an input variable (an instance of the BgpNodeConfig struct) and returns the cloud-init script as a string
• We install salt, but omit the -M flag so that we don’t make this machine a master as well
• We create our /etc/salt/grains file, and populate it with our anycast IPv4 address and subnet mask
• We use a similar minion configuration file as for the master, except we are placing our grains in a separate file
• We append the private IPv4 address as a grain to our /etc/salt/grains/ file as the bgp.localip grain, as well as adding our bgp role grain (used to autosign this minion)
• Finally, we run a SystemD daemon-reload, and then enable and restart the salt-minion process

We render the cloud-init script, and return it as a string.

Because the local IP, Salt Master IP and Anycast IP address and mask will only be known after the machines are provisioned, we must ensure that we are not hardcoding values into the cloud-init templates. This avoids them being out-of-date/invalid if we recreate our infrastructure.

Another point to note is nothing is specific to each node in this function, so we can use the same generated cloud-init script for one, two or ten BGP-speaking anycast nodes.

Referencing the function

Unlike the Salt Master cloud-init script, we must supply a few variables that are only known at the time of being provisioned. To do this, we use something like the below: -

		bgpCloudConfig := pulumi.All(
			anycastIpAllocation.Address,
			anycastIpAllocation.Netmask,
			saltMaster.AccessPrivateIpv4).ApplyT(
			func(args []interface{}) string {
				bgpCloudInitNodeConfig := BgpNodeConfig{
					anycastIP:     args[0].(string),
					anycastIPMask: args[1].(string),
					saltMasterIP:  args[2].(string),
				}
				return bgpCloudInitConfig(&bgpCloudInitNodeConfig)
			})

In the above we are:-

• Using an All Pulumi function to supply values not yet known (i.e. the Anycast IPv4 details and Salt Master IPv4 address) to an Apply function
• The Apply function will run after the creation of the Salt Master server and the Anycast IP allocation
• It creates an instance of the BgpNodeConfig struct, with the values derived from the provisioned anycast IP and Salt Master
• This instance of the struct is then passed to the bgpCloudInitConfig function
• The function returns the cloud-init script ready to pass to our BGP nodes, containing the correct anycast IP and Salt Master details

As mentioned, we don’t know the details required by the function until Pulumi runs, at which point they are considered Pulumi Outputs. The Juju cloudinit module has no concept of a Pulumi Output (i.e. it cannot natively evaluate a Pulumi Output type). Therefore we use the Pulumi Apply function to take the Output values, turn them into a more common type (e.g. a string or int) to pass to our cloud-init function.

The bgpCloudConfig will be of a Pulumi Output type after the Apply function runs, meaning that Pulumi is aware of the context and history of creating and evaluating the other Pulumi Outputs (i.e. the anycast IP details and Salt Master IP).

Creating the salt-master

Now that we have the correct cloud-init data for provisioning a salt-master, we can create one like so: -

		saltMaster, err := metal.NewDevice(ctx, "salt-master", &metal.DeviceArgs{
			Hostname:        pulumi.String("salt-master"),
			Plan:            pulumi.String("c3.small.x86"),
			Metro:           pulumi.String("am"),
			OperatingSystem: pulumi.String("ubuntu_20_04"),
			BillingCycle:    pulumi.String("hourly"),
			ProjectId:       projectId,
			ProjectSshKeyIds: pulumi.StringArray{
				sshkey.ID(),
			},
			UserData: pulumi.String(saltCloudInit),
		})

		if err != nil {
			return err
		}

A small difference between the previous Pulumi post and this one is that Equinix Metal now use the term Metro rather than Facility to refer to the location you create infrastructure in. The reason for this is Equinix Metal can now provide VLANs, elastic IPs and Private IPv4 connectivity across multiple facilities in a geographic area, rather than being limited to a single location. If you are familiar with AWS, think of a Metro like a Region, and a Facility like an Availability Zone.

This has the added benefit that more capacity is available across multiple facilities, without being limited by network topology (i.e. VLANs being restricted to a single facility) between the infrastructure you provision.

Other than this, the above is very similar to other servers we have created in Equinix Metal. We pass in our saltCloudInit variable, which contains the cloud-init script generated by the saltMasterCloudInitConfig function.

Creating the anycast IP

The anycast IP uses Equinix’s NewReservedIPBlock function to reserve a single IP from the metro’s IPv4 allocation. If you have your own IP space and use the Global BGP option when adding BGP to the initial Equinix Metal project, you are not limited by the metro as to where you can advertise the IPs from.

		anycastIpAllocation, err := metal.NewReservedIpBlock(ctx, "anycast-ip", &metal.ReservedIpBlockArgs{
			ProjectId: projectId,
			Metro:     pulumi.String("am"),
			Type:      pulumi.String("public_ipv4"),
			Quantity:  pulumi.Int(1),
		})

		if err != nil {
			return err
		}

The above allocates a single IPv4 address in the Equinix Metal am (Amsterdam) metro. We specify public_ipv4, as the other option (global_ipv4) is only valid if deploying directly to a facility.

Creating our anycast nodes and BGP sessions

Now that we have our anycast IP, and we have generated our cloud-init script, we can create our anycast nodes and the BGP sessions from Equinix to the nodes: -

		bgpPrimary, err := metal.NewDevice(ctx, "bgp-01", &metal.DeviceArgs{
			Hostname:        pulumi.String("bgp-01"),
			Plan:            pulumi.String("c3.small.x86"),
			Metro:           pulumi.String("am"),
			OperatingSystem: pulumi.String("debian_10"),
			BillingCycle:    pulumi.String("hourly"),
			ProjectId:       projectId,
			ProjectSshKeyIds: pulumi.StringArray{
				sshkey.ID(),
			},
			UserData: pulumi.StringOutput(pulumi.Sprintf("%v", bgpCloudConfig)),
		})

		if err != nil {
			return err
		}

		bgpPrimarySession, err := metal.NewBgpSession(ctx, "bgp-01_BGP_SESSION", &metal.BgpSessionArgs{
			AddressFamily: pulumi.String("ipv4"),
			DeviceId:      bgpPrimary.ID(),
		})

		if err != nil {
			return err
		}

In the device creation, we are using Debian Buster rather than Ubuntu 20.04 LTS (which we used for the ‘salt-master’), mainly due to ease of managing networks on Debian Buster currently. Also, we must turn the bgpCloudConfig Pulumi Output into a Pulumi String type (a StringOutput is fine in this context) as the UserData field doesn’t accept values of type Pulumi Output (only Pulumi String types).

After this, we create an IPv4 BGP session that is associated with this server.

The second anycast node we create is identical to this, except we use the word Secondary instead of Primary, and 02 instead of 01 for the hostname and resource names.

All the code

We have the code in two separate files, one called main.go (which contains all of the Equinix Metal-specific code) and one called cloud-init.go that contains all of the cloud-init-specific code.

main.go

package main

import (
	"fmt"
	"io/ioutil"
	"os/user"

	metal "github.com/pulumi/pulumi-equinix-metal/sdk/v2/go/equinix"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		conf := config.New(ctx, "")
		commonName := conf.Require("common_name")

		rootStack, err := pulumi.NewStackReference(ctx, "yetiops/equinix-metal-yetiops/staging", nil)

		if err != nil {
			return err
		}

		projectId := rootStack.GetStringOutput(pulumi.String("projectId"))

		user, err := user.Current()

		if err != nil {
			return err
		}

		sshkey_path := fmt.Sprintf("%v/.ssh/id_rsa.pub", user.HomeDir)

		sshkey_file, err := ioutil.ReadFile(sshkey_path)
		if err != nil {
			return err
		}

		sshkey_contents := string(sshkey_file)

		sshkey, err := metal.NewProjectSshKey(ctx, commonName, &metal.ProjectSshKeyArgs{
			Name:      pulumi.String(commonName),
			PublicKey: pulumi.String(sshkey_contents),
			ProjectId: projectId,
		})

		saltCloudInit := saltMasterCloudInitConfig()

		saltMaster, err := metal.NewDevice(ctx, "salt-master", &metal.DeviceArgs{
			Hostname:        pulumi.String("salt-master"),
			Plan:            pulumi.String("c3.small.x86"),
			Metro:           pulumi.String("am"),
			OperatingSystem: pulumi.String("ubuntu_20_04"),
			BillingCycle:    pulumi.String("hourly"),
			ProjectId:       projectId,
			ProjectSshKeyIds: pulumi.StringArray{
				sshkey.ID(),
			},
			UserData: pulumi.String(saltCloudInit),
		})

		if err != nil {
			return err
		}

		anycastIpAllocation, err := metal.NewReservedIpBlock(ctx, "anycast-ip", &metal.ReservedIpBlockArgs{
			ProjectId: projectId,
			Metro:     pulumi.String("am"),
			Type:      pulumi.String("public_ipv4"),
			Quantity:  pulumi.Int(1),
		})

		if err != nil {
			return err
		}

		bgpCloudConfig := pulumi.All(
			anycastIpAllocation.Address,
			anycastIpAllocation.Netmask,
			saltMaster.AccessPrivateIpv4).ApplyT(
			func(args []interface{}) string {
				bgpCloudInitNodeConfig := BgpNodeConfig{
					anycastIP:     args[0].(string),
					anycastIPMask: args[1].(string),
					saltMasterIP:  args[2].(string),
				}
				return bgpCloudInitConfig(&bgpCloudInitNodeConfig)
			})

		if err != nil {
			return err
		}

		bgpPrimary, err := metal.NewDevice(ctx, "bgp-01", &metal.DeviceArgs{
			Hostname:        pulumi.String("bgp-01"),
			Plan:            pulumi.String("c3.small.x86"),
			Metro:           pulumi.String("am"),
			OperatingSystem: pulumi.String("debian_10"),
			BillingCycle:    pulumi.String("hourly"),
			ProjectId:       projectId,
			ProjectSshKeyIds: pulumi.StringArray{
				sshkey.ID(),
			},
			UserData: pulumi.StringOutput(pulumi.Sprintf("%v", bgpCloudConfig)),
		})

		if err != nil {
			return err
		}

		bgpPrimarySession, err := metal.NewBgpSession(ctx, "bgp-01_BGP_SESSION", &metal.BgpSessionArgs{
			AddressFamily: pulumi.String("ipv4"),
			DeviceId:      bgpPrimary.ID(),
		})

		if err != nil {
			return err
		}

		bgpSecondary, err := metal.NewDevice(ctx, "bgp-02", &metal.DeviceArgs{
			Hostname:        pulumi.String("bgp-02"),
			Plan:            pulumi.String("c3.small.x86"),
			Metro:           pulumi.String("am"),
			OperatingSystem: pulumi.String("debian_10"),
			BillingCycle:    pulumi.String("hourly"),
			ProjectId:       projectId,
			ProjectSshKeyIds: pulumi.StringArray{
				sshkey.ID(),
			},
			UserData: pulumi.StringOutput(pulumi.Sprintf("%v", bgpCloudConfig)),
		})

		if err != nil {
			return err
		}

		bgpSecondarySession, err := metal.NewBgpSession(ctx, "bgp-02_BGP_SESSION", &metal.BgpSessionArgs{
			AddressFamily: pulumi.String("ipv4"),
			DeviceId:      bgpSecondary.ID(),
		})

		if err != nil {
			return err
		}

		ctx.Export("saltMasterIP", saltMaster.AccessPublicIpv4)
		ctx.Export("bgpPrimaryIP", bgpPrimary.AccessPublicIpv4)
		ctx.Export("bgpSecondaryIP", bgpSecondary.AccessPublicIpv4)
		ctx.Export("bgpPrimarySessionStatus", bgpPrimarySession.Status)
		ctx.Export("bgpSecondarySessionStatus", bgpSecondarySession.Status)
		ctx.Export("anycastIP", anycastIpAllocation.Address)
		return nil
	})
}

cloud-init.go

package main

import (
	"fmt"

	"github.com/juju/juju/cloudconfig/cloudinit"
)

type BgpNodeConfig struct {
	anycastIP     string
	anycastIPMask string
	saltMasterIP  string
}

func saltMasterCloudInitConfig() string {
	c, err := cloudinit.New("focal")

	if err != nil {
		panic(err)
	}

	c.AddRunCmd("curl -fsSL https://bootstrap.saltproject.io -o install_salt.sh")
	c.AddRunCmd("sh install_salt.sh -P -M -x python3")

	c.AddRunTextFile("/etc/salt/master.d/master.conf", `autosign_grains_dir: /etc/salt/autosign-grains
fileserver_backend:
  - roots
  - gitfs

gitfs_remotes:
  - https://gitlab.com/stuh84/salt-anycast-equinix:
    - root: states
    - base: main
    - update_interval: 120


pillar_roots:
  base:
    - /srv/salt/pillars

ext_pillar:
  - git:
    - main https://gitlab.com/stuh84/salt-anycast-equinix:
      - root: pillars
      - env: base`, 0644)

	c.AddRunTextFile("/etc/salt/minion.d/minion.conf", `autosign_grains:
- role
startup_states: highstate
grains:
  role: master`, 0644)

	c.AddRunCmd("mkdir -p /etc/salt/autosign-grains")

	c.AddRunCmd("echo -e \"master\nbgp\n\" > /etc/salt/autosign-grains/role")

	c.AddRunCmd("PRIVATE_IP=$(curl -s https://metadata.platformequinix.com/metadata | jq -r '.network.addresses | map(select(.public==false)) | first | .address')")

	c.AddRunCmd("mkdir -p /srv/salt")

	c.AddRunCmd("echo interface: ${PRIVATE_IP} > /etc/salt/master.d/private-interface.conf")
	c.AddRunCmd("echo master: ${PRIVATE_IP} > /etc/salt/minion.d/master.conf")
	c.AddRunCmd("systemctl daemon-reload")
	c.AddRunCmd("systemctl enable salt-master.service")
	c.AddRunCmd("systemctl restart --no-block salt-master.service")
	c.AddRunCmd("systemctl enable salt-minion.service")
	c.AddRunCmd("systemctl restart --no-block salt-minion.service")

	script, err := c.RenderScript()

	if err != nil {
		panic(err)
	}

	return script
}

func bgpCloudInitConfig(config *BgpNodeConfig) string {
	c, err := cloudinit.New("focal")

	if err != nil {
		panic(err)
	}

	c.AddRunCmd("curl -fsSL https://bootstrap.saltproject.io -o install_salt.sh")
	c.AddRunCmd("sh install_salt.sh -P -x python3")

	c.AddRunCmd("PRIVATE_IP=$(curl -s https://metadata.platformequinix.com/metadata | jq -r '.network.addresses | map(select(.public==false)) | first | .address')")

	c.AddRunCmd(fmt.Sprintf("echo master: %s > /etc/salt/minion.d/master.conf", config.saltMasterIP))

	c.AddRunTextFile("/etc/salt/grains",
		fmt.Sprintf(
			"anycast:\n  ipv4:\n    - address: %s\n      mask: %s\n",
			config.anycastIP,
			config.anycastIPMask,
		), 0400,
	)

	c.AddRunTextFile("/etc/salt/minion.d/minion.conf", `autosign_grains:
- role
startup_states: highstate`, 0644)

	c.AddRunCmd("echo \"bgp:\n  localip: ${PRIVATE_IP}\nrole: bgp\" >> /etc/salt/grains")

	c.AddRunCmd("systemctl daemon-reload")
	c.AddRunCmd("systemctl enable salt-minion.service")
	c.AddRunCmd("systemctl restart --no-block salt-minion.service")

	script, err := c.RenderScript()

	if err != nil {
		panic(err)
	}

	return script
}

Provisioning

Now that we have created our Pulumi code and our Salt configuration files, we can run Pulumi and watch everything create!

Below is an Asciinema output of my terminal when running pulumi up, so you can see the resources being created: -

We can also take a look at what the generated cloud-init script files look like too: -

salt-master

#!/bin/bash
set -e
test -n "$JUJU_PROGRESS_FD" || (exec {JUJU_PROGRESS_FD}>&2) 2>/dev/null && exec {JUJU_PROGRESS_FD}>&2 || JUJU_PROGRESS_FD=2
(

function package_manager_loop {
    local rc=
    while true; do
        if ($*); then
                return 0
        else
                rc=$?
        fi
        if [ $rc -eq 100 ]; then
                sleep 10s
                continue
        fi
        return $rc
    done
}

curl -fsSL https://bootstrap.saltproject.io -o install_salt.sh
sh install_salt.sh -P -M -x python3
install -D -m 644 /dev/null '/etc/salt/master.d/master.conf'
printf '%s\n' 'autosign_grains_dir: /etc/salt/autosign-grains
fileserver_backend:
  - roots
  - gitfs

gitfs_remotes:
  - https://gitlab.com/stuh84/salt-anycast-equinix:
    - root: states
    - base: main
    - update_interval: 120


pillar_roots:
  base:
    - /srv/salt/pillars

ext_pillar:
  - git:
    - main https://gitlab.com/stuh84/salt-anycast-equinix:
      - root: pillars
      - env: base' > '/etc/salt/master.d/master.conf'
install -D -m 644 /dev/null '/etc/salt/minion.d/minion.conf'
printf '%s\n' 'autosign_grains:
- role
startup_states: highstate
grains:
  role: master' > '/etc/salt/minion.d/minion.conf'
mkdir -p /etc/salt/autosign-grains
echo -e "master
bgp
" > /etc/salt/autosign-grains/role
PRIVATE_IP=$(curl -s https://metadata.platformequinix.com/metadata | jq -r '.network.addresses | map(select(.public==false)) | first | .address')
mkdir -p /srv/salt
echo interface: ${PRIVATE_IP} > /etc/salt/master.d/private-interface.conf
echo master: ${PRIVATE_IP} > /etc/salt/minion.d/master.conf
systemctl daemon-reload
systemctl enable salt-master.service
systemctl restart --no-block salt-master.service
systemctl enable salt-minion.service
systemctl restart --no-block salt-minion.service
)  2>&1