In a previous post I covered how to use Consul for service discovery of standard exporters, allowing Prometheus to automatically discover what services to monitor.

However, this configuration didn’t cater to exporters like the snmp_exporter or blackbox_exporter.

What is interesting about both of the above is that rather than generating metrics for a local application, they are a proxy for other services. For example, you can use BlackBox exporter to do ICMP checks or HTTPS checks, without running an exporter on the services themselves. This means you could do checks on say, Cloudflare’s 1.1.1.1 service, without running an exporter within Cloudflare.

The standard configuration for the Consul Service Discovery within Prometheus didn’t lend itself to this kind of service. Through some searching though, I came across a couple of useful resources pertaining to how to configure these kind of services (see the end of this page).

Relabelling

Prometheus can use a concept called relabelling, where information gleaned from the service discovery mechanism can override defaults. For example, you could override the module name (useful for SNMP, as the SNMP exporter can have multiple different modules configured in one instance).

You can also use it to relabel the target of the requests, essentially meaning that we can add a tag to a service in Consul, and use it to override the target that Blackbox or SNMP exporters will attempt to make requests to.

Consul Service configuration - blackbox_exporter

The below is a service definition for a Blackbox exporter service: -

{"service":
  {"name": "icmp",
   "tags": [
     "icmp:192.168.10.3",
     "prometheus-bb-icmp"
     ]
  }
}

In the above, we have given the service two tags, the prometheus-bb-icmp tag (which can be used by Prometheus to match service) and the icmp:192.168.0.3 tag. With a little bit of regex magic, we can extract the IP from the service, and override the target of an ICMP Blackbox exporter check.

The corresponding section in the Prometheus configuration is as such: -

  - job_name: 'blackbox_consul'
    metrics_path: /probe
    params:
      module: [icmp_ipv4]
    consul_sd_configs:
      - server: '192.168.0.7:8500'
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*,prometheus-bb-icmp,.*
        action: keep
      - source_labels: [__meta_consul_tags]
        regex: .*,icmp:([^,]+),.*
        replacement: '${1}'
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

As you can see, we are matching the __meta_consul_tags label. Within this, we are matching everything after icmp:, and then pushing it to the target_label of __param_target. This is what overrides the target parameter, which Prometheus will tell the blackbox_exporter to use as a target.

Multiple Service Definitions on one node

If you want to check multiple targets (say, multiple external targets), you can construct a file containing multiple services, and allow Consul to pick this up. An example of this is below: -

{"services": [
  {"id": "icmp_cloudflare_dns",
   "name": "icmp",
   "tags": [
      "icmp:1.1.1.1",
      "prometheus-bb-icmp"
       ]
  },
  {"id": "icmp_router",
   "name": "icmp",
   "tags": [
      "icmp:192.168.0.1",
      "prometheus-bb-icmp"
     ]
  },
  {"id": "icmp_mdns-01",
   "name": "icmp",
   "tags": [
      "icmp:192.168.0.21",
      "prometheus-bb-icmp"
     ]
  },
[...]

The structure of this file is quite important (a list of JSON statements, not replicating the services keyword etc), and you also must specify unique IDs for the services. I personally generate this using Ansible, but it is up to you how you would go about this.

Pay attention to the keyword services rather than service. Without this, the configuration will not have the intended effect.

Consul Service configuration - snmp_exporter

For the SNMP exporter, the configuration is very similar. As I did this one later in my quest, it also includes the module override, as well as target override. This means I do not need configuration for each different SNMP module.

I can use the same configuration in Prometheus for Unifi, MikroTik, and anything else, only needing to update the Consul service, and the snmp_exporter configuration itself if I had further modules down the line.

{"services": [
  {"id": "snmp_ifmib_meshuggah",
   "name": "snmp",
   "tags": [
      "snmp:192.168.0.7",
      "module:if_mib",
      "prometheus-bb-snmp"
     ]
  },
  {"id": "snmp_mikrotik_router",
   "name": "snmp",
   "tags": [
      "snmp:192.168.0.1",
      "module:mikrotik",
      "prometheus-bb-snmp"
     ]
  },
  {"id": "snmp_ubiquiti_officeap",
   "name": "snmp",
   "tags": [
      "snmp:192.168.0.248",
      "module:ubiquiti_unifi",
      "prometheus-bb-snmp"
     ]
  },
[...]

As you can see, we have the same style tags as per the Blackbox Exporter check, but also we have the module tag as well. This specifies the Module to override the SNMP exporter configuration with.

The corresponding Prometheus configuration is below: -

  - job_name: 'blackbox_consul_snmp'
    metrics_path: /snmp
    params:
      module: [ifmib_snmp]
    consul_sd_configs:
      - server: '192.168.0.7:8500'
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*,prometheus-bb-snmp,.*
        action: keep
      - source_labels: [__meta_consul_tags]
        regex: .*,module:([^,]+),.*
        replacement: '${1}'
        target_label: __param_module
      - source_labels: [__param_module]
        target_label: job
      - source_labels: [__meta_consul_tags]
        regex: .*,snmp:([^,]+),.*
        replacement: '${1}'
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116

In the above, you can see that we have the target configuration, similar to the blackbox exporter configuration. However, we also match the module tag, saying anything after module: will override the target label __param_module.

We have a default module parameter of ifmib_snmp, which means that if nothing is specified, we’ll default to using the ifmib_snmp module (which in my configuration means nothing more than using the standard SNMP ifmib MIB definitions for stats).

Going back to the service definition above, we have tags with module:ubiquiti_unifi and module:mikrotik. These will override the SNMP configuration module, meaning that for MikroTik hosts the query to the SNMP exporter will use mikrotik as the module parameter, and ubiquiti_unifi for any Ubiquiti equipment.

Targets in Prometheus

The resulting targets in Prometheus are discovered as such: -

ICMP targets

Prometheus Consul Service Discovery with ICMP

SNMP targets

Prometheus Consul Service Discovery with SNMP

Summary

After doing all of this, my Prometheus configuration is now almost entirely fed by Consul. I have two small services I need to move (with the hold up being about trying to not commit secrets in version control).

The two main resources allowing me to refactor my configuration like this are this issue on the Prometheus GitHub and the below talk from PromCon 2019 on migrating from Nagios to Prometheus: -

The talk (and all the others from PromCon 2019) are highly recommended.