In my previous posts on Prometheus, most of the monitoring has been geared to either getting metrics from Linux hosts. There have been a couple of exceptions, like the blackbox_exporter (for ICMP/Ping, HTTP(S) and TCP socket monitoring) or the snmp_exporter (more commonly used for networking gear).

Linux is not the only operating system you can monitor node metrics for though. Other operating systems have either their own port of the node_exporter, or via other exporters entirely.

I’m going to cover the following: -

  • OpenWRT
  • Windows
  • OpenBSD
  • FreeBSD

OpenWRT

OpenWRT has a Linux base, but is more akin to an embedded version of Linux (running a modified form of BusyBox) than your “full fat” Linux that you’d find in something like Debian or CentOS.

Typically, OpenWRT devices have very little flash storage and RAM. For example, a TP-Link C58 Archer v1 has the following: -

[root@tpap] ~ # free -mh
             total       used       free     shared    buffers     cached
Mem:         60288      39632      20656       1408       1168       7076
-/+ buffers/cache:      31388      28900
Swap:            0          0          0

[root@tpap] ~ # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 2.8M      2.8M         0 100% /rom
tmpfs                    29.4M      1.2M     28.2M   4% /tmp
/dev/mtdblock5            3.5M      2.3M      1.2M  66% /overlay
overlayfs:/overlay        3.5M      2.3M      1.2M  66% /
tmpfs                   512.0K         0    512.0K   0% /dev

In total we have 64M of RAM available, and 4M of flash storage. If you take a look at the releases page of the Prometheus node_exporter, the smallest binary available for Linux is already 7M. So what do we do?

OpenWRT Node Exporter

Thankfully, someone has already written a node_exporter package for OpenWRT. This exposes metrics the same way the standard node_exporter does (a HTTP server on TCP port 9100)

To install this, you can do it via the OpenWRT GUI in System > Software, search for promtheus-node-exporter, and you’ll get a number of results. Alternatively, done via the OpenWRT CLI: -

[root@tpap] ~ # opkg list *prometheus*
prometheus-node-exporter-lua - 2019.08.14-1 - Provides node metrics as Prometheus scraping endpoint.
 
 This service is a lightweight rewrite in LUA of the offical Prometheus node_exporter.
prometheus-node-exporter-lua-bmx6 - 2019.08.14-1 - Prometheus node exporter (bmx6 links collector)
prometheus-node-exporter-lua-bmx7 - 2019.08.14-1 - Prometheus node exporter (bmx7 links collector)
prometheus-node-exporter-lua-nat_traffic - 2019.08.14-1 - Prometheus node exporter (nat_traffic collector)
prometheus-node-exporter-lua-netstat - 2019.08.14-1 - Prometheus node exporter (netstat collector)
prometheus-node-exporter-lua-openwrt - 2019.08.14-1 - Prometheus node exporter (openwrt collector)
prometheus-node-exporter-lua-textfile - 2019.08.14-1 - Prometheus node exporter (textfile collector)
prometheus-node-exporter-lua-wifi - 2019.08.14-1 - Prometheus node exporter (wifi collector)
prometheus-node-exporter-lua-wifi_stations - 2019.08.14-1 - Prometheus node exporter (wifi_stations collector)

As you can see by the descriptions, there are a number of extensions to the node_exporter to export additional statistics. If you are only interested in the statistics for Network Address Translation, you can install the prometheus-node-exporter-lua-nat_traffic module. In my installation I have tended to use the openwrt, netstat, wifi and wifi_stations modules.

To install the packages, just do opkg install prometheus-node-exporter-lua and opkg install prometheus-node-exporter-lua-$MODULE

Configuration

A small amount of configuration is required after you have installed the packages, to make the node_exporter listen on the correct IP: -

$ cat /etc/config/prometheus-node-exporter-lua 

config prometheus-node-exporter-lua 'main'
	option listen_address '192.168.0.245'
	option listen_port '9100'

Change the listen_address to the IP of your router, and run /etc/init.d/prometheus-node-exporter-lua restart

Verification

You can quickly check if the port is listening correctly with: -

## netstat -an | grep -i 9100
tcp        0      0 192.168.0.245:9100      0.0.0.0:*               LISTEN

You should then be able to visit the IP address, either via your browser or CuRL: -

$ curl 192.168.0.245:9100/metrics 
# TYPE node_scrape_collector_duration_seconds gauge
# TYPE node_scrape_collector_success gauge
# TYPE node_nf_conntrack_entries gauge
node_nf_conntrack_entries 51
# TYPE node_nf_conntrack_entries_limit gauge
node_nf_conntrack_entries_limit 16384
[...]
wifi_stations{ifname="wlan1"} 1
node_scrape_collector_duration_seconds{collector="wifi_stations"} 0.010385036468506
node_scrape_collector_success{collector="wifi_stations"} 1

Add this to your Prometheus instance using your chosen method of configuration management, and the statistics should appear in Prometheus.

Windows

Currently, node_exporter is not built for Windows. Instead there is an exporter called the wmi_exporter, which can be obtained here. This exposes the same kind of metrics you’d expect from node_exporter (CPU, disk, services, storage and more).

Binaries are available from the GitHub repository to install it, as well as instructions for installing it. It is also worth noting that the Chocolatey package is up to date at the time of writing.

My chosen method of installation is with Chocolatey

Chocolatey

Chocolatey is a package manager for Windows. It is not included with Windows by default, so you’ll need install it from here.

Once this is done, you can install wmi_exporter like so: -

PS C:\Users\Administrator> choco install prometheus-wmi-exporter.install
Chocolatey v0.10.15
Installing the following packages:
prometheus-wmi-exporter.install
By installing you accept licenses for the packages.
Progress: Downloading chocolatey-core.extension 1.3.4... 100%
Progress: Downloading prometheus-wmi-exporter.install 0.9.0... 100%

chocolatey-core.extension v1.3.4 [Approved]
chocolatey-core.extension package files install completed. Performing other installation steps.
 Installed/updated chocolatey-core extensions.
 The install of chocolatey-core.extension was successful.
  Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-core'

prometheus-wmi-exporter.install v0.9.0 [Approved]
prometheus-wmi-exporter.install package files install completed. Performing other installation steps.
The package prometheus-wmi-exporter.install wants to run 'chocolateyinstall.ps1'.
Note: If you don't run this script, the installation will fail.
Note: To confirm automatically next time, use '-y' or consider:
choco feature enable -n allowGlobalConfirmation
Do you want to run the script?([Y]es/[A]ll - yes to all/[N]o/[P]rint): A

Downloading prometheus-wmi-exporter.install 64 bit
  from 'https://github.com/martinlindhe/wmi_exporter/releases/download/v0.9.0/wmi_exporter-0.9.0-amd64.msi'
Progress: 100% - Completed download of C:\Users\Administrator\AppData\Local\Temp\chocolatey\prometheus-wmi-exporter.install\0.9.0\wmi_exporter-0.9.0-amd64.msi (7.18 MB).
Download of wmi_exporter-0.9.0-amd64.msi (7.18 MB) completed.
Hashes match.
Installing prometheus-wmi-exporter.install...
prometheus-wmi-exporter.install has been installed.
  prometheus-wmi-exporter.install may be able to be automatically uninstalled.
 The install of prometheus-wmi-exporter.install was successful.
  Software installed as 'MSI', install location is likely default.

Chocolatey installed 2/2 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Verification

To verify that it is running, you can check with: -

PS C:\Users\Administrator> netstat -an | Select-String 9182

  TCP    0.0.0.0:9182           0.0.0.0:0              LISTENING
  TCP    [::]:9182              [::]:0                 LISTENING

You should now also be able to retrieve metrics from the HTTP endpoint: -

$ curl 192.168.56.181:9182/metrics 
[...]
# HELP wmi_system_context_switches_total Total number of context switches (WMI source: PerfOS_System.ContextSwitchesPersec)
# TYPE wmi_system_context_switches_total counter
wmi_system_context_switches_total 580353
# HELP wmi_system_exception_dispatches_total Total number of exceptions dispatched (WMI source: PerfOS_System.ExceptionDispatchesPersec)
# TYPE wmi_system_exception_dispatches_total counter
wmi_system_exception_dispatches_total 33872
# HELP wmi_system_processor_queue_length Length of processor queue (WMI source: PerfOS_System.ProcessorQueueLength)
# TYPE wmi_system_processor_queue_length gauge
wmi_system_processor_queue_length 4
# HELP wmi_system_system_calls_total Total number of system calls (WMI source: PerfOS_System.SystemCallsPersec)
# TYPE wmi_system_system_calls_total counter
wmi_system_system_calls_total 6.957803e+06
# HELP wmi_system_system_up_time System boot time (WMI source: PerfOS_System.SystemUpTime)
# TYPE wmi_system_system_up_time gauge
wmi_system_system_up_time 1.573736799799976e+09
# HELP wmi_system_threads Current number of threads (WMI source: PerfOS_System.Threads)
# TYPE wmi_system_threads gauge
wmi_system_threads 417
# HELP wmi_textfile_scrape_error 1 if there was an error opening or reading a file, 0 otherwise
# TYPE wmi_textfile_scrape_error gauge
wmi_textfile_scrape_error 0

Add this into your Prometheus configuration ($IP-OF-WINDOWS-MACHINE:9182) and you should see metrics available.

Dashboards

A number of Grafana dashboards are available on the Grafana Dashboards site for the wmi_exporter. I personally like this one.

OpenBSD

On the latest available stable version of OpenBSD (6.6 at the time of writing), node_exporter is available in the OpenBSD package repository by default (rather than in the Ports tree).

To check, do: -

openbsd-util$ pkg_info -Q node_exporter
node_exporter-0.18.0

To install: -

openbsd-util$ doas pkg_add node_exporter
doas (stuh84@openbsd-util) password: 
quirks-3.182 signed on 2019-11-13T20:45:48Z
quirks-3.182: ok
node_exporter-0.18.0: ok
The following new rcscripts were installed: /etc/rc.d/node_exporter
See rcctl(8) for details.

And to run it: -

openbsd-util$ doas rcctl enable node_exporter
doas (stuh84@openbsd-util) password: 
openbsd-util$ doas rcctl start node_exporter 
doas (stuh84@openbsd-util) password: 
node_exporter(ok)

Verification

As before, check with netstat: -

openbsd-util$ netstat -an | grep -i 9100
tcp          0      0  *.9100                 *.*                    LISTEN

Check with CuRL or your browser: -

curl 192.168.56.182:9100/metrics       
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
[...]
# HELP node_network_transmit_bytes_total Network device statistic transmit_bytes.
# TYPE node_network_transmit_bytes_total counter
node_network_transmit_bytes_total{device="enc0"} 0
node_network_transmit_bytes_total{device="hvn0"} 354191
node_network_transmit_bytes_total{device="hvn1"} 94687
node_network_transmit_bytes_total{device="lo0"} 0
node_network_transmit_bytes_total{device="pflog0"} 0
[...]

Metrics

The metrics you will get out of OpenBSD compared to Linux differ, due to differences in how they are calculated. A good example is Load.

In the Linux world, load is a metric that takes into account CPU and demand for disks (see Brendan Gregg’s brilliant and mind-bending blog post on this). In the BSD world however, Load is purely a CPU metric.

If you intend to graph metrics across Linux and BSD systems, the same queries will give drastically different results.

FreeBSD

On the latest available stable version of FreeBSD (12.1 at the time of writing), node_exporter is available in the FreeBSD package repository by default (rather than in the Ports tree).

To check: -

$ pkg search node_exporter
node_exporter-0.18.1           Prometheus exporter for machine metrics

To install: -

$ sudo pkg install node_exporter
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	node_exporter: 0.18.1

Number of packages to be installed: 1

The process will require 10 MiB more space.
3 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching node_exporter-0.18.1.txz: 100%    3 MiB   1.1MB/s    00:03    
Checking integrity... done (0 conflicting)
[1/1] Installing node_exporter-0.18.1...
[1/1] Extracting node_exporter-0.18.1: 100%
=====
Message from node_exporter-0.18.1:

--
If upgrading from a version of node_exporter <0.15.0 you'll need to update any
custom command line flags that you may have set as it now requires a
double-dash (--flag) instead of a single dash (-flag).
The collector flags in 0.15.0 have now been replaced with individual boolean
flags and the -collector.procfs` and -collector.sysfs` flags have been renamed
to --path.procfs and --path.sysfs respectively.

To enable and start: -

$ sudo sysrc node_exporter_enable=YES
node_exporter_enable:  -> YES
$ sudo service node_exporter start
Starting node_exporter.

Verification

Again, check with netstat: -

$ netstat -an | grep -i 9100
tcp46      0      0 *.9100                 *.*                    LISTEN

Check with CuRL or your browser: -

curl 192.168.56.182:9100/metrics       
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
[...]
# HELP node_uname_info Labeled system information as provided by the uname system call.
# TYPE node_uname_info gauge
node_uname_info{domainname="(none)",machine="amd64",nodename="freebsd-util",release="12.1-RELEASE",sysname="FreeBSD",version="FreeBSD 12.1-RELEASE r354233 GENERIC "} 1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

Metrics

As noted in the OpenBSD section, the metrics you will scrape from FreeBSD are calculated differently than in Linux. Take this into account when building your queries and dashboards, especially for dashboards mixing BSD and Linux metrics.

Other systems

In future posts I’ll try and cover other operating systems (e.g. Illumos).

Hopefully this post has given you a good indication of the usual steps you take to verify an installation of node_exporter (or wmi_exporter for Windows). On other systems it should just be a case of finding out where to source node_exporter from, whether that is through binaries, package management or building from source.

Update (28/11/2019)

Not long after I posted this blog, a project for monitoring FreeBSD GEOM devices appeared called gstat_exporter. If you are wanting to monitor ZFS and Disk Utilization, this is a great way to start.

Go look at this excellent blog post “All in a days work: Prometheus gstat_exporter and Grafana Dashboard” by Tykling