Metrics: the Janus exporter

Every node serves its own Prometheus metrics: what only Janus knows about the node - certificate expiry, which slot it booted, pending upgrades, HAProxy as janusd runs it, extension services, time sync, SELinux. It doesn’t repeat what other exporters already give:

Exporter	Where	What
Janus (this page)	built into every node, `:10056/metrics`	the node as Janus manages it
HAProxy	your HAProxy configuration (below)	frontends, backends, servers, traffic
prometheus-node-exporter	optional extension, `:9100/metrics`	CPU, memory, disks, filesystems, network

Turning it on and off

The exporter is on by default, plain HTTP on port 10056 - the first port free after the Prometheus exporters’ default allocations. Its settings are kept on the node across reboots and upgrades:

janusctl system metrics                  # show
janusctl system metrics -port 10100      # move it
janusctl system metrics -disable         # turn it off
janusctl system metrics -enable

In the Controller: Apps › Janus exporter. A port that can’t be bound is refused, and the exporter stays where it was.

Like node_exporter, it has no authentication: anyone who reaches the port can read the metrics (versions, certificate names and expiry dates, API call counts - no secrets). Restrict who reaches it in your network, or with the node’s firewall.

scrape_configs:
  - job_name: janus
    static_configs:
      - targets: ['node1.example.net:10056', 'node2.example.net:10056']

The node exporter

A node built with the prometheus-node-exporter extension (Images and extensions) also runs Prometheus’s node_exporter: the host’s CPU, memory, disks, filesystems and network. Its settings are kept on the node, like the Janus exporter’s: whether it runs, the address and port it listens on (every address and 9100 by default), and its collectors - from a fixed list, each one known to work on a Janus node. The default ones: cpu, diskstats, filefd, filesystem, loadavg, meminfo, netdev, netstat, os, pressure, sockstat, stat, time, timex, uname, vmstat. More can be turned on: arp, conntrack, cpufreq, dmi, entropy, interrupts, netclass, nvme, softirqs, softnet, thermal_zone, udp_queues.

janusctl system node-exporter                                  # show
janusctl system node-exporter -address 192.0.2.10 -port 9200   # listen elsewhere
janusctl system node-exporter -collectors cpu,meminfo,netdev,softnet
janusctl system node-exporter -collectors default
janusctl system node-exporter -disable                         # stop it, and keep it stopped

In the Controller: Apps › Node exporter. A change restarts node_exporter with it. Like a stock node_exporter, it has no authentication: restrict who reaches the port.

Metrics

Metric	Type	Labels	Meaning
`janus_build_info`	gauge	`version`, `go_version`, `arch`, `schematic`	Always 1: the release and image schematic the node runs
`janus_extension_info`	gauge	`extension`, `version`	Always 1, per extension in the image
`janus_boot_info`	gauge	`slot`, `kernel`	Always 1: the A/B slot booted, and the kernel
`janus_daemon_start_time_seconds`	gauge		When janusd started - it changes when janusd restarts
`janus_upgrade_pending_confirmation`	gauge		1 while an upgrade waits for its health confirmation; the node reverts if it doesn’t come
`janus_certificate_expiry_timestamp_seconds`	gauge	`source`, `certificate`, `cn`	When a certificate expires: the node API’s CA and server certificates (`source="api"`), and every certificate HAProxy has loaded (`source="haproxy"`, by file or store name)
`janus_haproxy_up`	gauge		1 if HAProxy answers on its stats socket - HAProxy’s own metrics can’t report it down
`janus_haproxy_starts_total`	counter		HAProxy processes janusd started, reloads included
`janus_haproxy_reloads_total`	counter		Seamless reloads
`janus_haproxy_unexpected_exits_total`	counter		HAProxy processes that exited without being stopped or replaced - crashes
`janus_haproxy_config_applies_total`	counter	`result` (`accepted`, `rejected`)	Configurations applied through the API
`janus_haproxy_config_last_apply_timestamp_seconds`	gauge		When the last configuration was applied through the API
`janus_service_state`	gauge	`service`, `extension`, `state`	1 for the current state (`running`, `waiting`, `restarting`, `stopped`) of each extension service
`janus_service_restarts_total`	counter	`service`, `extension`	Times an extension service exited and was restarted
`janus_time_synchronized`	gauge		1 once janusd’s NTP client has set the clock
`janus_time_last_sync_timestamp_seconds`	gauge		Last NTP synchronization
`janus_time_offset_seconds`	gauge		The clock offset measured then
`janus_time_stratum`	gauge		The server’s stratum
`janus_network_trial_pending`	gauge		1 while a network configuration is on trial (it reverts unless confirmed)
`janus_network_trial_revert_timestamp_seconds`	gauge		When it reverts, while on trial
`janus_firewall_configured`	gauge		1 if a firewall ruleset is saved (firewall, the nftables extension)
`janus_firewall_trial_pending`	gauge		1 while a firewall ruleset is on trial: it reverts unless confirmed
`janus_firewall_set_elements`	gauge	`family`, `table`, `set`	Elements in each named set of the live ruleset
`janus_vrrp_instance_state`	gauge	`instance`, `interface`, `state`	1 for each VRRP instance’s current state (`MASTER`, `BACKUP`, `FAULT`, `INIT`, `STOP`) - VRRP, the keepalived extension
`janus_vrrp_instance_effective_priority`	gauge	`instance`	Its priority after tracking
`janus_vrrp_instance_became_master_total`	counter	`instance`	Times it became master since keepalived started
`janus_bgp_protocol_up`	gauge	`protocol`, `proto`	1 if each BIRD protocol is up - BGP, the bird extension
`janus_bgp_session_established`	gauge	`protocol`, `neighbor`	1 if each BGP session is established
`janus_bgp_routes`	gauge	`protocol`, `channel`, `direction` (`imported`, `exported`)	Routes each protocol imported and exported
`janus_bgp_protocol_held_down`	gauge	`protocol`	1 while janusd keeps a `haproxy_*` protocol down: HAProxy doesn’t answer
`janus_acme_account_registered`	gauge		1 once the ACME CA knows the account - Let’s Encrypt, the letsencrypt extension
`janus_acme_certificate_state`	gauge	`name`, `state`	1 for each certificate’s current state (`pending`: not obtained yet, `valid`, `due`, `expired`)
`janus_acme_certificate_failures`	gauge	`name`	Failed attempts in a row to obtain it
`janus_acme_certificate_renew_timestamp_seconds`	gauge	`name`	When it becomes due for renewal (0 until obtained)
`janus_acme_certificate_last_success_timestamp_seconds`	gauge	`name`	When it was last obtained (0: never)
`janus_selinux_enforcing`	gauge		1 if SELinux is enforcing
`janus_selinux_denials_total`	counter		SELinux denials in the kernel log since boot - there should be none
`janus_kernel_oom_kills_total`	counter		Processes the kernel killed for lack of memory since boot
`janus_state_filesystem_errors`	gauge		Errors the kernel recorded on STATE (PKI, configuration) - it should be 0
`janus_api_requests_total`	counter	`method`, `code`	Calls to the node’s API since janusd started, refused ones included

A metric that doesn’t apply is absent rather than 0: no NTP synchronization yet, no configuration applied yet, no extension.

Alerts

groups:
  - name: janus
    rules:
      - alert: JanusCertificateExpiresSoon
        expr: janus_certificate_expiry_timestamp_seconds - time() < 14 * 86400
        labels: {severity: warning}
        annotations:
          summary: '{{ $labels.certificate }} ({{ $labels.cn }}) on {{ $labels.instance }} expires in {{ $value | humanizeDuration }}'
      - alert: JanusHAProxyDown
        expr: janus_haproxy_up == 0
        for: 1m
        labels: {severity: critical}
      - alert: JanusHAProxyCrashed
        expr: increase(janus_haproxy_unexpected_exits_total[15m]) > 0
        labels: {severity: warning}
      - alert: JanusExtensionServiceDown
        expr: janus_service_state{state="restarting"} == 1
        for: 5m
        labels: {severity: warning}
      - alert: JanusSELinuxDenials
        expr: increase(janus_selinux_denials_total[1h]) > 0
        labels: {severity: warning}
      - alert: JanusOOMKill
        expr: increase(janus_kernel_oom_kills_total[1h]) > 0
        labels: {severity: warning}
      - alert: JanusStateErrors
        expr: janus_state_filesystem_errors > 0
        labels: {severity: critical}
      - alert: JanusClockNotSynchronized
        expr: janus_time_synchronized == 0 or time() - janus_time_last_sync_timestamp_seconds > 3600
        for: 15m
        labels: {severity: warning}
      - alert: JanusUpgradeUnconfirmed
        expr: janus_upgrade_pending_confirmation == 1
        for: 10m
        labels: {severity: warning}
      - alert: JanusAPIRefusals
        expr: sum by (instance) (increase(janus_api_requests_total{code=~"PermissionDenied|Unauthenticated"}[15m])) > 10
        labels: {severity: warning}
      - alert: JanusVRRPFault
        expr: janus_vrrp_instance_state{state="FAULT"} == 1
        for: 1m
        labels: {severity: critical}
      - alert: JanusBGPSessionDown
        expr: janus_bgp_session_established == 0
        for: 2m
        labels: {severity: critical}
      - alert: JanusACMERenewalFailing
        expr: janus_acme_certificate_failures >= 3
        labels: {severity: warning}
      - alert: JanusACMECertificateNotObtained
        expr: janus_acme_certificate_state{state=~"pending|expired"} == 1
        for: 1h
        labels: {severity: critical}

HAProxy’s own metrics

Janus’s HAProxy is built with its Prometheus exporter. Serve it from any frontend of your configuration:

frontend prometheus
  bind :8405
  http-request use-service prometheus-exporter if { path /metrics }
  no log