Development docs, for main @ 60ec1d6 - what's coming, not yet released. This page for v2026.11.13, the latest release →

Log forwarding

A node sends its logs to your log servers - syslog (RFC 5424 or 3164, over UDP, TCP or TLS) and Elasticsearch - itself: janusd’s log, HAProxy’s messages and access logs, the extensions’, the kernel’s, the event log. What it keeps on disk is sent at least once, even across a server outage or a reboot.

There’s no syslog daemon to configure on a node, and no agent to install: janusd sends the logs (internal/logship), set through the API - the node’s page, janusctl system log-forwarding, the SystemService.LogForwarding* calls (API reference) - or for a fleet by the Controller, which sends its own log and audit too (below).

What’s sent

Each target - a log server - takes the sources it names:

Source	What	Kept on the node
`janusd`	janusd’s log: what the node does, every API call that changes something (`api: <Method>: <caller>`)	LOGS partition
`haproxy`	HAProxy’s own messages: `[NOTICE]`, `[WARNING]` (a server down), `[ALERT]`	LOGS partition
`access`	HAProxy’s access logs - its global section’s `log stdout`	memory, or the LOGS partition with `access_logs_on_disk`
`extensions`	The extensions’ services: keepalived, BIRD, Consul, node_exporter, the guest agent, Let’s Encrypt	LOGS partition
`kernel`	The kernel’s messages: AVC denials, OOM kills, devices	LOGS partition
`events`	The event log: configuration applied, reboots, updates, failovers…	STATE
`audit`	Every call that changed something or read what a reader may not - who, from where, how it ended - and every call refused (who did what)	STATE

A target that names no source takes them all but access: HAProxy logs a line per request, thousands a second under load - ask for them.

Setting it up

A configuration lists the targets:

{
  "labels": {"env": "prod"},
  "targets": [
    {
      "name": "nas",
      "syslog": {"address": "logs.example.com:6514", "transport": "tls"},
      "tls": {
        "fingerprint": "3F:1C:9A:52:07:E4:B8:6D:21:F0:A3:94:5C:7E:18:B2:60:D9:4F:A7:13:8E:C5:02:6B:F1:39:DA:84:57:2E:C0"
      }
    },
    {
      "name": "access-logs",
      "sources": ["access"],
      "syslog": {"address": "10.0.0.20", "transport": "udp", "format": "rfc3164", "facility": "local1"}
    }
  ]
}

janusctl -n lb1 system log-forwarding set syslog.json   # applied at once, kept on STATE
janusctl -n lb1 system log-forwarding status            # each target: sent, bytes, dropped, waiting, its interface, the last error
janusctl -n lb1 system log-forwarding test nas          # one record now - or why it can't
janusctl -n lb1 system log-forwarding get               # the configuration, without its secrets

A new target starts with what’s logged from then on. A target’s name is where it keeps its place: rename it, and it starts again from now.

Syslog

Field	Values
`address`	`host[:port]` - 514 by default, 6514 with `tls`
`transport`	`udp` (the default), `tcp`, `tls`
`format`	`rfc5424` (the default) or `rfc3164` (BSD)
`framing`	over `tcp` and `tls`: `octet-counting` (RFC 6587, the default with `rfc5424`) or `lf` (a message a line, the default with `rfc3164`)
`facility`	the services’ lines: `local0` (the default) to `local7`, `daemon`, `user`, `auth`. The kernel’s messages are `kern`, the audit’s `authpriv`

A message carries the node’s hostname, the service as its app name (janusd, haproxy, keepalived, kernel…), the source as its message ID (RFC 5424), and the severity the line says - HAProxy’s [WARNING] is a warning, Consul’s [ERROR] an error, an api: line a notice, the rest info.

A member of a cluster says which in RFC 5424’s structured data - one element, its parameters named like the janus_cluster_info metric’s labels:

<134>1 2026-10-26T09:00:00.000000Z edge-1 janusd - janusd [janus@32473 cluster_id="6f1c2a9e-3b4d-4e5f-8a7b-0c1d2e3f4a5b" cluster_name="edge" member="edge-1"] cluster: ...

A node in no cluster sends none (-). rsyslog reads the element with mmpstrucdata ($!rfc5424-sd!janus@32473!cluster_name). 32473 is the enterprise number IANA keeps for examples (RFC 5612): Janus has none of its own yet - the ID is provisional. RFC 3164 has no structured data: its messages don’t say the cluster - in the tag or the message it would change what servers filter and parse on, for members only. Use rfc5424 or Elasticsearch to tell clusters apart.

A Synology NAS receives them in Log Center › Log Receiving: a rule in IETF format (rfc5424) or BSD (rfc3164), over UDP, or TCP with SSL - transport tls, the port the rule says, and the certificate Log Center exports as tls.ca_pem (or its fingerprint).

Elasticsearch

{
  "targets": [
    {
      "name": "elastic",
      "sources": ["janusd", "haproxy", "access", "extensions", "kernel", "events", "audit"],
      "elasticsearch": {
        "url": "https://elastic.example.com:9200",
        "namespace": "prod",
        "apiKey": "VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw==",
        "gzip": true
      },
      "tls": {
        "fingerprint": "93:BA:4A:A9:69:30:C1:B5:0D:DF:89:AD:FC:CA:5A:88:5D:D4:14:62:B9:2A:0F:1F:DF:FB:D9:60:FF:22:56:D9"
      }
    }
  ]
}

Each record is a document of the data stream logs-janus.<source>-<namespace> - logs-janus.janusd-prod, logs-janus.access-prod… - which Elasticsearch 8 and 9 make on their own (their built-in logs-*-* template). On the cluster, all it takes is an API key that may create documents there - in Kibana’s Dev Tools:

POST /_security/api_key
{
  "name": "janus-nodes",
  "role_descriptors": {
    "janus": {"index": [{"names": ["logs-janus*"], "privileges": ["create_doc", "auto_configure"]}]}
  }
}

Its encoded value is the target’s api_key (id:key works too); a user and password instead: username and password. A cluster whose HTTPS certificate Elasticsearch made at its first start is trusted by the fingerprint it printed then (“HTTP CA certificate SHA-256 fingerprint”), or openssl x509 -noout -fingerprint -sha256 -in config/certs/http_ca.crt. gzip sends a fifth of the bytes for some CPU.

The documents have ECS fields: @timestamp, message, log.level, host.name and host.id (the node’s CA SHA-256: the console’s pki: this node's CA line), service.name, event.dataset, an event’s event.action (its type) and janus.payload, an audit entry’s user.name, user.roles, source.ip, event.action (the method), event.outcome and event.duration, labels.*, janus.slot and janus.schematic. In Kibana, a data view on logs-janus* shows them all.

A member of a cluster adds ECS’s fields for one, orchestrator.cluster.id, orchestrator.cluster.name and orchestrator.type (janus), and its member name as janus.cluster.member - ECS has no field for it. orchestrator.cluster.name : "edge" in Kibana shows every member’s logs at once; a node in no cluster has none of these fields.

The cluster, in a document or a syslog message, is the one the node is in when the record is sent - like its hostname: a line logged before it joined but sent after (its target was down meanwhile) says the cluster. A document’s ID doesn’t depend on it.

A batch sent again - a timeout, the cluster busy (429) - is stored once: each document’s ID is the node’s and the record’s, the copies answered 409. A document the cluster refuses for good (its mapping) is dropped, said once a minute in janusd’s log.

HAProxy’s access logs formatted as JSON become fields - haproxy.status, haproxy.path… - with no ingest pipeline:

frontend web
    bind :80
    log-format "%{+json}o %(client_ip)ci %(method)HM %(path)HP %(status)ST %(bytes)B %(duration)Ta"

Trusting the server

A tls target checks the server’s certificate:

by default, against the node’s trust store - the usual public CAs;
tls.ca_pem: against your CA (PEM), the one that signed the server’s certificate;
tls.fingerprint: the SHA-256 fingerprint of the server’s certificate, or of a CA in the chain it presents - colons or not; a wrong one is refused with the server’s own. Many servers (Logstash, rsyslog) present their own certificate alone: pin that one, or give the CA as tls.ca_pem;
tls.server_name: the name the certificate carries, when it isn’t the address’s;
tls.client_cert_pem and tls.client_key_pem: a client certificate, for a server that asks for one.

The secrets - an API key, a password, a client key - are kept on STATE, readable by janusd only: they never come back through the API (log-forwarding get shows a target without them, and status names those it has), nor in a Controller’s backups. A configuration sent without a target’s secret keeps the one saved under its name.

Two ways for HAProxy’s access logs

Through janusd (the access source): HAProxy’s global section logs to its stdout - log stdout format raw local0, what Janus’s examples do - and janusd sends each line to the targets that take access, with the node’s metadata. A target that can’t keep up loses them - counted - from its queue in memory (20 000 lines), unless access_logs_on_disk is always: then they’re kept on the LOGS partition too and sent at least once, at the cost of writing every request to the disk.

From HAProxy itself: its own log lines, without janusd -

# HAProxy sending its own access logs, without janusd: every request
# straight to a syslog server over UDP, and through a ring to another
# one over TLS. docs/guide/log-forwarding.md says when to prefer this to
# janusd's log forwarding (no Elasticsearch, no Janus metadata, nothing
# kept on the node; the least CPU under heavy load).
global
    # UDP: one datagram a request, lost when the server is down.
    log 192.0.2.10:514 local0
    # TLS: kept in the ring while the server is away (its size).
    log ring@remote-logs local0
    stats socket /run/janus/haproxy-admin.sock mode 660 level admin
    chroot /var/empty
    uid 1000
    gid 1000

ring remote-logs
    format rfc5424
    size 1048576
    timeout connect 5s
    timeout server 10s
    # The server's certificate checked against the system's CAs - or a
    # CA of yours put in HAProxy's files (janusctl haproxy file-put
    # certs/logs-ca.pem ca.pem): ca-file /etc/haproxy/files/certs/logs-ca.pem
    server logs 192.0.2.11:6514 ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt verifyhost logs.example.com log-proto octet-count

defaults
    mode http
    log global
    option httplog
    timeout connect 5s
    timeout client 30s
    timeout server 30s

frontend web
    bind :80
    http-request return status 200 content-type text/plain string "ok\n"

	Through janusd	From HAProxy
Elasticsearch	yes	no
The node’s metadata, labels	yes	HAProxy’s hostname only
A server down	a queue in memory per target, or the LOGS partition	UDP: lost; a ring: its size
CPU	HAProxy writes to a pipe, janusd reads, formats and sends	HAProxy alone
Also in `janusctl system logs haproxy`	yes	only with `log stdout` too

What it costs

Measured on the worst case: a 2-vCPU node of an older Xeon (E3-1220 v3) where HAProxy answers every request itself, saturated by four clients for a minute - so a log line costs as much as the request it logs. Behind it, a single-CPU Logstash and a 2-CPU Elasticsearch.

HAProxy’s access logs	Requests/s	janusd’s CPU	Lines HAProxy dropped	What the server kept
None	124 900	-	-	-
`log stdout`, sent nowhere	64 600	0.27	8.6 %	-
`log stdout`, janusd to syslog over UDP	57 900	0.62	4.7 %	3.6 % (Logstash full)
`log stdout`, janusd to Elasticsearch	60 200	0.50	11 %	~6 000 a second, the rest dropped by janusd, counted
The same, `access_logs_on_disk: always`	64 000	0.48	8.8 %	all of them, 7 minutes after the minute (3.5 million)
HAProxy’s own `log` over UDP	72 200	-	0	5.4 % (Logstash full)
HAProxy’s own `log` through a `ring`, over TLS	98 600	-	0	5.2 % (the ring wrote over the rest)

What it says:

Logging every request is HAProxy’s own biggest cost, whichever way: it formats and writes a line per request - half the requests/s of a node that logs nothing, here. With backends behind it, the share is far smaller. Fewer lines cost less: option dontlog-normal logs only the errors, log ... sample 1:100 one request in a hundred.
janusd reads them for about 5 µs of CPU a line on that Xeon, sends them to syslog for about 6 µs more, to Elasticsearch in compressed batches.
The server caps what arrives: a server that can’t keep up loses UDP datagrams on its side; janusd drops what an Elasticsearch cluster can’t take - counted (dropped). HAProxy never slows down for them.
access_logs_on_disk: always writes every line to the LOGS partition - 10 MB/s here, 860 GB a day - and sends them from there at the server’s pace: Elasticsearch had every one of the minute’s 3.5 million lines 7 minutes after it. But the partition (512 MiB) holds about a minute of such traffic - a longer burst loses the oldest, counted - and it wears the disk. Keep it for a moderate traffic you can’t afford to lose.
HAProxy drops lines itself rather than wait: when its stdout is full (janusd behind), and when two of its threads write to its stdout at once - it gives up after a few tries: about 1 % of the lines at 50 000 requests/s on a recent CPU, 8 % on that Xeon. Its own log to a UDP server shares no pipe: none dropped. janusctl haproxy show-info says how many it dropped (Dropped logs), and the metrics too (janus_haproxy_dropped_logs_total). A ring costs HAProxy the least - a copy in memory - but writes over its oldest lines, uncounted, when its server can’t keep up.

The console

What a service prints goes three ways: janusd’s own copy (janusctl system logs haproxy, the Logs page), the LOGS partition and the log targets - all of it, every line; and the node’s console, a share of it: 2 KiB a second with a 64 KiB burst, a few lines, the rest left out and said so every ten seconds (N lines of haproxy's output not shown on the console - janusctl system logs haproxy). janusd’s own log always reaches the console: it is the boot’s record, and it is little.

That share has a cost. The console is a serial port the hypervisor emulates, and every byte written to it is a round trip through the hypervisor; init, which relays what janusd prints to it, pays for it in CPU. On a node that logs every request (log stdout), the budget is always full: init takes about 2 % of a 2013 Xeon core, 0.8 % of a single 2012 core, 0.4 % of a 2023 one - the same bytes, slower cores. Nothing of it reaches HAProxy; it is worth knowing where it comes from.

A node that wants less of its console, or none of it - a node with a serial console nobody reads - sets its budget:

janusctl system console-budget                 # what it is
janusctl system console-budget -rate 512       # 512 bytes a second
janusctl system console-budget -rate 0         # nothing of the services' output
janusctl system console-budget -default        # 2 KiB/s, 64 KiB again

Or the Controller’s System › Services, the Console card. The budget is kept on STATE (config/console.json) across reboots and updates and applies at once; a lower one takes hold within a second, what was already in hand being capped. 0 keeps janusd’s own log on the console and nothing else; every line of the services is still kept whole where it was - this changes the console alone. What’s left out is counted, by service (janus_console_lines_left_out_total), and the budget is exported (janus_console_budget_bytes_per_second, Metrics).

What’s guaranteed

From disk, at least once: what the LOGS partition and STATE keep - janusd’s log, HAProxy’s messages, the extensions’, the kernel’s, the event log, the audit - each target reads behind a cursor saved as it goes. A server down, then back: what was logged meanwhile goes then, in order. A reboot, janusd restarted: each target goes on from its cursor. As long as the partition still holds it: the oldest records go when it’s full - a target that falls that far behind counts the ones it never read as dropped, whatever their source.
From memory: HAProxy’s access logs (unless access_logs_on_disk is always), and everything on a node installed before the LOGS partition existed - a queue per target, the oldest dropped when it’s full, counted (dropped, janus_logship_records_dropped_total). What’s logged as janusd starts, before it reads its configuration - the kernel’s boot, its first lines - waits for it (8 192 lines), then goes under the node’s hostname.
UDP has no acknowledgement: a server down loses what’s sent. Over TCP and TLS, syslog has none either - a server that goes away in the middle of a batch loses it; one that went away between two is noticed, and the batch sent again once it’s back. Elasticsearch confirms each document: a batch is sent again until it’s stored, and stored once.
Nothing waits for a target: a line is queued, written to disk or dropped - HAProxy, janusd and the API never slow down for a log server.

Pausing a target

A paused target gets nothing and keeps its place: the server down for maintenance, a noisy target cut while you look into something - without removing it and typing it again later, secrets included.

janusctl -n lb1 system log-forwarding pause nas    # nothing goes to nas from now on
janusctl -n lb1 system log-forwarding status       # nas: disabled
janusctl -n lb1 system log-forwarding resume nas   # from where it stopped

On a node’s page (Logs › Forwarding) and the Controller’s Log forwarding page, Pause and Resume are on the target’s row, beside Edit and Remove - and like them, they apply once the page is saved. In a configuration, a paused target is "disabled": true; its secrets stay saved.

Resumed, a target gets what the node kept for it while it was paused:

What the LOGS partition holds - janusd’s log, HAProxy’s messages, the extensions’, the kernel’s, and the access logs with access_logs_on_disk: always - goes from where the target stopped, in order, a reboot in between or not. As long as the partition still holds it: a pause longer than the partition lasts loses the oldest, counted as dropped once the target is resumed.
The event log and the audit (STATE) go from where it stopped, within what they keep: the last 1 000 events, the last 2 000 audit entries - a pause through more loses the oldest, uncounted.
What waits in memory isn’t kept for a paused target: the access logs with access_logs_on_disk: never (the default), and, on a node without a LOGS partition, everything but the event log and the audit. The other targets get it; the paused one never will.

While paused, status says disabled and counts nothing - its counts start again once it’s resumed. The Controller’s own targets pause the same way: its audit is sent from where a target stopped, its log (memory) isn’t kept for it.

A node whose configuration a fleet profile set (below) isn’t paused on its own: its page is read-only, janusctl ... pause refuses, and the Controller would put the target back within ten minutes. Pause the target on the profile: it pauses on every node the profile picks.

From the Controller

The Controller’s Log forwarding page (admins) sets two things:

The Controller’s own logs: its log and its audit - every change made through it, every sign-in - to its own targets, its audit from where it stopped after a restart. An audit entry about something names it as it was then: , node edge-1 at the end of the message, janus.audit.subject.kind, .id and .name in Elasticsearch.
Fleet profiles: each a label selector (team=web,env=prod; empty: every node) and the log targets of the nodes it picks. A node takes the first profile whose labels it has - its own, and those it inherits from its hypervisor (Hypervisors) -, and the Controller keeps it applied: at once when a profile, a node’s labels or its hypervisor’s change, every ten minutes otherwise - every minute while a node can’t take it yet (an older release being updated, a node away) - a change made on the node is undone, a profile that no longer picks a node is taken off it. A node no profile picks keeps its own configuration, set on its page (Logs › Forwarding) or with janusctl. The page lists each node’s state: applied, unmanaged, removed, unsupported (a release without log forwarding).

A node’s page shows what the Controller set as read-only, and its configuration says by whom (managed_by: the Controller and the profile). The secrets - API keys, passwords, client keys - are sealed with the Controller’s master key and sent to the nodes the profile picks; neither the page nor the API gives them back. A target is tested as the page shows it, saved or not, from the Controller (its own) or from one of the profile’s nodes; what’s added, changed, paused or removed applies once the page is saved - it says so until then.

The Controller in a container keeps writing its log on its standard error too: docker logs and Docker’s own logging drivers still see it.

Watching it

janusctl system log-forwarding status - each target’s state (ok, failing), records sent and the bytes they took (syslog messages with their framing, Elasticsearch request bodies as sent - gzipped when the target says so), dropped, waiting, the interface its logs leave by (VIA: the kernel’s route to the server the target is connected to, from the address it’s connected from - so a policy route or a VRRP address are what the kernel says, not a guess), the last error. A target failing for 30 seconds is said so in the event log (logship.failing, then logship.recovered) and janusd’s log, and in the metrics: janus_logship_target_up, janus_logship_records_dropped_total, janus_logship_bytes_sent_total; the Grafana dashboard’s Kernel and STATE tile counts the targets failing and the records lost. The node’s Traffic flow (the Controller’s Monitoring › Traffic flow, janusctl tui’s f) draws each target beyond the interface it goes by, with its records and bytes a second.