Logs and metrics

reported.json

After every pass, the daemon writes reported.json in paths.state_dir — /var/lib/nibrunner/reported.json on a host the installer configured. It is the state of every volume, instance, checkpoint and export the document names, and why, when the why is not obvious: an instance that cannot start, a document the host refused, an export that failed.

jq '.instances[] | {appId, state, message}' /var/lib/nibrunner/reported.json

The reference is every field.

Logs

Each app's output goes to <paths.state_dir>/logs/<appId>.log, one line per line the program wrote:

2026-09-16T10:41:07.318Z stdout 6f1c…/42 listening on :3000

That is when the host saw it, which stream, an id for the microVM's log session and the line's sequence in it, then the line. A gap where the host could not keep up says how many bytes it dropped, and a restart of the program inside the guest is a line on stderr saying why.

The host keeps the newest 256 MiB of each app, in two files: when <appId>.log passes the cap it becomes <appId>.log.1, over the one before it, and a fresh <appId>.log starts. tail -F follows across the cut. An app that floods fills its own share of the disk and no more of it, and [logs] in config.toml is the share:

[logs]
keep_mib_per_app = 256

The daemon's own log is in the journal:

journalctl -u nibrunnerd -f

NIBRUNNER_LOG in the daemon's environment is a tracing filter — debug, or nibrunnerd::domain=trace — for one restart's worth of detail.

Metrics

Add [metrics] to config.toml and the host serves a Prometheus page at /metrics:

[metrics]
listen_address = "127.0.0.1"
port = 9100

The page is rendered from the same state reported.json is written from, so a scraper and the file cannot disagree. Every series is prefixed nibrunner_: the state of each instance, volume, checkpoint and export as a one-hot; what a converge, a sleep and a wake took, phase by phase; the proxy's requests by what it answered with — an app's own answer is served, whatever its status, and the 502 the proxy writes when an app does not answer at all is unreachable; each app's requests by status class and the proxy's TLS handshakes; each app's CPU, memory, disk and network; the host's slots, memory and capacity.

The one to alert on is nibrunner_conntrack_entries{of="used"} against {of="max"}: the kernel's connection-tracking table, which every flow between the proxy and a guest and every flow a guest opens is an entry of, for its lifetime and 120 s after. Full, the kernel drops packets for every app on the host and says so only in dmesg. The daemon says it in its log at four fifths, once, and once more when it is back under; max_apps sizes the table.