Logs and metrics
reported.json, per-app logs, and the Prometheus endpoint.
reported.json
After every reconcile pass, the daemon writes reported.json to paths.state_dir
(/var/lib/nibrunner/reported.json on a host set up by the installer). It holds the state of
every volume, instance, checkpoint and export in desired.json, with a reason where one is
needed: an instance that cannot start, a file the host rejected, an export that failed. It also
holds what the host has and what is left of it, which is what decides whether another app fits.
Instances carry their latest guest CPU and memory readings as compute; volumes carry their
filesystem readings as usage. Both keys are required and are null when no measurement is
available. Available readings keep their original measuredAt timestamp. Cumulative usage remains
on the metrics endpoint below.
jq '.instances[] | {appId, state, message}' /var/lib/nibrunner/reported.jsonThe reference lists every field.
cron lists the registered jobs for each current deployment, using the same job IDs and time
zone as the scheduler. Sleeping apps keep their next run times. Stopped apps keep their jobs
with enabled: false and nextRunAt: null. Every job includes nextRunAt, which is null
when no next run is scheduled. Reading these registrations never wakes a guest.
The cron key is required: null means the registry could not be read, while [] means
there are no registrations.
Logs
Each app's output goes to <paths.state_dir>/logs/<appId>.jsonl, one JSON record per event:
{"appId":"app-1","deploymentId":"dep-1","sourceId":"session-1","sequence":42,"observedAt":"2026-09-16T10:41:07.318Z","kind":"data","stream":"stdout","text":"listening on :3000\n"}Each record preserves the app, deployment, source, sequence and observation time. Its kind
identifies output, cron output, a buffering gap or a tenant restart. Output text preserves
whitespace and LF or CRLF line endings inside the JSON string. Partial output is emitted
without adding a newline. The tenant log reference lists every field
from the generated schema. Clients parse these records to display or format the output.
The host keeps the newest 256 MiB per app, in two files. When <appId>.jsonl reaches the limit it
is renamed to <appId>.jsonl.1, replacing the previous one, and a new <appId>.jsonl starts.
tail -F follows across the rotation. An app that logs too much fills only its own share of the
disk. [logs] in config.toml sets that share:
[logs]
keep_mib_per_app = 256Both files go when desired.json stops naming the app. An app that is stopped, asleep, or being
replaced by a new deploymentId is still named, so it keeps what it has written. Take a copy of
a log you want to read after the app is gone.
The HTTP proxy also writes one JSON line per routed request to
<paths.state_dir>/logs/<appId>.http.jsonl. Each line has at, method, path, status,
durationMicros and deploymentId. The path excludes its query string; headers and request
bodies are not kept. at is when the proxy had the response headers;
durationMicros measures the wait for those headers, including a cold wake, rather than the
transfer of the response body.
The proxy keeps at most 1 MiB in this file and 1 MiB in .http.jsonl.1 for each app. When
proxy traffic outruns the bounded writer queue, it drops HTTP request records and warns in the daemon
journal. An app's HTTP request logs go when desired.json stops naming it.
The daemon's own log is in the journal:
journalctl -u nibrunnerd -fNIBRUNNER_LOG in the daemon's environment is a tracing filter, such as debug or
nibrunnerd::domain=trace. Set it for one restart when you need more detail.
Two libraries the daemon uses are held below that: backhand, which narrates every filesystem
layer it packs, and rustls, which warns about clients that put an IP address where a hostname
belongs. Errors from both still reach the journal, and naming either in NIBRUNNER_LOG —
backhand=info — brings the rest back.
Metrics
A host set up by the installer serves Prometheus metrics at /metrics on loopback. The
[metrics] section of config.toml sets where:
[metrics]
listen_address = "127.0.0.1"
port = 9100The endpoint also publishes cumulative usage. Every series is prefixed nibrunner_, and
what follows names what is being measured: app_ for one app, host_ for the machine,
volume_, checkpoint_ and export_ for what it holds, proxy_ for the listener it is
reached through, and vm_ for an operation on a microVM. Anything labelled app is about one
app and nothing else, which is what to filter on for a per-app dashboard. They cover:
- The state of each instance, volume, checkpoint and export, one-hot encoded.
- How long a converge, a sleep and a wake took, by phase.
- The proxy's requests by outcome.
servedis any answer the app gave, whatever its status.unreachableis the 502 the proxy writes when the app did not answer at all.redirectedcounts fallback-hostname redirects, without charging those requests to an app. - Each app's requests by status class, and the proxy's TLS handshakes.
- Each app's CPU, memory, disk and network, both what it is using now and what it has used.
- The host's slots, memory and capacity.
Bill from the totals, not from a rate
nibrunner_app_cpu_seconds_total, _network_bytes_total, _disk_mib_seconds_total and
_time_seconds_total only ever go up. What a period cost is the figure at its end less the
figure at its start, so a scrape nobody took is resolution nobody has rather than usage nobody
billed. A counter that goes backwards is a host that was reinstalled.
Alert on conntrack
Alert on nibrunner_conntrack_entries{of="used"} approaching {of="max"}. This is the kernel's
connection-tracking table. Every flow between the proxy and a guest, and every flow a guest
opens, is an entry in it for the flow's lifetime plus 120 seconds. When the table is full, the
kernel drops packets for every app on the host and reports it only in dmesg. The daemon logs
a warning once when usage reaches 80%, and once more when it drops back below.
max_apps sizes the table.
Port restrictions
metrics.port cannot be proxy.http.port, a port in the slot range starting at 21000, or 9091
on a zerofs host, which ZeroFS uses.