Guides

Logs and metrics

reported.json, per-app logs, and the Prometheus endpoint.

reported.json

After every reconcile pass, the daemon writes reported.json to paths.state_dir (/var/lib/nibrunner/reported.json on a host set up by the installer). It holds the state of every volume, instance, checkpoint and export in desired.json, with a reason where one is needed: an instance that cannot start, a file the host rejected, an export that failed. It also holds what the host has and what is left of it, which is what decides whether another app fits.

Instances carry their latest guest CPU and memory readings as compute; volumes carry their filesystem readings as usage. Both keys are required and are null when no measurement is available. Available readings keep their original measuredAt timestamp. Cumulative usage remains on the metrics endpoint below.

jq '.instances[] | {appId, state, message}' /var/lib/nibrunner/reported.json

The reference lists every field.

cron lists the registered jobs for each current deployment, using the same job IDs and time zone as the scheduler. Sleeping apps keep their next run times. Stopped apps keep their jobs with enabled: false and nextRunAt: null. Every job includes nextRunAt, which is null when no next run is scheduled. Reading these registrations never wakes a guest. The cron key is required: null means the registry could not be read, while [] means there are no registrations.

Logs

Each app's output goes to <paths.state_dir>/logs/<appId>.jsonl, one JSON record per event:

{"appId":"app-1","deploymentId":"dep-1","sourceId":"session-1","sequence":42,"observedAt":"2026-09-16T10:41:07.318Z","kind":"data","stream":"stdout","text":"listening on :3000\n"}

Each record preserves the app, deployment, source, sequence and observation time. Its kind identifies output, cron output, a buffering gap or a tenant restart. Output text preserves whitespace and LF or CRLF line endings inside the JSON string. Partial output is emitted without adding a newline. The tenant log reference lists every field from the generated schema. Clients parse these records to display or format the output.

The host keeps the newest 256 MiB per app, in two files. When <appId>.jsonl reaches the limit it is renamed to <appId>.jsonl.1, replacing the previous one, and a new <appId>.jsonl starts. tail -F follows across the rotation. An app that logs too much fills only its own share of the disk. [logs] in config.toml sets that share:

[logs]
keep_mib_per_app = 256

Both files go when desired.json stops naming the app. An app that is stopped, asleep, or being replaced by a new deploymentId is still named, so it keeps what it has written. Take a copy of a log you want to read after the app is gone.

The HTTP proxy also writes one JSON line per routed request to <paths.state_dir>/logs/<appId>.http.jsonl. Each line has at, method, path, status, durationMicros and deploymentId. The path excludes its query string; headers and request bodies are not kept. at is when the proxy had the response headers; durationMicros measures the wait for those headers, including a cold wake, rather than the transfer of the response body. The proxy keeps at most 1 MiB in this file and 1 MiB in .http.jsonl.1 for each app. When proxy traffic outruns the bounded writer queue, it drops HTTP request records and warns in the daemon journal. An app's HTTP request logs go when desired.json stops naming it.

The daemon's own log is in the journal:

journalctl -u nibrunnerd -f

NIBRUNNER_LOG in the daemon's environment is a tracing filter, such as debug or nibrunnerd::domain=trace. Set it for one restart when you need more detail.

Two libraries the daemon uses are held below that: backhand, which narrates every filesystem layer it packs, and rustls, which warns about clients that put an IP address where a hostname belongs. Errors from both still reach the journal, and naming either in NIBRUNNER_LOG — backhand=info — brings the rest back.

Metrics

A host set up by the installer serves Prometheus metrics at /metrics on loopback. The [metrics] section of config.toml sets where:

[metrics]
listen_address = "127.0.0.1"
port = 9100

The endpoint also publishes cumulative usage. Every series is prefixed nibrunner_, and what follows names what is being measured: app_ for one app, host_ for the machine, volume_, checkpoint_ and export_ for what it holds, proxy_ for the listener it is reached through, and vm_ for an operation on a microVM. Anything labelled app is about one app and nothing else, which is what to filter on for a per-app dashboard. They cover:

  • The state of each instance, volume, checkpoint and export, one-hot encoded.
  • How long a converge, a sleep and a wake took, by phase.
  • The proxy's requests by outcome. served is any answer the app gave, whatever its status. unreachable is the 502 the proxy writes when the app did not answer at all. redirected counts fallback-hostname redirects, without charging those requests to an app.
  • Each app's requests by status class, and the proxy's TLS handshakes.
  • Each app's CPU, memory, disk and network, both what it is using now and what it has used.
  • The host's slots, memory and capacity.

Bill from the totals, not from a rate

nibrunner_app_cpu_seconds_total, _network_bytes_total, _disk_mib_seconds_total and _time_seconds_total only ever go up. What a period cost is the figure at its end less the figure at its start, so a scrape nobody took is resolution nobody has rather than usage nobody billed. A counter that goes backwards is a host that was reinstalled.

Alert on conntrack

Alert on nibrunner_conntrack_entries{of="used"} approaching {of="max"}. This is the kernel's connection-tracking table. Every flow between the proxy and a guest, and every flow a guest opens, is an entry in it for the flow's lifetime plus 120 seconds. When the table is full, the kernel drops packets for every app on the host and reports it only in dmesg. The daemon logs a warning once when usage reaches 80%, and once more when it drops back below. max_apps sizes the table.

Port restrictions

metrics.port cannot be proxy.http.port, a port in the slot range starting at 21000, or 9091 on a zerofs host, which ZeroFS uses.

On this page