# Metrics (/docs/reference/metrics)



What the host serves at `/metrics`, from the `[metrics]` section of `config.toml`. Every series is
prefixed `nibrunner_`, and what follows names what is being measured. Anything labelled `app` is
about one app and nothing else, which is what a per-app dashboard filters on.

A `histogram` publishes three series the table does not list one by one: `_bucket` with an extra
`le` label, `_sum` and `_count`.

## Per app [#per-app]

| Series                                         | Type                          | Labels                         | What it says                                                                                                                                                                                                                |
| ---------------------------------------------- | ----------------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `nibrunner_app_converged`                      | <MetricKind kind="gauge" />   | `app`                          | 1 while an app is what its document asks for, 0 while it is not.                                                                                                                                                            |
| `nibrunner_app_converging_seconds`             | <MetricKind kind="gauge" />   | `app`                          | How long an app has been on its way to what its document asks for. 0 once it got there.                                                                                                                                     |
| `nibrunner_app_cpu_seconds_total`              | <MetricKind kind="counter" /> | `app`                          | What an app's guest has reported spending, summed across the vCPUs it was given.                                                                                                                                            |
| `nibrunner_app_cpu_share`                      | <MetricKind kind="gauge" />   | `app`                          | The share of one vCPU a guest was using, when it was last measured.                                                                                                                                                         |
| `nibrunner_app_disk_mib_seconds_total`         | <MetricKind kind="counter" /> | `app`, `disk`                  | What an app has held on disk over time: what was set aside for it, and what its guest reported filling.                                                                                                                     |
| `nibrunner_app_failures_total`                 | <MetricKind kind="counter" /> | `app`, `reason`                | Times an app failed, by where on the way up it did: refused by the document, no slot, its layers, its volume, the boot, the microVM stopping without being asked, nothing answering inside it, or its restarts running out. |
| `nibrunner_app_health_probes_total`            | <MetricKind kind="counter" /> | `app`, `kind`, `result`        | Probes of an app, by the kind of check and what it found. For a boot-completed check, a probe is the microVM being up.                                                                                                      |
| `nibrunner_app_last_converge_seconds`          | <MetricKind kind="gauge" />   | `app`, `phase`                 | What the last ask for an app took, by the stretch of the work. Absent while one is under way.                                                                                                                               |
| `nibrunner_app_last_healthy_timestamp_seconds` | <MetricKind kind="gauge" />   | `app`                          | When an app last answered a probe, as seconds since the epoch. 0 for one that never has.                                                                                                                                    |
| `nibrunner_app_last_sleep_seconds`             | <MetricKind kind="gauge" />   | `app`                          | What the last sleep of an on-request app took, flush and snapshot. Absent until it has slept.                                                                                                                               |
| `nibrunner_app_last_wake_seconds`              | <MetricKind kind="gauge" />   | `app`                          | What the last wake of an on-request app took. Absent until it has been woken.                                                                                                                                               |
| `nibrunner_app_measured_timestamp_seconds`     | <MetricKind kind="gauge" />   | `app`                          | When a guest last reported what it was using, as seconds since the epoch. 0 for one that never has; one that stopped is one this host can no longer hear.                                                                   |
| `nibrunner_app_memory_used_bytes`              | <MetricKind kind="gauge" />   | `app`                          | What a guest reported using, when it was last measured.                                                                                                                                                                     |
| `nibrunner_app_network_bytes_total`            | <MetricKind kind="counter" /> | `app`, `direction`             | What has crossed an app's tap, by which way it went. Only what was let out is counted as sent.                                                                                                                              |
| `nibrunner_app_request_duration_seconds`       | <MetricKind kind="summary" /> | `app`                          | What requests to one app cost this proxy. Divide the rate of the sum by the rate of the count for that app's mean.                                                                                                          |
| `nibrunner_app_requests_open`                  | <MetricKind kind="gauge" />   | `app`                          | Requests routed to an app that are still being answered: from arrival until the last byte of the response body, or until a websocket closes. An app with one open is not quiet.                                             |
| `nibrunner_app_requests_total`                 | <MetricKind kind="counter" /> | `app`, `class`                 | Requests routed to an app, by the class of the status that came back. A 502 for an app that could not be reached is the proxy's and counted apart below.                                                                    |
| `nibrunner_app_requests_unreachable_total`     | <MetricKind kind="counter" /> | `app`                          | Requests routed to an app that nothing answered, so the proxy answered 502 for it.                                                                                                                                          |
| `nibrunner_app_restarts_total`                 | <MetricKind kind="counter" /> | `app`                          | Times an app's tenant has been restarted inside its guest since the host last booted the app afresh.                                                                                                                        |
| `nibrunner_app_sleeps_total`                   | <MetricKind kind="counter" /> | `app`, `outcome`               | Times a policy said an on-request app may sleep, by what came of it.                                                                                                                                                        |
| `nibrunner_app_start_attempts`                 | <MetricKind kind="gauge" />   | `app`                          | Starts attempted against an app's restart budget in its current window. Resets when a start has held for the policy's resetAfterMs.                                                                                         |
| `nibrunner_app_state`                          | <MetricKind kind="gauge" />   | `app`, `state`                 | 1 for the state an app is in, 0 for every state it is not.                                                                                                                                                                  |
| `nibrunner_app_time_seconds_total`             | <MetricKind kind="counter" /> | `app`, `holding`               | How long an app has been held, by what it was holding: memory while it runs, a snapshot on disk while it sleeps.                                                                                                            |
| `nibrunner_app_unhealthy_total`                | <MetricKind kind="counter" /> | `app`, `kind`                  | Times an app that had been answering stopped, for long enough to be called unhealthy, by the kind of check that said so.                                                                                                    |
| `nibrunner_app_wakes_total`                    | <MetricKind kind="counter" /> | `app`, `outcome`               | Times an on-request app was woken by a request, by how.                                                                                                                                                                     |
| `nibrunner_raw_port_bytes_total`               | <MetricKind kind="counter" /> | `app`, `protocol`, `direction` | What the relay carried on an app's raw ports, by which way it went: in is towards the app. A TCP session's bytes are counted when it ends.                                                                                  |
| `nibrunner_raw_port_sessions_total`            | <MetricKind kind="counter" /> | `app`, `protocol`, `outcome`   | Sessions on an app's raw ports — a TCP connection, or a UDP client address — by what came of them: relayed, the app was down, its wake was refused, or it would not take the session. Only for apps with a raw port.        |
| `nibrunner_volume_size_bytes`                  | <MetricKind kind="gauge" />   | `volume`, `app`                | What a volume was set aside as.                                                                                                                                                                                             |
| `nibrunner_volume_state`                       | <MetricKind kind="gauge" />   | `volume`, `app`, `state`       | 1 for the state a volume is in, 0 for every state it is not.                                                                                                                                                                |
| `nibrunner_volume_used_bytes`                  | <MetricKind kind="gauge" />   | `volume`, `app`                | What a guest reported filling of its volume, when it was last measured. Absent until it has been.                                                                                                                           |

## The host [#the-host]

| Series                                            | Type                            | Labels                                          | What it says                                                                                                                                                                                                                                                                                                             |
| ------------------------------------------------- | ------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `nibrunner_activator_responses_total`             | <MetricKind kind="counter" />   | `answer`                                        | What the HTTP activator answered requests for on-request apps with. The proxy counts every one of these as served, because serving it is what it did.                                                                                                                                                                    |
| `nibrunner_build_info`                            | <MetricKind kind="gauge" />     | `agent`, `guest_image`, `firecracker`, `zerofs` | What this host runs, as labels on a 1.                                                                                                                                                                                                                                                                                   |
| `nibrunner_checkpoint_state`                      | <MetricKind kind="gauge" />     | `checkpoint`, `volume`, `state`                 | 1 for the state a checkpoint is in, 0 for every state it is not.                                                                                                                                                                                                                                                         |
| `nibrunner_conntrack_entries`                     | <MetricKind kind="gauge" />     | `of`                                            | The kernel's connection-tracking table, which every flow between the proxy and a guest and every flow a guest opens is an entry of, for its lifetime and 120 s after. Full, the kernel drops packets for every app on the host. max\_apps in config.toml sizes the max. Absent where the kernel tracks nothing.          |
| `nibrunner_converge_outcomes_total`               | <MetricKind kind="counter" />   | `desired_state`, `cause`, `outcome`             | Asks this host set out on, by how they ended: the app got there, or the next ask for it came first.                                                                                                                                                                                                                      |
| `nibrunner_converge_seconds`                      | <MetricKind kind="histogram" /> | `phase`, `desired_state`, `cause`               | How long giving an app what its document asked for took, from the change being noticed, by the stretch of the work: fetching its layers, readying its volume, booting it, and waiting for it to be ready. Total is the whole of it.                                                                                      |
| `nibrunner_desired_state_unreadable_total`        | <MetricKind kind="counter" />   | —                                               | Times the desired state file was there and could not be read as a document.                                                                                                                                                                                                                                              |
| `nibrunner_export_size_bytes`                     | <MetricKind kind="gauge" />     | `export`                                        | What an export came to, once written. Absent until it has been.                                                                                                                                                                                                                                                          |
| `nibrunner_export_state`                          | <MetricKind kind="gauge" />     | `export`, `state`                               | 1 for the state an export is in, 0 for every state it is not.                                                                                                                                                                                                                                                            |
| `nibrunner_guest_startup_phase_seconds`           | <MetricKind kind="histogram" /> | `phase`                                         | Guest startup checkpoints through first host-observed readiness. Exec-to-ready aligns the guest clock with the sampling handshake midpoint; sample\_round\_trip bounds calibration uncertainty rather than startup work.                                                                                                 |
| `nibrunner_health_probe_seconds`                  | <MetricKind kind="histogram" /> | `result`                                        | A health probe, by what it found. One that found nothing ran out the check's timeout.                                                                                                                                                                                                                                    |
| `nibrunner_host_allocatable`                      | <MetricKind kind="gauge" />     | `resource`                                      | What the host has left to give a new app.                                                                                                                                                                                                                                                                                |
| `nibrunner_host_capacity`                         | <MetricKind kind="gauge" />     | `resource`                                      | What the host has, by resource.                                                                                                                                                                                                                                                                                          |
| `nibrunner_host_converged`                        | <MetricKind kind="gauge" />     | —                                               | 1 while every app is what its document asks for.                                                                                                                                                                                                                                                                         |
| `nibrunner_host_deferred_work`                    | <MetricKind kind="gauge" />     | —                                               | 1 while the last pass left work it could not do yet, such as a volume still held by an app on its way down.                                                                                                                                                                                                              |
| `nibrunner_host_isolated`                         | <MetricKind kind="gauge" />     | —                                               | 1 while the isolation ruleset is applied. Nothing is started or woken while it is not.                                                                                                                                                                                                                                   |
| `nibrunner_host_memory_available_bytes`           | <MetricKind kind="gauge" />     | —                                               | What the kernel says it could give out without swapping, as against the promises nibrunner\_host\_allocatable adds up. Absent where the kernel does not say.                                                                                                                                                             |
| `nibrunner_host_reconciled`                       | <MetricKind kind="gauge" />     | —                                               | 1 once a pass over the document has run to the end since this daemon started.                                                                                                                                                                                                                                            |
| `nibrunner_host_state`                            | <MetricKind kind="gauge" />     | `state`                                         | 1 for the state the host reports itself in, 0 for every state it is not.                                                                                                                                                                                                                                                 |
| `nibrunner_layer_fetch_bytes_total`               | <MetricKind kind="counter" />   | —                                               | What has been pulled from the artifact store, as the store sized it.                                                                                                                                                                                                                                                     |
| `nibrunner_layer_preparation_phase_seconds`       | <MetricKind kind="histogram" /> | `phase`, `outcome`                              | Cache-miss layer preparation by stage: waiting for preparation capacity, reading an artifact, verifying its digest, packing an executable directly to a staged file, and publishing its image. Cache hits do not repeat these stages. OCI extraction has its own storage operation.                                      |
| `nibrunner_layers_cached_total`                   | <MetricKind kind="counter" />   | —                                               | Layers a pass asked for that were already in the cache, so nothing was fetched.                                                                                                                                                                                                                                          |
| `nibrunner_process_start_time_seconds`            | <MetricKind kind="gauge" />     | —                                               | When this daemon started, as seconds since the epoch.                                                                                                                                                                                                                                                                    |
| `nibrunner_proxy_request_duration_seconds`        | <MetricKind kind="histogram" /> | —                                               | Deciding a route and getting an answer back from the app. Ends when the response is handed on to be written, so it counts neither the handshake before it nor the write after.                                                                                                                                           |
| `nibrunner_proxy_requests_in_flight`              | <MetricKind kind="gauge" />     | —                                               | Requests the proxy has taken and not yet answered.                                                                                                                                                                                                                                                                       |
| `nibrunner_proxy_requests_total`                  | <MetricKind kind="counter" />   | `outcome`                                       | Requests this proxy answered, by what it answered with.                                                                                                                                                                                                                                                                  |
| `nibrunner_proxy_tls_handshakes_total`            | <MetricKind kind="counter" />   | `outcome`                                       | TLS handshakes at the proxy, by how they ended.                                                                                                                                                                                                                                                                          |
| `nibrunner_reconcile_last_pass_timestamp_seconds` | <MetricKind kind="gauge" />     | —                                               | When the last pass over the document finished, as seconds since the epoch. 0 until one has.                                                                                                                                                                                                                              |
| `nibrunner_reconcile_seconds`                     | <MetricKind kind="histogram" /> | `trigger`                                       | A pass from the document to the host, by what set it going.                                                                                                                                                                                                                                                              |
| `nibrunner_refresh_seconds`                       | <MetricKind kind="histogram" /> | —                                               | A pass from the host to the record: probing what is up, settling each state, applying the ruleset and the routes, writing down what it found.                                                                                                                                                                            |
| `nibrunner_report_written_timestamp_seconds`      | <MetricKind kind="gauge" />     | —                                               | When reported.json was last written, as seconds since the epoch. 0 until it has been.                                                                                                                                                                                                                                    |
| `nibrunner_sleep_outcomes_total`                  | <MetricKind kind="counter" />   | `reason`, `outcome`                             | Times a policy said an app may sleep, by what came of it: it slept, its microVM refused to be snapshotted, or the snapshot failed.                                                                                                                                                                                       |
| `nibrunner_sleep_phase_seconds`                   | <MetricKind kind="histogram" /> | `phase`, `reason`                               | Putting an app to sleep, by why and the stretch of it: how late the policy was acted on past what it allowed, asking the guest to flush, and the whole of the flush and the snapshot. The snapshot alone is nibrunner\_vm\_snapshot\_duration\_seconds.                                                                  |
| `nibrunner_slots`                                 | <MetricKind kind="gauge" />     | `of`                                            | Slots on this host: each holds an app's ports, tap and guest address, and an app with none is refused. The total is max\_apps in config.toml.                                                                                                                                                                            |
| `nibrunner_start_refusals_total`                  | <MetricKind kind="counter" />   | `reason`                                        | Starts a pass refused, by why: no room in memory beside what is already up, or the isolation ruleset not applied. One refused for room waits, pending, and is planned again the next pass.                                                                                                                               |
| `nibrunner_storage_operation_seconds`             | <MetricKind kind="histogram" /> | `operation`, `outcome`                          | Something done to storage or the artifact store on an app's behalf, by what and how it ended.                                                                                                                                                                                                                            |
| `nibrunner_up`                                    | <MetricKind kind="gauge" />     | —                                               | Whether this daemon answered the scrape.                                                                                                                                                                                                                                                                                 |
| `nibrunner_vm_boot_phase_seconds`                 | <MetricKind kind="histogram" /> | `phase`                                         | Successful cold microVM boots by phase: staging the jail and devices, spawning Firecracker, waiting for guest initialization, and the total before application readiness.                                                                                                                                                |
| `nibrunner_vm_clock_sync_phase_seconds`           | <MetricKind kind="histogram" /> | `phase`                                         | Successful restore clock handshakes by phase: connecting to the guest control port, waiting for its READY reply, and setting its clock and confirming tenant release.                                                                                                                                                    |
| `nibrunner_vm_restore_duration_seconds`           | <MetricKind kind="histogram" /> | —                                               | Bringing a guest back from the snapshot it slept in. The other half of the sleep, and cheaper than it by orders of magnitude.                                                                                                                                                                                            |
| `nibrunner_vm_restore_phase_seconds`              | <MetricKind kind="histogram" /> | `phase`                                         | Successful native snapshot restores by phase, before application readiness and request forwarding. Snapshot verification includes its blocking-worker wait; load includes waiting for the VMM API socket.                                                                                                                |
| `nibrunner_vm_snapshot_duration_seconds`          | <MetricKind kind="histogram" /> | —                                               | Pausing a guest that has gone quiet and writing its memory out. Scales with the memory the app was given, so it is the cost of the sleep rather than of the app.                                                                                                                                                         |
| `nibrunner_vm_start_capacity_timeouts_total`      | <MetricKind kind="counter" />   | `caller`                                        | Wake requests whose VM start capacity queue deadline expired, by caller. Cancelled requests and terminal wake failures are not timeouts.                                                                                                                                                                                 |
| `nibrunner_vm_start_capacity_wait_seconds`        | <MetricKind kind="histogram" /> | `caller`                                        | Time spent awaiting each VM start capacity check, by caller. Includes immediately available capacity and cancelled waits; excludes boot, restore and readiness work. A request can wait more than once when admission races.                                                                                             |
| `nibrunner_vm_start_capacity_waiters`             | <MetricKind kind="gauge" />     | `caller`                                        | Callers currently awaiting VM start capacity, excluding those already booting, restoring or checking readiness.                                                                                                                                                                                                          |
| `nibrunner_wake_duration_seconds`                 | <MetricKind kind="histogram" /> | —                                               | Bringing a sleeping app back for the request that asked for it, from the request reaching the waker to the app being ready to be forwarded one, whether it set the wake going or joined one under way. On an on-request host this is the population that makes the tail of nibrunner\_proxy\_request\_duration\_seconds. |
| `nibrunner_wake_first_response_seconds`           | <MetricKind kind="histogram" /> | —                                               | The request that woke an app, from being handed to the app to the app answering it. What being ready was worth.                                                                                                                                                                                                          |
| `nibrunner_wake_phase_seconds`                    | <MetricKind kind="histogram" /> | `phase`, `outcome`                              | One wake, by how it was done and the stretch of it: waiting for what readyWhen names once the microVM was back, and the whole. Total less ready is bringing the microVM back; the VMM's own part of that is nibrunner\_vm\_restore\_duration\_seconds.                                                                   |
| `nibrunner_wake_refusals_total`                   | <MetricKind kind="counter" />   | `reason`                                        | Wakes this host refused, by why.                                                                                                                                                                                                                                                                                         |
| `nibrunner_wake_requests_coalesced_total`         | <MetricKind kind="counter" />   | —                                               | Requests that found a wake of their app already under way and waited on it.                                                                                                                                                                                                                                              |
