Metrics
Every series the host's Prometheus endpoint publishes.
What the host serves at /metrics, from the [metrics] section of config.toml. Every series is
prefixed nibrunner_, and what follows names what is being measured. Anything labelled app is
about one app and nothing else, which is what a per-app dashboard filters on.
A histogram publishes three series the table does not list one by one: _bucket with an extra
le label, _sum and _count.
Per app
| Series | Type | Labels | What it says |
|---|---|---|---|
nibrunner_app_converged | gauge | app | 1 while an app is what its document asks for, 0 while it is not. |
nibrunner_app_converging_seconds | gauge | app | How long an app has been on its way to what its document asks for. 0 once it got there. |
nibrunner_app_cpu_seconds_total | counter | app | What an app's guest has reported spending, summed across the vCPUs it was given. |
nibrunner_app_cpu_share | gauge | app | The share of one vCPU a guest was using, when it was last measured. |
nibrunner_app_disk_mib_seconds_total | counter | app, disk | What an app has held on disk over time: what was set aside for it, and what its guest reported filling. |
nibrunner_app_failures_total | counter | app, reason | Times an app failed, by where on the way up it did: refused by the document, no slot, its layers, its volume, the boot, the microVM stopping without being asked, nothing answering inside it, or its restarts running out. |
nibrunner_app_health_probes_total | counter | app, kind, result | Probes of an app, by the kind of check and what it found. For a boot-completed check, a probe is the microVM being up. |
nibrunner_app_last_converge_seconds | gauge | app, phase | What the last ask for an app took, by the stretch of the work. Absent while one is under way. |
nibrunner_app_last_healthy_timestamp_seconds | gauge | app | When an app last answered a probe, as seconds since the epoch. 0 for one that never has. |
nibrunner_app_last_sleep_seconds | gauge | app | What the last sleep of an on-request app took, flush and snapshot. Absent until it has slept. |
nibrunner_app_last_wake_seconds | gauge | app | What the last wake of an on-request app took. Absent until it has been woken. |
nibrunner_app_measured_timestamp_seconds | gauge | app | When a guest last reported what it was using, as seconds since the epoch. 0 for one that never has; one that stopped is one this host can no longer hear. |
nibrunner_app_memory_used_bytes | gauge | app | What a guest reported using, when it was last measured. |
nibrunner_app_network_bytes_total | counter | app, direction | What has crossed an app's tap, by which way it went. Only what was let out is counted as sent. |
nibrunner_app_request_duration_seconds | summary | app | What requests to one app cost this proxy. Divide the rate of the sum by the rate of the count for that app's mean. |
nibrunner_app_requests_open | gauge | app | Requests routed to an app that are still being answered: from arrival until the last byte of the response body, or until a websocket closes. An app with one open is not quiet. |
nibrunner_app_requests_total | counter | app, class | Requests routed to an app, by the class of the status that came back. A 502 for an app that could not be reached is the proxy's and counted apart below. |
nibrunner_app_requests_unreachable_total | counter | app | Requests routed to an app that nothing answered, so the proxy answered 502 for it. |
nibrunner_app_restarts_total | counter | app | Times an app's tenant has been restarted inside its guest since the host last booted the app afresh. |
nibrunner_app_sleeps_total | counter | app, outcome | Times a policy said an on-request app may sleep, by what came of it. |
nibrunner_app_start_attempts | gauge | app | Starts attempted against an app's restart budget in its current window. Resets when a start has held for the policy's resetAfterMs. |
nibrunner_app_state | gauge | app, state | 1 for the state an app is in, 0 for every state it is not. |
nibrunner_app_time_seconds_total | counter | app, holding | How long an app has been held, by what it was holding: memory while it runs, a snapshot on disk while it sleeps. |
nibrunner_app_unhealthy_total | counter | app, kind | Times an app that had been answering stopped, for long enough to be called unhealthy, by the kind of check that said so. |
nibrunner_app_wakes_total | counter | app, outcome | Times an on-request app was woken by a request, by how. |
nibrunner_raw_port_bytes_total | counter | app, protocol, direction | What the relay carried on an app's raw ports, by which way it went: in is towards the app. A TCP session's bytes are counted when it ends. |
nibrunner_raw_port_sessions_total | counter | app, protocol, outcome | Sessions on an app's raw ports — a TCP connection, or a UDP client address — by what came of them: relayed, the app was down, its wake was refused, or it would not take the session. Only for apps with a raw port. |
nibrunner_volume_size_bytes | gauge | volume, app | What a volume was set aside as. |
nibrunner_volume_state | gauge | volume, app, state | 1 for the state a volume is in, 0 for every state it is not. |
nibrunner_volume_used_bytes | gauge | volume, app | What a guest reported filling of its volume, when it was last measured. Absent until it has been. |
The host
| Series | Type | Labels | What it says |
|---|---|---|---|
nibrunner_activator_responses_total | counter | answer | What the HTTP activator answered requests for on-request apps with. The proxy counts every one of these as served, because serving it is what it did. |
nibrunner_build_info | gauge | agent, guest_image, firecracker, zerofs | What this host runs, as labels on a 1. |
nibrunner_checkpoint_state | gauge | checkpoint, volume, state | 1 for the state a checkpoint is in, 0 for every state it is not. |
nibrunner_conntrack_entries | gauge | of | The kernel's connection-tracking table, which every flow between the proxy and a guest and every flow a guest opens is an entry of, for its lifetime and 120 s after. Full, the kernel drops packets for every app on the host. max_apps in config.toml sizes the max. Absent where the kernel tracks nothing. |
nibrunner_converge_outcomes_total | counter | desired_state, cause, outcome | Asks this host set out on, by how they ended: the app got there, or the next ask for it came first. |
nibrunner_converge_seconds | histogram | phase, desired_state, cause | How long giving an app what its document asked for took, from the change being noticed, by the stretch of the work: fetching its layers, readying its volume, booting it, and waiting for it to be ready. Total is the whole of it. |
nibrunner_desired_state_unreadable_total | counter | — | Times the desired state file was there and could not be read as a document. |
nibrunner_export_size_bytes | gauge | export | What an export came to, once written. Absent until it has been. |
nibrunner_export_state | gauge | export, state | 1 for the state an export is in, 0 for every state it is not. |
nibrunner_guest_startup_phase_seconds | histogram | phase | Guest startup checkpoints through first host-observed readiness. Exec-to-ready aligns the guest clock with the sampling handshake midpoint; sample_round_trip bounds calibration uncertainty rather than startup work. |
nibrunner_health_probe_seconds | histogram | result | A health probe, by what it found. One that found nothing ran out the check's timeout. |
nibrunner_host_allocatable | gauge | resource | What the host has left to give a new app. |
nibrunner_host_capacity | gauge | resource | What the host has, by resource. |
nibrunner_host_converged | gauge | — | 1 while every app is what its document asks for. |
nibrunner_host_deferred_work | gauge | — | 1 while the last pass left work it could not do yet, such as a volume still held by an app on its way down. |
nibrunner_host_isolated | gauge | — | 1 while the isolation ruleset is applied. Nothing is started or woken while it is not. |
nibrunner_host_memory_available_bytes | gauge | — | What the kernel says it could give out without swapping, as against the promises nibrunner_host_allocatable adds up. Absent where the kernel does not say. |
nibrunner_host_reconciled | gauge | — | 1 once a pass over the document has run to the end since this daemon started. |
nibrunner_host_state | gauge | state | 1 for the state the host reports itself in, 0 for every state it is not. |
nibrunner_layer_fetch_bytes_total | counter | — | What has been pulled from the artifact store, as the store sized it. |
nibrunner_layer_preparation_phase_seconds | histogram | phase, outcome | Cache-miss layer preparation by stage: waiting for preparation capacity, reading an artifact, verifying its digest, packing an executable directly to a staged file, and publishing its image. Cache hits do not repeat these stages. OCI extraction has its own storage operation. |
nibrunner_layers_cached_total | counter | — | Layers a pass asked for that were already in the cache, so nothing was fetched. |
nibrunner_process_start_time_seconds | gauge | — | When this daemon started, as seconds since the epoch. |
nibrunner_proxy_request_duration_seconds | histogram | — | Deciding a route and getting an answer back from the app. Ends when the response is handed on to be written, so it counts neither the handshake before it nor the write after. |
nibrunner_proxy_requests_in_flight | gauge | — | Requests the proxy has taken and not yet answered. |
nibrunner_proxy_requests_total | counter | outcome | Requests this proxy answered, by what it answered with. |
nibrunner_proxy_tls_handshakes_total | counter | outcome | TLS handshakes at the proxy, by how they ended. |
nibrunner_reconcile_last_pass_timestamp_seconds | gauge | — | When the last pass over the document finished, as seconds since the epoch. 0 until one has. |
nibrunner_reconcile_seconds | histogram | trigger | A pass from the document to the host, by what set it going. |
nibrunner_refresh_seconds | histogram | — | A pass from the host to the record: probing what is up, settling each state, applying the ruleset and the routes, writing down what it found. |
nibrunner_report_written_timestamp_seconds | gauge | — | When reported.json was last written, as seconds since the epoch. 0 until it has been. |
nibrunner_sleep_outcomes_total | counter | reason, outcome | Times a policy said an app may sleep, by what came of it: it slept, its microVM refused to be snapshotted, or the snapshot failed. |
nibrunner_sleep_phase_seconds | histogram | phase, reason | Putting an app to sleep, by why and the stretch of it: how late the policy was acted on past what it allowed, asking the guest to flush, and the whole of the flush and the snapshot. The snapshot alone is nibrunner_vm_snapshot_duration_seconds. |
nibrunner_slots | gauge | of | Slots on this host: each holds an app's ports, tap and guest address, and an app with none is refused. The total is max_apps in config.toml. |
nibrunner_start_refusals_total | counter | reason | Starts a pass refused, by why: no room in memory beside what is already up, or the isolation ruleset not applied. One refused for room waits, pending, and is planned again the next pass. |
nibrunner_storage_operation_seconds | histogram | operation, outcome | Something done to storage or the artifact store on an app's behalf, by what and how it ended. |
nibrunner_up | gauge | — | Whether this daemon answered the scrape. |
nibrunner_vm_boot_phase_seconds | histogram | phase | Successful cold microVM boots by phase: staging the jail and devices, spawning Firecracker, waiting for guest initialization, and the total before application readiness. |
nibrunner_vm_clock_sync_phase_seconds | histogram | phase | Successful restore clock handshakes by phase: connecting to the guest control port, waiting for its READY reply, and setting its clock and confirming tenant release. |
nibrunner_vm_restore_duration_seconds | histogram | — | Bringing a guest back from the snapshot it slept in. The other half of the sleep, and cheaper than it by orders of magnitude. |
nibrunner_vm_restore_phase_seconds | histogram | phase | Successful native snapshot restores by phase, before application readiness and request forwarding. Snapshot verification includes its blocking-worker wait; load includes waiting for the VMM API socket. |
nibrunner_vm_snapshot_duration_seconds | histogram | — | Pausing a guest that has gone quiet and writing its memory out. Scales with the memory the app was given, so it is the cost of the sleep rather than of the app. |
nibrunner_vm_start_capacity_timeouts_total | counter | caller | Wake requests whose VM start capacity queue deadline expired, by caller. Cancelled requests and terminal wake failures are not timeouts. |
nibrunner_vm_start_capacity_wait_seconds | histogram | caller | Time spent awaiting each VM start capacity check, by caller. Includes immediately available capacity and cancelled waits; excludes boot, restore and readiness work. A request can wait more than once when admission races. |
nibrunner_vm_start_capacity_waiters | gauge | caller | Callers currently awaiting VM start capacity, excluding those already booting, restoring or checking readiness. |
nibrunner_wake_duration_seconds | histogram | — | Bringing a sleeping app back for the request that asked for it, from the request reaching the waker to the app being ready to be forwarded one, whether it set the wake going or joined one under way. On an on-request host this is the population that makes the tail of nibrunner_proxy_request_duration_seconds. |
nibrunner_wake_first_response_seconds | histogram | — | The request that woke an app, from being handed to the app to the app answering it. What being ready was worth. |
nibrunner_wake_phase_seconds | histogram | phase, outcome | One wake, by how it was done and the stretch of it: waiting for what readyWhen names once the microVM was back, and the whole. Total less ready is bringing the microVM back; the VMM's own part of that is nibrunner_vm_restore_duration_seconds. |
nibrunner_wake_refusals_total | counter | reason | Wakes this host refused, by why. |
nibrunner_wake_requests_coalesced_total | counter | — | Requests that found a wake of their app already under way and waited on it. |