Metrics

Every series the host's Prometheus endpoint publishes.

What the host serves at /metrics, from the [metrics] section of config.toml. Every series is prefixed nibrunner_, and what follows names what is being measured. Anything labelled app is about one app and nothing else, which is what a per-app dashboard filters on.

A histogram publishes three series the table does not list one by one: _bucket with an extra le label, _sum and _count.

Per app

SeriesTypeLabelsWhat it says
nibrunner_app_convergedgaugeapp1 while an app is what its document asks for, 0 while it is not.
nibrunner_app_converging_secondsgaugeappHow long an app has been on its way to what its document asks for. 0 once it got there.
nibrunner_app_cpu_seconds_totalcounterappWhat an app's guest has reported spending, summed across the vCPUs it was given.
nibrunner_app_cpu_sharegaugeappThe share of one vCPU a guest was using, when it was last measured.
nibrunner_app_disk_mib_seconds_totalcounterapp, diskWhat an app has held on disk over time: what was set aside for it, and what its guest reported filling.
nibrunner_app_failures_totalcounterapp, reasonTimes an app failed, by where on the way up it did: refused by the document, no slot, its layers, its volume, the boot, the microVM stopping without being asked, nothing answering inside it, or its restarts running out.
nibrunner_app_health_probes_totalcounterapp, kind, resultProbes of an app, by the kind of check and what it found. For a boot-completed check, a probe is the microVM being up.
nibrunner_app_last_converge_secondsgaugeapp, phaseWhat the last ask for an app took, by the stretch of the work. Absent while one is under way.
nibrunner_app_last_healthy_timestamp_secondsgaugeappWhen an app last answered a probe, as seconds since the epoch. 0 for one that never has.
nibrunner_app_last_sleep_secondsgaugeappWhat the last sleep of an on-request app took, flush and snapshot. Absent until it has slept.
nibrunner_app_last_wake_secondsgaugeappWhat the last wake of an on-request app took. Absent until it has been woken.
nibrunner_app_measured_timestamp_secondsgaugeappWhen a guest last reported what it was using, as seconds since the epoch. 0 for one that never has; one that stopped is one this host can no longer hear.
nibrunner_app_memory_used_bytesgaugeappWhat a guest reported using, when it was last measured.
nibrunner_app_network_bytes_totalcounterapp, directionWhat has crossed an app's tap, by which way it went. Only what was let out is counted as sent.
nibrunner_app_request_duration_secondssummaryappWhat requests to one app cost this proxy. Divide the rate of the sum by the rate of the count for that app's mean.
nibrunner_app_requests_opengaugeappRequests routed to an app that are still being answered: from arrival until the last byte of the response body, or until a websocket closes. An app with one open is not quiet.
nibrunner_app_requests_totalcounterapp, classRequests routed to an app, by the class of the status that came back. A 502 for an app that could not be reached is the proxy's and counted apart below.
nibrunner_app_requests_unreachable_totalcounterappRequests routed to an app that nothing answered, so the proxy answered 502 for it.
nibrunner_app_restarts_totalcounterappTimes an app's tenant has been restarted inside its guest since the host last booted the app afresh.
nibrunner_app_sleeps_totalcounterapp, outcomeTimes a policy said an on-request app may sleep, by what came of it.
nibrunner_app_start_attemptsgaugeappStarts attempted against an app's restart budget in its current window. Resets when a start has held for the policy's resetAfterMs.
nibrunner_app_stategaugeapp, state1 for the state an app is in, 0 for every state it is not.
nibrunner_app_time_seconds_totalcounterapp, holdingHow long an app has been held, by what it was holding: memory while it runs, a snapshot on disk while it sleeps.
nibrunner_app_unhealthy_totalcounterapp, kindTimes an app that had been answering stopped, for long enough to be called unhealthy, by the kind of check that said so.
nibrunner_app_wakes_totalcounterapp, outcomeTimes an on-request app was woken by a request, by how.
nibrunner_raw_port_bytes_totalcounterapp, protocol, directionWhat the relay carried on an app's raw ports, by which way it went: in is towards the app. A TCP session's bytes are counted when it ends.
nibrunner_raw_port_sessions_totalcounterapp, protocol, outcomeSessions on an app's raw ports — a TCP connection, or a UDP client address — by what came of them: relayed, the app was down, its wake was refused, or it would not take the session. Only for apps with a raw port.
nibrunner_volume_size_bytesgaugevolume, appWhat a volume was set aside as.
nibrunner_volume_stategaugevolume, app, state1 for the state a volume is in, 0 for every state it is not.
nibrunner_volume_used_bytesgaugevolume, appWhat a guest reported filling of its volume, when it was last measured. Absent until it has been.

The host

SeriesTypeLabelsWhat it says
nibrunner_activator_responses_totalcounteranswerWhat the HTTP activator answered requests for on-request apps with. The proxy counts every one of these as served, because serving it is what it did.
nibrunner_build_infogaugeagent, guest_image, firecracker, zerofsWhat this host runs, as labels on a 1.
nibrunner_checkpoint_stategaugecheckpoint, volume, state1 for the state a checkpoint is in, 0 for every state it is not.
nibrunner_conntrack_entriesgaugeofThe kernel's connection-tracking table, which every flow between the proxy and a guest and every flow a guest opens is an entry of, for its lifetime and 120 s after. Full, the kernel drops packets for every app on the host. max_apps in config.toml sizes the max. Absent where the kernel tracks nothing.
nibrunner_converge_outcomes_totalcounterdesired_state, cause, outcomeAsks this host set out on, by how they ended: the app got there, or the next ask for it came first.
nibrunner_converge_secondshistogramphase, desired_state, causeHow long giving an app what its document asked for took, from the change being noticed, by the stretch of the work: fetching its layers, readying its volume, booting it, and waiting for it to be ready. Total is the whole of it.
nibrunner_desired_state_unreadable_totalcounter—Times the desired state file was there and could not be read as a document.
nibrunner_export_size_bytesgaugeexportWhat an export came to, once written. Absent until it has been.
nibrunner_export_stategaugeexport, state1 for the state an export is in, 0 for every state it is not.
nibrunner_guest_startup_phase_secondshistogramphaseGuest startup checkpoints through first host-observed readiness. Exec-to-ready aligns the guest clock with the sampling handshake midpoint; sample_round_trip bounds calibration uncertainty rather than startup work.
nibrunner_health_probe_secondshistogramresultA health probe, by what it found. One that found nothing ran out the check's timeout.
nibrunner_host_allocatablegaugeresourceWhat the host has left to give a new app.
nibrunner_host_capacitygaugeresourceWhat the host has, by resource.
nibrunner_host_convergedgauge—1 while every app is what its document asks for.
nibrunner_host_deferred_workgauge—1 while the last pass left work it could not do yet, such as a volume still held by an app on its way down.
nibrunner_host_isolatedgauge—1 while the isolation ruleset is applied. Nothing is started or woken while it is not.
nibrunner_host_memory_available_bytesgauge—What the kernel says it could give out without swapping, as against the promises nibrunner_host_allocatable adds up. Absent where the kernel does not say.
nibrunner_host_reconciledgauge—1 once a pass over the document has run to the end since this daemon started.
nibrunner_host_stategaugestate1 for the state the host reports itself in, 0 for every state it is not.
nibrunner_layer_fetch_bytes_totalcounter—What has been pulled from the artifact store, as the store sized it.
nibrunner_layer_preparation_phase_secondshistogramphase, outcomeCache-miss layer preparation by stage: waiting for preparation capacity, reading an artifact, verifying its digest, packing an executable directly to a staged file, and publishing its image. Cache hits do not repeat these stages. OCI extraction has its own storage operation.
nibrunner_layers_cached_totalcounter—Layers a pass asked for that were already in the cache, so nothing was fetched.
nibrunner_process_start_time_secondsgauge—When this daemon started, as seconds since the epoch.
nibrunner_proxy_request_duration_secondshistogram—Deciding a route and getting an answer back from the app. Ends when the response is handed on to be written, so it counts neither the handshake before it nor the write after.
nibrunner_proxy_requests_in_flightgauge—Requests the proxy has taken and not yet answered.
nibrunner_proxy_requests_totalcounteroutcomeRequests this proxy answered, by what it answered with.
nibrunner_proxy_tls_handshakes_totalcounteroutcomeTLS handshakes at the proxy, by how they ended.
nibrunner_reconcile_last_pass_timestamp_secondsgauge—When the last pass over the document finished, as seconds since the epoch. 0 until one has.
nibrunner_reconcile_secondshistogramtriggerA pass from the document to the host, by what set it going.
nibrunner_refresh_secondshistogram—A pass from the host to the record: probing what is up, settling each state, applying the ruleset and the routes, writing down what it found.
nibrunner_report_written_timestamp_secondsgauge—When reported.json was last written, as seconds since the epoch. 0 until it has been.
nibrunner_sleep_outcomes_totalcounterreason, outcomeTimes a policy said an app may sleep, by what came of it: it slept, its microVM refused to be snapshotted, or the snapshot failed.
nibrunner_sleep_phase_secondshistogramphase, reasonPutting an app to sleep, by why and the stretch of it: how late the policy was acted on past what it allowed, asking the guest to flush, and the whole of the flush and the snapshot. The snapshot alone is nibrunner_vm_snapshot_duration_seconds.
nibrunner_slotsgaugeofSlots on this host: each holds an app's ports, tap and guest address, and an app with none is refused. The total is max_apps in config.toml.
nibrunner_start_refusals_totalcounterreasonStarts a pass refused, by why: no room in memory beside what is already up, or the isolation ruleset not applied. One refused for room waits, pending, and is planned again the next pass.
nibrunner_storage_operation_secondshistogramoperation, outcomeSomething done to storage or the artifact store on an app's behalf, by what and how it ended.
nibrunner_upgauge—Whether this daemon answered the scrape.
nibrunner_vm_boot_phase_secondshistogramphaseSuccessful cold microVM boots by phase: staging the jail and devices, spawning Firecracker, waiting for guest initialization, and the total before application readiness.
nibrunner_vm_clock_sync_phase_secondshistogramphaseSuccessful restore clock handshakes by phase: connecting to the guest control port, waiting for its READY reply, and setting its clock and confirming tenant release.
nibrunner_vm_restore_duration_secondshistogram—Bringing a guest back from the snapshot it slept in. The other half of the sleep, and cheaper than it by orders of magnitude.
nibrunner_vm_restore_phase_secondshistogramphaseSuccessful native snapshot restores by phase, before application readiness and request forwarding. Snapshot verification includes its blocking-worker wait; load includes waiting for the VMM API socket.
nibrunner_vm_snapshot_duration_secondshistogram—Pausing a guest that has gone quiet and writing its memory out. Scales with the memory the app was given, so it is the cost of the sleep rather than of the app.
nibrunner_vm_start_capacity_timeouts_totalcountercallerWake requests whose VM start capacity queue deadline expired, by caller. Cancelled requests and terminal wake failures are not timeouts.
nibrunner_vm_start_capacity_wait_secondshistogramcallerTime spent awaiting each VM start capacity check, by caller. Includes immediately available capacity and cancelled waits; excludes boot, restore and readiness work. A request can wait more than once when admission races.
nibrunner_vm_start_capacity_waitersgaugecallerCallers currently awaiting VM start capacity, excluding those already booting, restoring or checking readiness.
nibrunner_wake_duration_secondshistogram—Bringing a sleeping app back for the request that asked for it, from the request reaching the waker to the app being ready to be forwarded one, whether it set the wake going or joined one under way. On an on-request host this is the population that makes the tail of nibrunner_proxy_request_duration_seconds.
nibrunner_wake_first_response_secondshistogram—The request that woke an app, from being handed to the app to the app answering it. What being ready was worth.
nibrunner_wake_phase_secondshistogramphase, outcomeOne wake, by how it was done and the stretch of it: waiting for what readyWhen names once the microVM was back, and the whole. Total less ready is bringing the microVM back; the VMM's own part of that is nibrunner_vm_restore_duration_seconds.
nibrunner_wake_refusals_totalcounterreasonWakes this host refused, by why.
nibrunner_wake_requests_coalesced_totalcounter—Requests that found a wake of their app already under way and waited on it.

On this page