Guides

Host capacity

How many apps a host is sized for, what the machine can hold, and how to change it.

max_apps is a top-level key in config.toml, above every section:

max_apps = 300

Everything sized by slot count follows from it: the slot ring, the eight loopback ports each slot reserves from 21000, the nbd minors on a zerofs host, the 1024 conntrack entries per app, and nibrunner_slots{of="total"} on the metrics page. A full host reports it by naming this key.

What the machine can hold

install measures three limits, assuming each app uses 1 vCPU, 256 MiB of memory and an 8 GiB volume:

LimitHow it is computed
Runs at onceGuest memory (total, minus the host and the storage cache) divided by 256 MiB.
Holds on diskThe disk under paths.snapshot_dir, minus the storage cache and an 8 GiB reserve, divided by the size of one snapshot. On a local-file host, one volume per app is added to that size.
Fits on ports5567. Each slot reserves eight ports from 21000, and the last one must be at most 65535.

The installer sets max_apps to the lowest of the three, with a comment above the key listing all three. Every nibrunnerd start prints the three limits against the configured value:

max_apps = 300: assuming an app is 1 vCPU, 256 MiB and 8 GiB to start, this host runs 29 at once, holds 320 on disk, and fits 5567 on its ports; above 29 counts on apps sleeping

Exceeding the memory limit is only reported. That is what sleeping is for. Exceeding the disk or port limit makes start print the value to set instead. Both are advice, not errors: an app with more resources counts for more, and the value is yours to set.

Unused guest pages

The bundled guest reports unused pages through Firecracker's virtio balloon device. The host can reclaim their physical memory without reducing the guest's configured memory limit. The balloon target remains zero; no polling controller or forced inflation is used. The bundled kernel must enable CONFIG_MEMORY_BALLOON, CONFIG_VIRTIO_BALLOON and CONFIG_PAGE_REPORTING.

This can reduce resident memory and improve snapshot compressibility. Full snapshots still read all guest memory, and guests may allocate their full limit again. Admission therefore continues to reserve the configured limit. Reporting is not a promise that more guests fit concurrently.

The memory limit is enforced

A reconcile pass boots an app only if its memory fits alongside every app that is up or starting, including the ones started earlier in the same pass. An app that does not fit stays pending, with the shortfall on its record (waiting to be started: its host is 256 MiB short of the memory it needs). No start attempt is consumed, and it is retried every pass. So if desired.json names a thousand on-request apps on a host that runs 240 at once, 240 boot and the rest follow as the first ones sleep. A wake that does not fit is refused the same way, and its request gets a 503. The number of waiting starts is logged once per pass and counted in nibrunner_start_refusals_total{reason="no_room"}.

Changing it

  1. Edit max_apps.
  2. Run nibrunnerd start.

Raising only max_apps on a local-file host grows its capacity in place. Existing slots and open proxy connections stay where they are. start reports reloaded; repeating it with the same configuration reports running. A changed executable, startup file, or other setting still takes the restart path.

Note

  • Raising it on a zerofs host needs a reboot. The nbd module is loaded with max_apps + 1 minors, which cannot be changed in place. start says so rather than reloading a module with volumes attached.
  • Lowering it below a slot an app occupies moves that app to a new slot on the next start, with a new loopback port and a new guest address.

HTTP concurrency

By default the HTTP router imposes no concurrency limit. To bound requests across apps and per app, add an optional section under your HTTP listener in config.toml:

[proxy.http.concurrency]
host_concurrent = 128
app_concurrent = 16

These are example capacities, not measured application throughput. Values must be between 1 and 65535. The per-app ceiling applies automatically to every app, including newly added apps. All hostnames of an app share its limit. Run nibrunnerd start after changing host configuration. This restarts the daemon when needed, interrupting proxy connections, without redeploying guests.

A full app or host returns HTTP 503 with Retry-After: 1 immediately. Requests waiting for a cold start, uploads, streaming response bodies and upgraded connections keep their permits until they finish or disconnect. An ordinary request releases its permit only after both its upload and response finish or are dropped. No waiting queue is created. Unknown hostnames do not allocate concurrency state. This covers the hostname router, not raw ports, guest-to-guest connections or the private activator listeners. Limits bound proxy work; they do not cancel work a tenant continues after a client disconnects.

Concurrent VM starts

The required top-level max_concurrent_vm_starts in config.toml bounds simultaneous Firecracker boots and snapshot restores. The installer writes 2; values must be between 1 and 65535. Existing configs must add this key before starting the updated daemon. Run nibrunnerd start after changing it.

max_concurrent_vm_starts = 2

A full budget defers reconciliation. HTTP requests and cron wakes wait up to 30 seconds for start capacity. Requests for an app already being woken still share its single wake. A cold boot holds its permit until the guest's control port accepts a connection, confirming PID 1 initialized. A restore holds it through the guest clock-sync and release handshake. A guest that does not initialize within 60 seconds is stopped and releases its permit. Deferred boots remain pending without spending the app's restart budget. Application health checks, artifact preparation and volume attachment are outside this limit. HTTP request concurrency is separate.

VM CPU and memory limits

Every new VM boot and snapshot restore applies cgroup v2 limits through the Firecracker jailer. The app's resources in desired state supplies both values; no additional host configuration is needed.

ResourceHost limit
CPUvcpuCount host CPUs in aggregate, over each 100 ms period.
MemorymemoryMib plus 64 MiB and 12.5% of guest RAM for Firecracker and kernel overhead.
SwapDisabled for the VM process.

For example, a VM with 1 vCPU and 256 MiB of guest RAM has a 100% CPU ceiling and a 352 MiB host memory ceiling. CPU limits throttle the whole VMM, including its I/O threads. Exceeding the memory ceiling can kill the VM; its existing restart policy handles that exit. The allowance is covered by guest boot, memory pressure, sleep and restore tests; it does not reserve physical memory or change admission.

While Firecracker is paused for a snapshot, its memory ceiling temporarily adds one guest RAM to cover the snapshot's filesystem cache. A failed sleep removes the allowance before resuming the guest. A completed sleep releases the stopped VMM's cgroup; retained snapshot pages remain host storage charges rather than consuming the restored VM's running limit. This also applies to snapshots on tmpfs.

The host must provide the cgroup v2 CPU and memory controllers. If limits cannot be applied, the VM fails to start. Existing VMs keep their current limits until their next boot or restore; restarting the daemon readopts them.

On this page