Guides

Sleep and wake

When an app sleeps, and what wakes it.

An app with no traffic is snapshotted and suspended. The next request wakes it. This is how a host holds more apps than it can run at once. Which apps sleep, and after how long, is set per instance in desired.json.

Desired state

desiredState says whether an instance should be up:

desiredStateBehaviour
runningKeep the microVM up.
on-requestStart it on first deploy and on every request that arrives while it sleeps. Let it sleep in between.
stoppedStop the microVM. Requests to its hostnames get a 503.

When it sleeps

activation.sleepWhen says what puts an on-request instance to sleep:

{
  "desiredState": "on-request",
  "activation": {
    "sleepWhen": { "kind": "traffic-idle", "timeoutMs": 900000 }
  }
}
sleepWhenThe microVM is suspended when
{ "kind": "traffic-idle", "timeoutMs": N }Nothing has been sent to it for N ms. A request or stream the proxy is still carrying counts as traffic, however quiet it is.
{ "kind": "max-lifetime", "ttlMs": N }It has been up for N ms, busy or not. A request still open at that point is given up to 30 s more to finish.
{ "kind": "never" }Never. Only desired.json takes it down.

Without sleepWhen, an on-request instance sleeps after five minutes idle. Any other desiredState never sleeps.

Note

  • Only an on-request instance may set a sleepWhen other than never. Nothing would wake any other kind, and the next reconcile pass would start it again. A file that does this is rejected as a whole.
  • idleTimeoutMs is the old spelling of sleepWhen: traffic-idle. A file that sets both is rejected.
  • A host that has not managed to read an app's traffic lately leaves it up rather than taking the silence for quiet, and says so in its log. max-lifetime runs on the clock either way.
  • A sleep that does not happen leaves the app up. The host tries again after 30 s, then waits twice as long after each further failure, up to 10 minutes. Traffic in between resets the wait.

What wakes it

Any inbound traffic: an HTTP request on one of its hostnames, or anything on a raw port. The connection is accepted and held, then forwarded once the app's health check passes. The client sees a slow first response, not a refused connection.

Snapshot integrity

Before resuming a restored app, the host sets the guest's wall clock to the current host time. The app stays frozen until that succeeds. If the guest cannot synchronize its clock, the host starts it from cold instead.

Guests started before a host and guest-image upgrade keep running. Their snapshots retain the image version they booted with. On the next wake, a snapshot from a different or unknown guest image is discarded and the app starts from cold with the installed image. Files on its volume survive, but in-memory state does not.

If an older guest's init does not support the freeze protocol, the host refuses sleep and retries after its normal backoff. Redeploy the app to start it with the new guest image.

The host commits a snapshot only after its state, memory and integrity record have been synced to disk. It checks both files' sizes and the state file's SHA-256 hash before restoring. On filesystems supporting fs-verity, memory is sealed during publication: wake compares its fs-verity digest, and the kernel verifies pages as they are read. Other filesystems retain full SHA-256 verification of memory on each wake. Existing SHA-256 snapshots remain readable. An fs-verity page error can surface after resume. The host replaces the memory-faulted VM from cold without spending the app's restart window. A missing, damaged or older snapshot without integrity records is discarded; the request starts the app from cold instead. Files on its volume survive, but in-memory state does not.

Snapshot cost

Both paths read memory during sleep. Only full-file SHA-256 verification reads all memory again before wake; fs-verity measures its digest in constant time. The guest stays paused until the snapshot is committed, so a failed commit can resume it without losing memory.

Restore timings

nibrunner_vm_restore_duration_seconds measures successful native restores. nibrunner_vm_restore_phase_seconds{phase="..."} breaks the same restores into histograms with sub-millisecond buckets. App readiness checks and request forwarding follow the restore and have their own wake and proxy metrics.

phaseWork measured
snapshot_verifyIntegrity verification, including the wait for its blocking worker.
compatibilityChecking the recorded VMM and guest image against the snapshot.
jailConsuming the snapshot stamp and rebuilding the jail.
resourcesAttaching log and cron channels, setting TAP ownership and linking snapshot files.
spawnLaunching the jailed VMM process.
loadLoading the snapshot, including waiting for the VMM's API socket.
resumeThe VMM resume API call.
clock_syncSynchronizing the guest clock and releasing the tenant freeze.
neighbourRefreshing the guest's network neighbour entry.
cleanupRemoving consumed snapshot files and temporary links.

On this page