MicroVMs From The Bottom Up

Chapter 16: Snapshot And Restore

Booting a microVM from scratch takes on the order of 200 milliseconds — the kernel decompresses, the init process brings up devices, the runtime initializes. For most batch workloads that cost is amortized. For an invocation-per-request serverless platform running millions of events per second, it is the difference between a product that works and one that does not.

The mechanism that eliminates cold-start latency is snapshot and restore: freeze a running microVM, write its complete state to disk, then resume copies of that state on demand. The restored microVM picks up exactly where the frozen one left off — the kernel is already running, the runtime is already warm, the JIT cache is already populated. Firecracker snapshot restore with file-backed memory achieves as little as 4 ms on an EC2 m5.12xlarge. AWS Lambda SnapStart used this mechanism to bring Java function cold starts from over six seconds to under 200 ms.

The mechanism is not free. When the same frozen state resumes into multiple clones, every clone inherits identical entropy pools, identical clock values, identical session tokens, and identical PRNG seeds. What looks like a performance optimization is also a security surface that has to be explicitly managed.

Two Files On Disk

Every PUT /snapshot/create request produces exactly two output files. Firecracker does not manage block device contents; that is the operator's responsibility — block device files must be flushed and synced by the guest before the VM is paused.

The microVM state file is named by the snapshot_path field. It holds everything that is not guest memory: the serialized MicrovmState struct covering device emulation state, KVM VM state, and per-vCPU register state. The guest memory file is named by mem_file_path. It is a flat file of exactly mem_size_mib * 1024 * 1024 bytes: a complete or partial image of guest physical memory, depending on snapshot type.

The API requires the VM to be paused before snapshotting:

PATCH /vm               {"state": "Paused"}
PUT /snapshot/create    {"snapshot_type": "Full", "snapshot_path": "/srv/snap/state", "mem_file_path": "/srv/snap/mem"}

On the restore host, PUT /snapshot/load reads both files and leaves the microVM paused, ready to resume. If resume_vm: true is set in the load body, Firecracker issues the resume automatically; otherwise a separate PATCH /vm {"state": "Resumed"} is required.

The State File Format

The microVM state file has a fixed binary layout:

| 64-bit magic_id | | version string | | bitcode blob | | 64-bit CRC |

The magic ID is an architecture discriminant. On x86_64, SNAPSHOT_MAGIC_ID = 0x0710_1984_8664_0000u64; on aarch64, 0x0710_1984_AAAA_0000u64. These constants live in src/vmm/src/snapshot/mod.rs. An attempt to load an aarch64 snapshot on an x86_64 host fails immediately at the magic check, before any deserialization occurs.

The version string encodes the snapshot format version in semver. As of main (June 2026), SNAPSHOT_VERSION = Version::new(11, 0, 0) in src/vmm/src/persist.rs. The compatibility rule is strict: the MAJOR version of the snapshot and the running Firecracker binary must match; the snapshot's MINOR must be less than or equal to the binary's MINOR; any PATCH version is accepted. Because Firecracker uses the bitcode crate rather than bincode for serialization — bitcode does not support backward-compatible schema evolution — every structural change to MicrovmState bumps the MAJOR version. Snapshot format versions are independent of Firecracker release versions; a v1.8.0 binary may write a format version 9.x snapshot.

The trailing eight bytes are a CRC-64 over all preceding bytes. The validation property is that crc64(0, entire_file_bytes) == 0 when those bytes are the correct checksum. If the file is truncated, corrupted, or written by a different architecture, the check fails and restore is aborted. The deserializer enforces a 10 MB size limit (SNAPSHOT_DESERIALIZATION_BYTES_LIMIT = 10_000_000) to bound the cost of a malformed input.

The bitcode blob deserializes to a MicrovmState:

pub struct MicrovmState { pub vm_info: VmInfo, pub kvm_state: KvmState, pub vm_state: VmState, pub vcpu_states: Vec<VcpuState>, pub device_states: DevicesState, }

VmInfo carries the static configuration: mem_size_mib, SMT setting, CPU template, boot source, and huge page policy. KvmState records the KVM capability modifiers in effect when the snapshot was taken. VmState is architecture-specific: on x86_64 it includes IRQ routing tables, memory layout, and the kvmclock value. DevicesState covers the virtio block, net, and balloon devices; two device types are explicitly excluded from the snapshot — the serial emulation state and the vsock backend. Neither can be meaningfully resumed; vsock listen sockets survive with updated CIDs, but open connections are torn down.

Full vs. Diff Snapshots

The snapshot_type field in PUT /snapshot/create selects between two memory-file strategies. Both produce sparse files; they differ in what pages are written.

Full Snapshots

A full snapshot calls dump() in src/vmm/src/vstate/memory.rs. It iterates all memory slots in order. Plugged (active) slots are written sequentially via write_all_volatile. Unplugged slots — memory that has been balloon-removed from the guest — are skipped with SeekFrom::Current(slot_len_bytes), leaving a hole in the file at that range. The file is sparse: the kernel's page cache never populates those hole ranges, so the on-disk size reflects only the guest's active memory.

Diff Snapshots

A diff snapshot calls dump_dirty(). Instead of writing every plugged page, it unions two dirty-page sources and writes only the result.

The first source is the KVM dirty bitmap: KVM_GET_DIRTY_LOG returns a u64 array for each memory slot, where bit j of element i represents page (i * 64 + j). One ioctl call per slot. Clean pages are skipped with a seek forward; dirty pages are written at their correct file offset. The second source is Firecracker's internal AtomicBitmap tracking writes to virtio queue memory. KVM does not observe virtio queue activity directly at runtime — the device emulation layer in Firecracker does — so without the AtomicBitmap, virtio ring updates would silently go unrecorded.

After the diff file is written, Firecracker resets both sources: reset_dirty_bitmap() calls get_dirty_log() for each slot (clearing KVM's internal bitmap, not just reading it) and zeros the AtomicBitmap. It then calls mark_virtio_queue_memory_dirty() to pre-mark virtio queue memory dirty for the next diff cycle, since those pages will be written again before the next snapshot.

If track_dirty_pages was false at load time, KVM dirty-page logging was never enabled, and the diff path falls back to mincore(2) — the set of resident pages in memory — as its proxy for dirtiness. This fallback requires swap to be disabled; any swapped-out page would be misreported as clean.

When the output mem_file_path already exists and its size exactly matches mem_size_mib * 1024 * 1024, the diff patches only the dirty pages into that existing file. A size mismatch truncates and rewrites from scratch. An application can therefore take a diff snapshot after each phase of work and always merge into the same path without growing file count with depth.

flowchart LR
    A["PUT /snapshot/create\ntype=Diff"] --> B["dump_dirty()"]
    B --> C["KVM_GET_DIRTY_LOG\n(per memory slot)"]
    B --> D["AtomicBitmap\n(virtio queue writes)"]
    C --> E["union dirty set"]
    D --> E
    E --> F["write dirty pages\n(skip clean pages\nwith seek forward)"]
    F --> G["sparse mem_file_path"]
    G --> H["reset_dirty_bitmap()\nmark_virtio_queue_memory_dirty()"]

vCPU State: The Ioctl Sequence

Saving and restoring a vCPU is not a single ioctl. It is an ordered sequence where each step must precede the next because of field-level dependencies within KVM's internal state. The save and restore sequences in src/vmm/src/arch/x86_64/vcpu.rs make the ordering explicit.

Saving starts with KVM_GET_MP_STATE because that call triggers kvm_apic_accept_events() internally, which can modify the LAPIC state. If you read the LAPIC first, you may capture a stale value. Then come the general-purpose registers (KVM_GET_REGS), segment registers and control registers (KVM_GET_SREGS), extended processor state (KVM_GET_XSAVE), extended control registers (KVM_GET_XCRS), debug registers (KVM_GET_DEBUGREGS), and then the LAPIC (KVM_GET_LAPIC, kvm_lapic_state, 0x400 bytes). After the LAPIC: the TSC frequency via get_tsc_khz(), the CPUID configuration, and the MSRs in chunks via KVM_GET_MSRS. The sequence ends with KVM_GET_VCPU_EVENTS — pending exceptions, interrupts, NMIs, and SMIs — because it is affected by all of the preceding GETs and must come last to be accurate.

Restoring reverses the dependency direction. KVM_SET_CPUID2 comes first because the BSP identity inside set_mp_state depends on CPUID. Then KVM_SET_MP_STATE, then KVM_SET_REGS — which clears pending exceptions and therefore must precede KVM_SET_VCPU_EVENTS. KVM_SET_SREGS restores the APIC base MSR and must precede KVM_SET_LAPIC. After KVM_SET_XSAVE, KVM_SET_XCRS, and KVM_SET_DEBUGREGS, the LAPIC is set with KVM_SET_LAPIC — which must follow KVM_SET_SREGS (APIC base) and must precede KVM_SET_MSRS (TSC deadline MSR reads the LAPIC timer mode). The MSRs are set in chunks via KVM_SET_MSRS. The sequence ends with KVM_SET_VCPU_EVENTS, which requires all other vCPUs to be paused — a constraint that holds during restore because the microVM begins life in the paused state.

The Restore Flow

restore_from_snapshot() in src/vmm/src/persist.rs executes in a fixed order. It reads and deserializes MicrovmState, validating the magic ID, version compatibility, and CRC-64 in the process. It applies any network_overrides from the load request — these remap tap device names in the deserialized net device state, which is how a clone can be given a different host tap interface than the one the base snapshot used. It applies any vsock_override. It then calls snapshot_state_sanity_check(): at least one memory region must exist, at least one DRAM region must exist, and each DRAM region must have exactly one plugged slot.

Before proceeding to device reconstruction, Firecracker issues a cross-vendor warning. On x86_64 it calls get_vendor_id_from_host() and compares the host's CPU vendor string to the one recorded in the snapshot; on aarch64 it compares manufacturer IDs. It warns, rather than aborts, because some cross-model restores work in practice — but cross-architecture restores do not, and that failure mode is caught earlier by the magic ID.

With the sanity checks passed, Firecracker maps guest memory from the memory file and calls builder::build_microvm_from_snapshot() to reconstruct all devices from DevicesState, restore the KVM VM state, and run the per-vCPU restore sequence described above.

sequenceDiagram
    participant Op as operator
    participant FC as firecracker
    participant KVM as KVM
    participant Mem as guest memory

    Op->>FC: PUT /snapshot/load
    FC->>FC: read + validate MicrovmState
    FC->>FC: apply network_overrides
    FC->>FC: sanity_check()
    FC->>Mem: mmap(MAP_PRIVATE) or register UFFD
    FC->>KVM: rebuild devices, KVM_SET_USER_MEMORY_REGION
    loop per vCPU
        FC->>KVM: KVM_SET_CPUID2
        FC->>KVM: KVM_SET_MP_STATE
        FC->>KVM: KVM_SET_REGS / KVM_SET_SREGS / ...
        FC->>KVM: KVM_SET_VCPU_EVENTS
    end
    FC-->>Op: microVM paused, ready

Copy-on-Write Memory Backing

Guest memory is not copied into the restored microVM's address space. It is mapped. The choice of mapping strategy is the backend_type field in the mem_backend object of PUT /snapshot/load.

File Backend

With backend_type: "File", Firecracker creates a MAP_PRIVATE mapping of the guest memory file. Pages are loaded on demand: when a vCPU reads a guest physical address that has not been touched yet, the kernel page-faults in the corresponding page from the file's page cache. When that page is first written, the kernel allocates a private anonymous copy — the standard POSIX copy-on-write behavior. The snapshot file is never modified.

This is what makes fast clones cheap at the memory level. One base snapshot file, mapped MAP_PRIVATE by any number of restored microVM processes: all clones share the physical pages of the file via the page cache until each clone writes to them, at which point only the written pages escape into private allocation. The clone cost is proportional to the working set, not the total guest memory size.

UFFD Backend

With backend_type: "Uffd", Firecracker outsources page-fault handling to a separate process over a Unix domain socket. The flow is:

  1. Firecracker creates an anonymous mmap sized to the guest memory description.
  2. It registers that region with a userfaultfd file descriptor via UFFDIO_REGISTER with mode UFFDIO_REGISTER_MODE_MISSING.
  3. It connects to the handler process listening on the Unix socket named by backend_path, then sends the UFFD fd via SCM_RIGHTS ancillary data together with GuestRegionUffdMapping structs describing each guest memory region: base_host_virt_addr, size, offset, and page_size.
  4. The handler blocks on the UFFD fd reading UFFD_EVENT_PAGEFAULT events, and responds to each with an UFFDIO_COPY that copies the requested page from wherever the handler stores its backing data.

On kernels >= 6.1, the UFFD fd is created via the /dev/userfaultfd device and the USERFAULTFD_IOC_NEW ioctl. Firecracker v1.5.0 added this path as the preferred route, with a fallback to the userfaultfd(2) syscall on kernels that predate the device file.

The handler process can back pages from anywhere — a local file, a distributed in-memory cache, an S3 bucket — without any change to Firecracker. That pluggability is why AWS Lambda SnapStart uses the UFFD path. The Lambda snapshot handler divides the guest memory file into 512 KB chunks and serves them from a two-level hierarchy: an L1 worker-local cache at roughly 1 ms per chunk and an L2 availability-zone-wide distributed cache at single-digit milliseconds per chunk, with S3 as the ultimate fallback. This tiered fetch is invisible to the guest; from the microVM's perspective, the page just arrives when needed.

One operational hazard: if the UFFD handler process crashes while the microVM is running, Firecracker hangs indefinitely on the next page fault. There is no timeout and no fallback mechanism. The handler must be treated as a critical daemon, not a best-effort service. There is also a balloon interaction to handle: when the guest unplugs a memory range (balloon inflation), the UFFD subsystem emits UFFD_EVENT_REMOVE for that range. The handler must zero those pages; failing to do so leaves stale data accessible after the balloon is deflated again.

Fast Clones And Cold-Start Patterns

The snapshot-and-restore machinery was designed for cloning. The canonical pattern is:

Boot one microVM. Run the application through initialization: load the runtime, warm the JIT, prime the caches. Then pause the VM and take a full snapshot. That snapshot becomes the base image shared across all future instances. For each invocation, restore a new microVM from that base image using a MAP_PRIVATE file mapping. The clone's memory diverges from the base only as the invocation runs; unused pages are never faulted in.

At steady state, a fleet of clones shares the physical pages of a single base memory file through the page cache, with each clone contributing only the private anonymous pages it has dirtied. On an EC2 m5.12xlarge, Firecracker snapshot restore with the file backend takes as little as 4 ms — the dominant cost is not KVM setup but the latency of the first vCPU run after the memory mapping is established. Application-level cold start varies by runtime, from 20 ms to over 60 seconds unoptimized; the snapshot eliminates the portion spent in initialization.

The UFFD backend enables a further extension: transparent pre-warming. A handler that predicts the guest's access pattern can issue UFFDIO_COPY for likely-needed pages before the vCPU faults on them, masking fetch latency entirely.

Security Hazards

A snapshot is a frozen moment in a running machine's life. When that moment is resumed into multiple clones, every piece of state that looked unique at snapshot time is suddenly shared. The hazards are not theoretical: the arXiv paper by Brooker et al. (arXiv:2102.12892, 2021) documented successful TLS 1.0 attacks via snapshot reuse and catalogued five categories of cloned state: cryptographic key material, RNG seeds, session tokens, nonces, and UUID generators.

Entropy and the CSPRNG

The guest kernel's CSPRNG state is identical across all clones at the moment of restore. Divergence happens only through post-resume noise: timer jitter, interrupt timing, hardware RNG output. Until that noise accumulates, clones can produce identical key material, nonces, and session tokens.

Firecracker addresses this with VMGenID, an ACPI device (_HID = "VMGENCTR", _CID = "VM_Gen_Counter") that exposes a 128-bit generation counter in guest-physical memory. Firecracker added VMGenID support in v1.8.0 on x86_64. On every snapshot restore, Firecracker does three things: generates a fresh 128-bit value via aws_lc_rs::rand::fill(), writes it as little-endian bytes into the pre-allocated guest memory region, and triggers a Global System Interrupt to notify the guest. The ACPI path is _SB_.VGEN. The implementation lives in src/vmm/src/devices/acpi/vmgenid.rs.

The Linux VMGenID driver (drivers/virt/vmgenid.c) was merged in kernel 5.18 on x86_64. It detects the ID change and calls add_vmfork_randomness(), which calls crng_reseed(NULL) if the CSPRNG is already initialized, mixing in CPU HWRNG output (RDSEED/RDRAND) where available. The kernel logs "crng reseeded due to virtual machine fork" to confirm the reseed happened. On aarch64, the DeviceTree VMGenID binding arrived in kernel 6.10 — Firecracker supports guest kernels up to 6.1 on that architecture, so ARM users require a backported vmgenid patch.

VMGenID does not eliminate the hazard; it reduces the window. A race window exists between vCPU resumption and the VMGenID interrupt being processed. During that window, clones may produce identical output. Measuring the exact size of that window requires knowing when the guest kernel scheduler first runs the VMGenID driver's interrupt handler relative to any userspace code that calls getrandom(2).

For guest kernels older than 5.18 (or ARM kernels without the vmgenid backport), reseed must be done manually. The guest-side procedure requires CAP_SYS_ADMIN:

Note: The following ioctls require root inside the guest and a file descriptor opened on /dev/random.

Open /dev/random. Issue RNDADDENTROPY (_IOW('R', 0x03, int[2]), defined in include/uapi/linux/random.h) to mix entropy bytes into the input pool. Then issue RNDRESEEDCRNG (_IO('R', 0x07), also requiring CAP_SYS_ADMIN) to force an immediate CSPRNG reseed.

VMGenID does not reach userspace PRNGs: OpenSSL's DRBG, Go's math/rand, Java's SecureRandom, and similar language-level generators cache entropy and are not notified by the kernel reseed event. Applications that use those generators must implement their own clone-detection and reseed logic.

Three additional frozen items require attention per clone:

/proc/sys/kernel/random/boot_id is a UUID written at boot time and never changed. Some libraries treat it as a per-instance unique ID. On a clone, all instances share the same boot_id. The mitigation is to bind-mount a freshly generated UUID file over it after restore.

/var/lib/systemd/random-seed is read by systemd at startup to reseed the kernel pool. If all clones share the same seed file from the base image, they all replay the same initial seed. The mitigation is to delete the file before taking the base snapshot, so no seed file is replayed.

Userspace PRNG state cannot be addressed at the hypervisor level at all. Each application that holds cached entropy must detect the clone event — via the boot_id change or a side-channel agreed on at clone startup — and reseed independently.

Frozen Clocks

The guest kvmclock freezes at snapshot time. On restore, the clock resumes from the frozen value; wall-clock time spent between snapshot and restore is invisible to the guest.

The kvmclock ABI exposes this discontinuity explicitly. Each vCPU communicates a pvclock_vcpu_time_info struct to KVM via MSR_KVM_SYSTEM_TIME_NEW (defined in arch/x86/include/uapi/asm/kvm_para.h). The PVCLOCK_GUEST_STOPPED flag bit (0x2) in that struct's flags field signals that the vCPU was suspended. KVM sets this bit on restore, and the guest kernel (arch/x86/kernel/kvmclock.c) reads it to detect time discontinuities. Guest code that cares about monotonic time can check this flag and compensate; code that does not check it will see an apparent time jump at the first tick after resume.

Firecracker v1.16.0 changed the default restore behavior: prior to that release, Firecracker unconditionally advanced the kvmclock by the elapsed wall time, which caused the monotonic clock to jump forward abruptly on kernels >= 5.16. The fix changed the default to leave the kvmclock frozen at the snapshotted value. The clock_realtime field on PUT /snapshot/load (x86_64 only) opts back into the old behavior: when clock_realtime: true, Firecracker passes the KVM_CLOCK_REALTIME flag (1 << 2 = 0x4) in the flags field of the kvm_clock_data struct issued via KVM_SET_CLOCK (_IOW(KVMIO, 0x7b, struct kvm_clock_data), defined in include/uapi/linux/kvm.h), and KVM advances the guest kvmclock by the elapsed duration before the vCPUs start running.

Duplicated Tokens and Network State

Cloning is fundamentally incompatible with session-layer protocols that assume a single unique client identity. Firecracker closes vsock connections open at snapshot time when the snapshot is restored — the guest will observe a connection reset. Vsock listen sockets survive, with their CIDs updated. TCP connections inside the guest are not managed by Firecracker; they remain in the guest's network stack, where they will either time out or receive resets from the peer depending on whether the peer survived the snapshot interval.

The more serious problem is application-layer state cached in memory before the snapshot: TLS session tickets, OAuth tokens, signed cookies, HMAC keys, database connection credentials. Restoring the snapshot into two clones creates two clients presenting the same credentials to the same backends. Firecracker's network_overrides field remaps host tap device names between clones; it does not remap anything inside the guest.

The mitigations are application-level. The clone must either avoid caching credentials before the snapshot point, or must invalidate and re-acquire them as part of its resume initialization. The Lambda model solves this by snapshotting before the handler function is invoked — the function has not yet obtained any per-invocation credentials — and issuing fresh credentials to each clone via instance metadata.

Compatibility Constraints

Snapshot portability is more constrained than it appears. The magic ID enforces architecture separation: an x86_64 snapshot cannot be loaded on aarch64. Cross-CPU-vendor restore — AMD snapshot on an Intel host, or the reverse — is warned against via get_vendor_id_from_host() but not blocked; behavior depends on which instructions and MSRs were captured in the CPUID and MSR state. Cross-kernel-version restore is described in the documentation as "considered unstable," with one known-tested pair: Linux 5.10 host to 6.1 host on m5n, m6i, and m6a EC2 instance types.

On aarch64, the GIC (Generic Interrupt Controller) version becomes a constraint: snapshots cannot be restored across different GIC versions, because the interrupt controller state embedded in VmState is GIC-version-specific.

cgroups V1 causes unexpectedly high snapshot restoration latency. The documentation recommends cgroups V2 strongly. The mechanism is that cgroups V1's per-subsystem hierarchy introduces extra synchronization during process creation and mmap at restore time; V2's unified hierarchy avoids this.

The snapshot format MAJOR version must match between the snapshot file and the running Firecracker binary. Format version 11.0.0 was current as of June 2026. The versionize crate used in Firecracker's development preview period was archived on 2026-02-27 and no longer appears in the codebase.

Sources And Further Reading