Warm Pools: Pre-Booted VMs, Shared Snapshots, and Zero-RTT Networking

This is post 3 of a series. Post 1 covers the binary rewriter. Post 2 covers the VM architecture.

One VM is a demo. A thousand is a platform.

The previous post described a single VM — the shim, the memory layout, the hypervisor backstop. It boots in about a millisecond, uses a few megabytes, and can run a Python binary with full syscall-level visibility.

But a single VM isn’t useful in production. You need a pool of them — pre-booted, ready to accept work the instant a task arrives. And you need the networking to be fast, because the dominant workload is a process making HTTPS calls to LLM APIs, and every millisecond of connection setup is wasted time.

This post covers the pool daemon that manages pre-warmed VMs, how snapshots eliminate boot overhead, and how pre-established TCP connections give agents zero-RTT networking.

The pool daemon

The pool daemon (nexus-pool) is a host-side process that manages a fleet of pre-warmed VMM workers. On startup, it:

1. Loads a base snapshot into a memfd — a Python runtime that’s been booted to the point just before user code runs

2. Pre-faults all pages into RAM via MADV_WILLNEED — no page faults later

3. Spawns N worker processes, each a nexus-vmm instance in worker mode

4. Each worker inherits the memfd file descriptor — the kernel’s page cache means all workers share the same physical pages for the base snapshot

5. Pre-establishes warm TCP connections for configured destinations

6. Listens on a unix socket for incoming task requests

nexus-pool startup:

  memfd "nexus-base" (Python 3.12 snapshot, ~60MB)
  ├── sendfile: host file → memfd (kernel-space copy)
  ├── MADV_WILLNEED: pre-fault all pages
  └── All workers inherit this fd → shared page cache

  Worker 0: nexus-vmm --worker --memfd-base-fd=N --memfd-ring-fd=M
  Worker 1: nexus-vmm --worker --memfd-base-fd=N --memfd-ring-fd=M
  Worker 2: nexus-vmm --worker --memfd-base-fd=N --memfd-ring-fd=M
  ...
  Worker N: "READY" → sitting idle, waiting for a task

  Unix socket: /run/nexus.sock → accepting JSON requests

When a task arrives, the pool daemon picks an idle worker, sends it the task as JSON, and streams the output back to the client:

{
  "script": "/app/main.py",
  "argv": ["python", "-c", "print(42)"],
  "env": {"API_KEY": "sk-..."},
  "vfs": "stdlib.vfs"
}

The worker boots from the shared snapshot, loads the task-specific VFS files lazily, runs the script, and returns the output. From the client’s perspective, the task started almost instantly — the VM was already warm.

Snapshots: why boot when you can restore?

Booting a Python interpreter takes time. CPython’s startup sequence — importing site, loading encodings, initializing the import machinery — involves hundreds of syscalls and thousands of file reads. Even with the VFS cache, it takes ~200ms.

For a warm pool, that’s 200ms of wasted work repeated identically for every VM. The binary is the same. The standard library is the same. The initialization sequence is deterministic. There’s no reason to re-execute it.

Instead, the pool daemon boots Python once, takes a snapshot of the VM’s memory at the point where the interpreter is ready for user code, and uses that snapshot as the base for all workers. The snapshot is a memfd — an anonymous file in RAM — containing the full guest memory state: heap, stack, initialized globals, loaded modules, everything.

One-time snapshot creation:

  Boot CPython 3.12 → import site → import codecs → ...
  → snapshot point: PyRun_SimpleStringFlags
  → dump guest memory to memfd (sendfile, kernel-space)
  → pre-fault pages (MADV_WILLNEED)

Worker startup from snapshot:

  mmap(memfd) → KVM_SET_USER_MEMORY_REGION
  → all page table structures intact
  → all Python state intact
  → set RIP to snapshot point
  → KVM_RUN → immediately executing user code

  Time: ~microseconds (mmap + KVM setup, no Python boot)

Because the memfd is shared across all workers via the kernel’s page cache, the physical memory cost of the base snapshot is paid once, not per worker. Ten workers sharing a 60MB base snapshot use ~60MB total for the base, not 600MB. The copy-on-write semantics of mmap mean each worker only allocates private pages when it actually writes to them — which happens only for task-specific state (heap allocations, stack, task output).

The memory math

Memory accounting in a COW snapshot system is nuanced. The base snapshot is shared, but each worker dirties pages as it runs — heap allocations, stack growth, task-specific state. The ring memfds are per-worker but grow dynamically (they start near-empty and allocate ring pairs only as connections open). Here’s our best estimate based on what we observe in practice:

Base snapshot (shared):        ~19MB  (Python 3.12 static binary, pages shared
                                       via memfd page cache across all workers)
Per-worker private pages:      ~2-5MB (heap growth, stack, task-specific state;
                                       varies by workload)
Ring buffer memfd per worker:  ~0.5-1MB typical (2-4 active connections × 128KB/pair)
                               ~8MB worst case (all 64 ring pairs allocated)

These numbers are workload-dependent and we haven’t done rigorous measurement across a large fleet yet. What we can say with confidence:

• The base snapshot’s physical pages are shared — the kernel’s page cache ensures this when multiple processes mmap the same memfd

• Each worker’s COW pages are proportional to what it actually modifies, not the total address space size

• The ring memfd grows on demand — a worker with 2 connections pays ~256KB, not 8MB

The comparison to traditional containers is directional, not precise: Docker containers with Python typically consume 50-100MB each because each container loads its own copy of the Python runtime and standard library into separate address spaces. The shared snapshot eliminates that duplication.

The key insight: because the guest memory layout is deterministic (post 2), every worker starts from an identical state. There’s nothing to reconcile, no per-worker variation in the base image. This is what makes sharing work — you can’t share pages across VMs if each one might have different contents at the same address. But the exact savings depend on how much each task dirties — short tasks that read data and call an API dirty very little; long-running tasks that build large data structures dirty more.

Zero-RTT networking: warm TCP connections

A typical agent task looks like:

1. Read input data

2. Call an LLM API over HTTPS

3. Write output

Step 2 dominates the wall-clock time. But before the first API call, the process needs to: - Resolve DNS (~5-20ms) - TCP connect (~10-50ms depending on destination) - TLS handshake (~50-150ms — two round trips for TLS 1.3)

That’s 65-220ms of overhead before the first byte of the API request is sent. For a short agent task, that overhead can be larger than the actual computation.

The pool daemon eliminates this by pre-establishing TCP connections at startup. When the pool daemon starts, it connects to configured destinations and holds the sockets open:

Pool startup:

  --warm=api.anthropic.com:443
  --warm=api.openai.com:443

  For each destination:
    socket() → connect() → TCP established → idle socket ready
    Allocate ring pair in the worker's ring memfd
    Write warm pool table entry (dst_ip, dst_port, ring_idx)
    Register socket fd with io_uring net engine

When the guest process calls connect("api.anthropic.com", 443), the shim doesn’t go through the hypervisor at all. It checks the warm pool table — a 32-entry lookup in the first page of the ring memfd:

// shim/src/warm_pool.rs — zero VM exit connection lookup
pub fn lookup(dst_ip: u32, dst_port: u16) -> Option<u8> {
    let base = NEXUS_WARM_POOL_TABLE as *const u8;
    for i in 0..NEXUS_WARM_POOL_MAX_ENTRIES {
        let entry = unsafe { base.add(i * 24) };
        let ip = unsafe { core::ptr::read(entry as *const u32) };
        let port = unsafe { core::ptr::read(entry.add(4) as *const u16) };
        if ip == dst_ip && port == dst_port {
            let ring_idx = unsafe { *entry.add(6) };
            if ring_idx != 0 {
                return Some(ring_idx);
            }
        }
    }
    None
}

If the destination matches a warm pool entry, the shim returns the pre-allocated ring index. The process now has a “connected” socket that maps to a ring buffer pair — TX and RX — backed by a real TCP connection the pool daemon already established. No DNS lookup. No TCP handshake. Zero VM exits.

For TLS destinations, there’s a way to eliminate the remote TLS handshake cost entirely — a topic for a later post in this series.

The first send() writes directly into the TX ring buffer. The pool daemon’s io_uring net engine drains the TX ring and pushes the data over the pre-established socket. Incoming data lands in the RX ring, and the guest’s recv() reads it out.

Without warm pool:

StepLatencyDNS lookup5-20msTCP connect10-50msTLS handshake50-150msTotal65-220ms

With warm pool:

StepLatencyShim table lookup~nanosecondsTLS handshake50-150msTotal50-150msSaved15-70ms (DNS + TCP)

The TLS handshake remains the dominant cost on this path. We have a design that eliminates it — more on that in a later post.

The io_uring net engine

The pool daemon runs a dedicated thread — the net engine — that handles all network I/O across all workers using io_uring:

Net engine thread (single core, hot loop):

  loop {
      // Scan all jails' ring buffers
      for jail in &registry {
          for ring_pair in &jail.rings {
              if ring_pair.tx has data:
                  submit io_uring send(sock_fd, data)
              if ring_pair.rx has space:
                  submit io_uring recv(sock_fd, space)
          }
      }
      // Drain completions
      for completion in io_uring.drain() {
          match completion {
              send_done → advance tx_tail
              recv_done → advance rx_head
              EOF → reconnect if warm dest configured
          }
      }
  }

One thread, one io_uring instance, handling all network I/O for all VMs. No per-VM network threads. No per-connection event loops. The io_uring submission queue batches all pending sends and receives across all workers, and the kernel processes them in a single system call.

This is where the “no TCP stack in the guest” design from post 2 pays off at scale. Each guest thinks it’s making socket calls. The shim translates them to ring buffer operations. The net engine translates ring buffer operations to real TCP I/O via io_uring. The guest never touches a socket. The net engine never touches guest memory. The ring buffer is the clean interface between the two.

When a warm connection drops (server closes the socket, network error), the net engine detects the EOF in the io_uring completion and automatically reconnects to the configured destination. The guest sees a brief stall on the next recv() while the reconnection happens, but doesn’t crash — the ring buffer just runs dry temporarily.

Cold connections

Not every destination is in the warm pool. When the guest calls connect() on an address that isn’t pre-established, the shim escalates to the hypervisor via the governance mailbox. The hypervisor performs the real connect() on the host, allocates a ring pair, registers the socket with the net engine, and returns a ring index to the shim.

This “cold” path is slower — it pays the full TCP handshake cost plus a VM exit. But it only happens once per destination. After the connection is established, all I/O flows through the ring buffer at the same speed as warm connections.

The pool daemon can also dynamically register cold-connected sockets into the net engine. When a worker’s hypervisor establishes a cold connection, it sends an IPC message to the pool daemon: “I have a new socket fd for this worker’s ring pair N.” The pool daemon adds it to the io_uring polling set. From that point on, the connection is indistinguishable from a warm one.

The unix socket API

The pool daemon exposes a simple JSON-over-unix-socket interface. No HTTP server, no gRPC, no framework:

# Submit a task
echo '{"script":"/app/analyze.py","env":{"INPUT":"/data/q3.csv"}}' \
  | socat - UNIX-CONNECT:/run/nexus.sock

# Response: streamed stdout from the agent, then EOF

This is deliberately minimal. The pool daemon isn’t a platform — it’s a process manager that happens to manage VMs. The unix socket keeps the API surface small and the deployment simple. A Kubernetes operator, a systemd unit, or a custom orchestrator can all talk to the same socket.

The Kubernetes integration — a CRD for task definitions, a DaemonSet for the pool daemon, scheduler awareness of warm pool capacity — is future work. The unix socket is the interface it will build on.

---

What’s next

We have a pool of VMs that boot instantly from shared snapshots, connect to APIs without network setup overhead, and handle all I/O through lock-free ring buffers drained by a single io_uring thread. The next question is: what can’t this system do?

Tomorrow’s post is the honest tradeoffs — what we gave up, what’s more expensive than in a traditional container, and why the constraints are invisible for the workloads we’re targeting.

---

This is post 3 of a series on building a minimal VM runtime. Subscribe to get the rest.

If you have questions or want to discuss — reach out on LinkedIn.