1,500 Real Apps on One Box — and Where Docker and Firecracker Tap Out
1,500 Real Apps on One Box — and Where Docker and Firecracker Tap Out
This is post 5 of a series.
The numbers post was 1,500 Python agents on a single AWS c6i.metal — each in its own KVM virtual machine, warmed in 0.42s, 10.8 agents per GB. That was the proof the model works at scale.
It was also the easy case, and I knew it at the time. A Python agent is about the friendliest thing you can ask a sandbox to run. So this post is two things I’d been putting off: pointing the runtime at real software instead of a function, and — because it’s the comparison everyone runs in their head — putting it side by side with Docker and Firecracker on the same box.
To get there I had to do the harder thing first.
Python was the forgiving part
The real test of a sandbox architecture isn’t the first runtime you run. It’s the second.
The binary-rewriter post ended on a clean idea: every path to the kernel converges on one two-byte instruction, 0F 05, so we rewrite that instruction everywhere and own the process. That post even showed the backstop for code that doesn’t exist at rewrite time — JIT’d syscall instructions caught by an LSTAR handler that patches them in place on first execution. That was all true. It was also all measured on Python, and Python let one assumption hide in plain sight: one thread of execution.
Because of the global interpreter lock, a Python agent is effectively single-threaded. It doesn’t generate much machine code as it runs. It boots, imports, settles into a loop. One thread, one vCPU, one snapshot taken at one clean line. Node questions every one of those on the first line, and the assumptions don’t break gently — they break in ways that turn a one-line trick into a real piece of engineering. Three of them are worth walking through.
The self-healing patch, now with other threads watching
The LSTAR trick from that post catches a JIT’d syscall and rewrites it in place — overwrite 0F 05 with CC 90 (INT3 + NOP) so every future execution takes the fast trap instead of the slow LSTAR round-trip. On one thread that’s clean: the instruction that trapped isn’t running anywhere else, so we patch it and move on.
V8 has a dozen threads. Its background compilers emit code on their own threads while request threads execute it. So the moment we reach to patch a syscall, another core may be a few bytes away from fetching the very instruction we’re about to overwrite — and on x86 that’s cross-modifying code, the case the architecture manuals warn about, where one core writes bytes another core is mid-fetch on. Write the two bytes naively and the other core can fetch a half-patched instruction: the new 0xCC and the old second byte. That’s a corrupt instruction and a dead VM, and it shows up once in a thousand runs, which is the worst way for it to show up.
The fix follows the shape the CPU actually guarantees. INT3 is one byte (0xCC), and a single-byte write is atomic with respect to instruction fetch — any core fetching that address sees either the old byte or the new one, never a tear. So we patch the first byte to 0xCC first. The instant that write lands, every path through that address traps cleanly to the same INT3 handler; the second byte doesn’t matter yet, because execution never gets past the trap. Then we fix up the rest. The order is the whole trick: make the trap appear atomically, repair the instruction behind it at leisure. One thread or twelve, the first execution takes the slow path and every one after takes the fast path — and no core ever fetches a torn instruction.
Threads are vCPUs, so the shim is now concurrent
A Python agent was one vCPU. A Node process is a dozen kernel threads, and each one is a real vCPU in the guest — which means the shim that catches all those traps is suddenly handling genuine concurrency. Two threads can be inside a syscall at the same time. One thread can be blocked in recv() while another needs to allocate memory and a third hits a futex.
That breaks the simplest possible shim — a single handler that assumes it runs to completion. A blocking syscall can’t just spin the vCPU it’s on, or every other thread sharing the runtime stalls behind it. So the blocking work moves off the trap path: when a thread hits something that has to wait — a recv with no data, a futex that isn’t ready — the shim parks that vCPU and lets the others run, and a host-side engine wakes it when its data or its lock arrives. The trap handler stays short and non-blocking; the waiting happens somewhere it doesn’t cost anything. None of this exists for a single-threaded interpreter. All of it is mandatory the first time two guest threads run at once.
Freezing a process that’s already a dozen threads
This is the one that took the longest. Everything good in the warm-pool post — restore in microseconds, share the base image copy-on-write across the whole fleet — depends on the snapshot being one consistent thing. For Python that’s free: one thread parked at one instruction, dump its memory, done.
Node is multi-threaded before user code ever runs. By the time it’s ready to serve, V8’s compiler and GC threads and libuv’s I/O pool are all live. To snapshot that, we can’t catch one thread at a clean line — we have to bring every thread to a stop at the same instant, capture each one’s registers and stack, and resume them all so coherently the application can’t tell time stopped: no lock half-handed-off, no thread that believes it’s mid-syscall while its partner has moved on. And the instant has to be the right one — after the runtime has finished assembling itself, before it has done any real work. Finding that point took real work, it’s a different point for every runtime, and getting it wrong doesn’t crash loudly — it leaves a snapshot that restores into a process that’s subtly, intermittently wrong. That’s the difference between “freeze a function” and “freeze a server,” and it’s most of why a real app took as long as it did.
The edge case that nearly killed us: the wake-up that wasn’t in the snapshot
Here’s the one I’d put next to the binary-rewriter post’s LLVM story.
n8n shells out — it spawns child processes to run tasks. The first time a freshly restored n8n tried to, it hung. Not crashed — hung, deep inside libuv, waiting on a lock that would never be released. It reproduced every time, which at least made it honest.
libuv coordinates signals across threads with an old Unix trick: a self-pipe. A thread that needs to wake the event loop writes a byte into a pipe; the loop is watching the read end, sees the byte, wakes up. It’s the plumbing under child_process, and on a normal process it’s invisible.
Our snapshot captured the guest — memory, threads, registers, file descriptors. But a byte sitting in flight inside that pipe isn’t in the guest. In our design the pipe is mediated host-side, in the runtime’s own plumbing, so the wake-up byte lived outside the guest entirely — the very kind of state we’d deliberately pushed out of the image. We froze n8n a hair after a thread had written that byte but before the loop had drained it, and the snapshot had no idea the plumbing was holding anything. On restore, the byte was gone. The loop woke for a signal that, as far as it could tell, had never been sent; the lock it was supposed to release stayed held; the next spawn waited on it forever.
The bug wasn’t in the freeze logic or the threading. It was a definition-of-”state” bug: we’d been treating a process as its memory and its registers, and a live multi-threaded process is also the bytes in flight in its own plumbing. The fix was to make the in-flight pipe contents part of what a snapshot is — capture them on freeze, replay them on restore — so the loop wakes for the signal it was actually sent. A function never has bytes in flight when you freeze it. A server almost always does.
Why the second one is the expensive one
That’s the tax, and the reason the third runtime is cheap. The first multi-threaded runtime is where we built the whole substrate — concurrent shim, parked vCPUs, coherent multi-threaded freeze, in-flight state in the snapshot. The second mostly reuses it. A typical async Rust service lands on the same threading, the same parked-vCPU machinery, the same snapshot model Node already forced us to build — and it skips the two hardest parts, because it’s compiled ahead of time (no JIT writing syscalls we never saw) and links nearly statically (almost no library surface). Python proved the architecture. Node proved it was an architecture and not a Python loader. Rust would mostly be filling in a table.
A real application, not a function
A runtime is still not an application. So I went one step further, to n8n — a full workflow-automation server. Not a function someone rewrote to fit a platform: a real program, multi-threaded, with a database, schema migrations, and an HTTP editor, the kind of thing a company actually self-hosts.
The trick that makes a function cheap — boot it once, freeze it, fan out copies — has to survive a process that’s a dozen live threads with an open database connection and a bound listening socket. Getting n8n to start once, freeze it at the exact instant it’s serving, and restore that frozen-but-live image so it answers HTTP without re-initializing — that’s what turns “1,500 Python functions” into “1,500 real apps.” Once that worked, the obvious thing to do was run a lot of them, three different ways.
The number I threw out
My first comparison said we were ~5× lighter per instance than Firecracker. It was wrong, and it’s worth saying why, because it’s the mistake every density benchmark makes.
I’d measured Firecracker running a heavier build of n8n than ours, and measured it 30 seconds after restore — long enough for the guest to dirty memory — against one of our instances measured 3 seconds in, while it was barely warm. Different app, different moment. Same build, same settled state, and the gap collapsed.
Here it is — same n8n, same box, measured the same way, once everything has settled:
Per instance, the memory is a wash. We are not dramatically lighter than a container running the same app, and anyone who tells you their VM is hasn’t measured it twice. I’m leading with that because the interesting result isn’t in this table — it’s the column you normally have to give up. Container-class memory almost always means a shared kernel. Here it doesn’t: every one of those instances is a real VM with a hardware boundary around it. The question that matters isn’t how big one instance is. It’s what happens at fifteen hundred.
What happens at 1,500
I tried to launch 1,500 of each, same box, same n8n. One of the three got there.
Ours: 1,500. All restored, all serving their editor, in about 32 seconds. 113 GB of RAM — and about 101 megabytes of disk, total, for the whole fleet.
Docker: stalled at ~861. Not memory — ~100 GB used, plenty free. Not the database — the containers that came up served fine. It ran out of disk. Every container gets its own writable layer, and ~861 of them filled the root volume: no space left on device, and no more containers.
Firecracker: stalled at 1,023. Every micro-VM needs its own network interface on the host. The 1,024th tap couldn’t attach to the bridge — RTNETLINK answers: Exchange full, the Linux 1024-port bridge limit. At 1,023 VMs it was sitting at 88 GB, nowhere near a memory wall. It hit an interface wall first.
The shape is the point. Both alternatives ran out before 1,500, on different things — Docker on per-container disk, Firecracker on per-VM host networking — and neither ran out of memory. They ran out of the overhead that’s private to each instance.
That overhead is exactly what the runtime shares away, using seams the earlier posts already built. Our instances share one read-only copy of n8n and its libraries (the snapshot from the warm-pool post); an instance’s writable data is hardlinked from a single template, so per-instance disk is essentially zero — 1,500 in 101 MB instead of 1,500 separate trees. And there are no host taps at all: the runtime already sits in every instance’s network path — the same seam that holds the DNS cache and TLS sessions in the numbers post — so the host needs zero interfaces per VM instead of one.
One thing worth being direct about: both walls are tunable. Give Docker a bigger disk and it goes further; put Firecracker on routed interfaces instead of one bridge and it gets past 1,024. I’m not claiming they can’t reach 1,500. I’m saying the per-instance overhead is real, it bites by default, and sharing it away is what lets a fleet this size fit on one ordinary box.
The 32 seconds, and why there’s no warm pool
The other number in there is the 32 seconds, and it’s a different mechanism doing different work.
Docker and Firecracker were cold-booting 1,500 instances — creating each one, then running n8n’s full startup inside it: load Node, connect the database, run migrations, build the editor. Fifteen hundred times. We pay that startup once, at bake time. After that, every instance is a restore of an already-serving image — it comes back past startup, answering HTTP in about 25 milliseconds, already at the editor.
It comes back serving, not just booted, for a specific reason: the guest has no TCP stack, so its sockets live host-side in the pool, not inside the image we froze. Firecracker can’t do that — its network state sits in a real guest kernel, travels with the snapshot, and is stale on restore, so every restored clone has to rebuild its connections by hand and pay the handshake round-trips to do it. Ours pays none, because the pool was already holding the connection warm — the zero-RTT path from the warm-pool post. You can bring a Firecracker VM back; it comes back needing to reconnect.
That 25 ms changes the economics more than the memory does. The usual way to hide a cold start is a warm pool — a stack of idle-but-running instances kept around in case traffic shows up — and how big it has to be is just Little’s law: the arrival rate times how long one takes to bring up. Bring-up time is the whole lever, and it’s a per-instance number, not the aggregate launch time. For n8n, cold-booting a container means paying that full startup again — call it twenty to thirty seconds; restoring an already-serving image is about twenty-five milliseconds. Same traffic, same formula, and the ratio between those two numbers is the multiple of idle capacity you’re carrying for nothing. I’ll let you do that division. At twenty-five milliseconds the pool you’d need rounds away: you keep one shared image resident, restore on demand, and the warm pool of N collapses to a single shared base.
Once an idle instance costs essentially nothing, you stop sizing a box for total jobs and start sizing it for the number running at once. Agent work is bursty — most of a job’s wall-clock is spent waiting on an LLM, on I/O, on a person. If peak concurrency is a third of your total jobs, that’s three times the jobs on the same hardware.
I want to be careful here, because this is the one claim in the post I haven’t measured end to end. The 25 ms restore is measured. The idle cost collapsing to one shared image is measured. The 3–4× is a projection that depends entirely on the workload — if everything peaks at once, you get 1×. It’s a forecast resting on measured parts, and I’d rather say that than imply otherwise.
What it adds up to
The numbers post ended on a claim: KVM isolation doesn’t have to mean heavy. This is the same claim taken to its conclusion with real software instead of toy agents:
Per instance, a VM costs about what a container costs — ~100 MB, same app.
A box that fits 1,500 of those VMs fits ~860 Docker containers and ~1,020 Firecracker VMs of the same app, because the private overhead per instance — disk, host interfaces — runs out first, and that’s the part we share away.
Each instance is serving in 25 milliseconds, which means no warm pool, which means you provision for peak concurrency instead of total jobs.
Look again at where the other two tapped out — Docker on disk, Firecracker on host interfaces — and it’s the same wall twice: state stuck inside the instance, its files in one case, its network in the other. The early posts were spent moving that state out of the guest, for density and to put the runtime in the path; what’s left to snapshot is memory, registers, and threads, nothing that goes stale when you copy it. The same thin guest is why it’s both unusually dense and unusually easy to freeze while it’s serving.
None of these are separate systems bolted together. It’s one runtime that already sits in every instance’s path — the same seam that does the isolation does the sharing, the restore, and the network mediation. The density is what makes the isolation affordable. The isolation is the reason to bother.
At 1,500 it’s a clean experiment. At 10,000 it’s the difference between one box and three.
But fitting more instances on a box is only half the story, and I want to be straight about which half this post is. It proves the launch side — denser, faster to serving, cheaper to keep idle. It does not prove you get more work done. The number that actually decides cost is throughput per dollar: how much of a real workload finishes under a fixed budget.
That’s the next experiment, and it’s why the next app is Ray. Give Ray the same CPU and memory budget two ways — on containers, and on this runtime — hand both the same series of tasks, and measure which one finishes first, or finishes the same work for less. Launch is the setup; throughput-per-budget is the proof. (Ray is also the real test of whether “the second runtime is free” holds when the workload is a cluster instead of a single process.) The deployment story — all of this as ordinary Kubernetes resources — comes after that.
If you’re running real apps or agents in production and any of this is a live problem — reach out on LinkedIn.