Replaced CI/CD with Sandboxes

I moved our CI onto Sandbox. One job per VM, the VM deletes itself in ~1s, and a 50-subrequest cap on Cloudflare’s free plan designed half the system.

Our CI ran on GitHub-hosted runners. A standard runner gives you 2 vCPU and 7 GB of RAM. A Go monorepo build fits in that the way a couch fits in a Civic. Bigger runners exist, and they bill by the minute.

We already run a Firecracker microVM fleet in-house: about 30 ms to boot a VM, 0.21 s from create to first exec. So I pointed GitHub Actions at it.

One rule, set on day one: one job, one Sandbox, then destroy it. No pool, no reuse, nothing carried between jobs.

Thirty days later the controller runs in production, multi-tenant, across multiple regions. Here is what it took.

The shape of it

workflow_job (queued) → create sandbox → launch runner
                      → job runs → VM deletes itself

Enter fullscreen mode Exit fullscreen mode

GitHub’s workflow_job webhook carries a JIT runner config: a single-use token, so there is no registration token to rotate or leak. The controller is a Cloudflare Worker, roughly 3,500 lines of TypeScript, with one Durable Object holding job rows in SQLite. In 30 days it has handled 39,775 invocations and issued 904,397 subrequests to GitHub and the VM API.

const sandbox = await client.createSandbox({
  template: "ghar-runner",
  shape: "s-4vcpu-4gb",
  envs: { JIT_CONFIG: encodedJitConfig },
  disk_mib: 10240,
});
await sandbox.runCommand("/opt/start-runner.sh", { detached: true });

Enter fullscreen mode Exit fullscreen mode

Every VM call goes through @nodeops-createos/sandbox, the TypeScript SDK we wrote for our own Sandbox fleet. Zero runtime dependencies, which is why it runs inside a Cloudflare Worker at all: no crypto, no fs, nothing that assumes Node. That mattered more than it sounds. An SDK that pulls in a transitive dependency tree does not deploy to an edge runtime, and I would have been writing raw fetch calls with hand-rolled retries instead.

Queued to running takes about a second: the mint is warm, the VM boots in ~30 ms, and the launch post is the tail. The create phase dominates what is left.

Self-delete is the whole trick

Every VM runs a guest agent on 127.0.0.1:1029. The last line of start-runner.sh:

curl -s -X POST http://127.0.0.1:1029/self/delete

Enter fullscreen mode Exit fullscreen mode

The runner exits, the VM asks the host to destroy it, and it is gone in about 1 second. No webhook, no Worker CPU, no polling loop. Over 99% of teardowns take this path.

Two more paths exist because I don’t trust one path with money. The completed webhook is the normal fallback. A cron reconciler is the backstop: it reaps VMs whose runner isn’t online, re-drives jobs still sitting in queued, deletes orphaned runner registrations, and sweeps sandboxes nobody owns.

That last sweep runs even when GitHub is unreachable. A leaked VM burns capacity whatever GitHub’s status page says.

Picking a size per job

A bare createos label sent every job to the same 4 vCPU box. Lint jobs and Go builds are not the same workload, so labels now carry the shape:

runs-on: createos-8c-16g   # → 8 vCPU, 16 GB
runs-on: createos-2c-4g    # → 2 vCPU, 4 GB

Enter fullscreen mode Exit fullscreen mode

The controller reads the shape catalog, caches it for 5 minutes, and coalesces concurrent fetches, so a 20-variant matrix costs one catalog call instead of twenty.

Two things that cost me a day each

/dev/fd does not exist. The microVM’s devtmpfs shadows the image’s /dev, so bash process substitution <(…) failed outright. start-runner.sh now symlinks /dev/fd to /proc/self/fd before the runner starts.

actions/setup-python reads /etc/os-release. On Debian 12 it hard-fails: The version '3.12' was not found for debian 12. The fix is a lie: rewrite /etc/os-release to claim Ubuntu 22.04, and keep the truth at /etc/os-release.debian. Python 3.11, 3.12 and 3.13 have resolved cleanly since.

Thirty days in production

Straight off Cloudflare’s analytics API, Jul 5 to Aug 3 2026:

Controller invocations 39,775 Subrequests issued to GitHub and the VM API 904,397 Days with traffic 30 of 30 Busiest day 3,964 (Jul 28) Average subrequests per invocation 22.7 Workflows 185 Jobs 822

Constraints did the design work

A request is capped at 50 subrequests. That 22.7 average is what I watch, and the ceiling produced most of the good parts: the recovery scan got a budget and a round-robin cursor in SQLite, so no repo starves and no tick blows the cap. The shape catalog got a cache. Token minting got a warm credential registry, cutting 0.4-0.9 s of RSA signing to near zero on reuse.

One rule I’d keep on any project: warn on every bound. A truncated listRunners response reads as “no runners online”, which makes the sweeper destroy live VMs. It throws now instead of warning.

If your GitHub-hosted runners have stopped being enough and you want a bigger base runner, write to [email protected].