# Sandbox bench

> Sandbox bench: cloud sandbox providers for AI agents, timed through one SDK (Sandbox SDK, built by us; 0.2.0 is public on npm and the 1.0 used here will be published soon) from one server in Florida, US, on two tracks: the same 4 vCPU, 8 GB machine for every provider (the fair comparison, and the one the site opens on), and each provider's default machine. Medians, p95s and 95% confidence intervals per provider; every provider-side failure kept, failures caused by our own code set aside.

Providers (id → name): e2b → E2B, freestyle → Freestyle, tenki → Tenki, boxd → boxd, superserve → Superserve, upstash → Upstash Box, daytona → Daytona, upstash-ephemeral → Upstash Ephemeral Box, boat → Boat.

## Tracks

- `same-spec`: Same machine: 4 vCPU, 8 GB. Every provider asked for Boat's machine, 4 vCPU and 8 GB, or its nearest size.
- `default`: Default machines. Each provider's default sandbox, so machine sizes differ.

The page shows one track at a time: `/` opens the same-machine track and `/?track=default` the default-machines track.

## Quick picks

### Same machine: 4 vCPU, 8 GB

- Fastest cold start: Superserve 0.52 s (runner-up Upstash Ephemeral Box 0.68 s)
- Fastest agent work (tool replay): Daytona 12 s (runner-up Superserve 13 s)
- Fastest 25-sandbox fan-out: Superserve 0.79 s (runner-up E2B 1.1 s)
- Fastest reconnect: Daytona 0.30 s (runner-up Superserve 0.33 s)
- Cheapest task: Upstash Ephemeral Box $0.00026 per task, $0.26 per 1,000 (billed) (runner-up Upstash Box $0.00034 per task, $0.34 per 1,000 (billed))
- Most reliable: E2B, Tenki, Superserve, Upstash Box, Daytona, Upstash Ephemeral Box, Boat 100.00% (tied) (runner-up Freestyle 99.91%)

### Default machines

- Fastest cold start: E2B 0.50 s (runner-up Superserve 0.53 s)
- Fastest agent work (tool replay): Superserve 13 s (runner-up Daytona 15 s)
- Fastest 25-sandbox fan-out: Superserve 0.75 s (runner-up E2B 0.87 s)
- Fastest reconnect: Daytona 0.31 s (runner-up Superserve 0.32 s)
- Cheapest task: Upstash Ephemeral Box $0.00013 per task, $0.13 per 1,000 (billed) (runner-up Upstash Box $0.00017 per task, $0.17 per 1,000 (billed))
- Most reliable: E2B, Tenki, Superserve, Upstash Box, Daytona, Boat 100.00% (tied) (runner-up boxd 99.81%)

## Coming in the next update

Any snapshot or burst a chart marks as coming. Every result, cold start, SDK setup, snapshots, bursts and cost included, comes from the runs on our faster SDK (commit 1185278); Superserve now runs the same-machine track at 4 vCPU. Pause and resume are from the previous run (SDK a2316c3), because the new main runs skipped them. Daytona was re-collected on Oct 22 after its account quota was raised from 10 vCPU / 10 GiB; earlier Daytona numbers were limited by that quota. Upstash appears twice because Box (long-lived, can pause) and Ephemeral Box (short-lived, for one-off agent jobs) are separate products. A provider gets a second row only for a product it sells separately, not for a setting on the same product. Ephemeral Box has no pause, so it has no pause result. Daytona's billed cost can still change for up to 48 hours. Freestyle, Tenki and Boat costs are list-price estimates on our measured sandbox lifetime, because none gives a per-sandbox bill (Boat drops it when a sandbox is deleted). A burst size is published only when all three bursts finished. Failures caused by our own code are set aside, not counted.

## Data

- [results.json](/results.json): both tracks. Per provider: what we asked for and measured (allocation, equalSpec), cost per task with its kind (billed, metered, estimate), source and confidence, SDK-ready time, prices, regions and failures. Per metric: median, p95, attempts, failed and status. Start here.
- [results.md](/results.md): the same numbers as Markdown tables, one section per track.

## How to read it

Pick a track first: `tracks["same-spec"]` asks every provider for 4 vCPU and 8 GB, so prefer it for comparisons; `tracks.default` compares each provider's default sandbox, so sizes differ. In same-spec, `allocation.equalSpec: false` means the machine measured inside was not 4 vCPU, 8 GB, so compare that provider with care. median and p95 are in `unit`; `better` says which direction wins; results are sorted best first. attempts counts every try, failed included. A null median means the provider does not support the test, every attempt failed, or the result is excluded (see `status`). Cold start is the provider's own time, create() until the first command; Our SDK's setup after create is not included; on the faster SDK it measured 0.1 to 0.3 ms median per provider (20 creates each, separate run). `costPerTask.kind` is billed (provider charge), metered (provider usage at published rates) or estimate (list price, where billing gives no amount); a task is one sandbox that is created, runs the 100-call tool replay and is deleted, from a separate cost run: never compare them as if they were the same kind without saying so.

Prices cover compute only at list rates; disk, transfer, subscriptions and credits are excluded. Every call starts in Florida, so providers hosted in Europe (Boat in France, boxd in Germany) pay a transatlantic round trip per call. In the same-machine track, usable memory measured inside sits a little under 8 GB; equalSpec allows 7.6 to 8.4 GiB. Amounts are US dollars unless marked EUR.

## Metrics

- Start-up: Cold start (`cold`)
- Agent work: Agent tool replay (`tool-replay`), npm install (`npm-install`), pip install (`pip-install`), git clone (`git-clone`), Dev server ready (`dev-server`), Preview URL first byte (`preview-first-byte`), Edit, rebuild, reload (`edit-loop`)
- Everyday calls: Run a command (`command`), Write + read 1 KB (`files-1kb`), Write + read 10 MB (`files-10mb`), Reconnect (`reconnect`), Destroy (`destroy`)
- Pause and snapshots: Pause + resume (`pause-resume`), Take snapshot (`snapshot`), Start from snapshot (`create-from-snapshot`)
- Bursts: 10 at once, all ready (`burst-10`), 25 at once, all ready (`burst-25`)
- Machine speed: CPU, one core (`cpu-single`), CPU, all cores (`cpu-all`), Memory bandwidth (`memory`), Disk write (`disk-write`), Disk read (`disk-read`), 4K random write (`iops-write`), 4K random read (`iops-read`)