Benchmarks
Fast is a feature.
Gray is a single native binary with 151,582 lines of Rust behind it. Here's what that buys next to the agent CLIs on this machine — measured October 2026, linux x86_64 · warm page cache.
| Tool | Version | Cold --version | Install size | Peak RSS |
|---|---|---|---|---|
| ● gray | 0.2.2 | 24ms | 37 MB | 12 MB |
| claude | 2.1.304 | 62ms | 276 MB | 36 MB |
| devin | 3000.11.92 | 62ms | 192 MB | 13 MB |
| codex | 0.162.1 | 318ms | 432 MB | 49 MB |
| pi | 1.0.4 | 631ms | 166 MB | 103 MB |
| opencode | 1.18.35 | 1.52s | 178 MB | 176 MB |
| cline | 3.0.70 | 2.88s | 407 MB | 212 MB |
median of 25 runs · lower is better · versions as installed
Prompt → first byte, vs a local mock
How much time the tool itself burns before the model can speak: every row is the median of nine runs against a mock OpenAI-compatible endpoint on localhost that streams instantly — no model latency in the number. cline, pi, opencode, devin and cursor don't expose a plain OpenAI-compatible base URL on this build, so they can't be timed against a local mock.
| Tool | Spawn → first byte | Spawn → done | ||
|---|---|---|---|---|
| ● gray | 124ms | 246ms | ||
| codex | 10.56s | 10.56s |
How we measure
cold start
for i in $(seq 25); do t=$(date +%s%N); $TOOL --version; echo $((($(date +%s%N)-t)/1000000))ms; done | sort -n | median
install size
du -sb $(readlink -f $(command -v $TOOL)) # binaries du -sb <resolved node package dir> # node CLIs
peak RSS
python3 -c 'import subprocess,resource; subprocess.run(["$TOOL","--version"]); print(resource.getrusage(resource.RUSAGE_CHILDREN).ru_maxrss)'
prompt → first byte
python3 seo-tracking/bench/mock_provider.py 8077 & python3 seo-tracking/bench/ttft.py $TOOL <one-shot args vs http://127.0.0.1:8077/v1> # spawn → first stdout byte, median of 9
What this doesn't measure
Model quality, throughput under load, and agent competence are deliberately absent — they belong to the model, not the harness. These numbers measure the tax the tool itself charges you: how long it takes to wake up, how much disk it occupies, how much memory it burns printing its own version.
Numbers vary by machine and version. They're published because they're reproducible — run the commands on your box and the table sorts itself.
What an always-on agent costs to keep running — idle and active RAM and CPU, time to first reply — is measured separately in the agent harness benchmark.