Three-Arm Ablation

Run 1's plateau-then-surge pattern looked like the shared tool library was driving improvement. Is that true — or is a real model's mutation quality alone doing all the work, with the library just along for the ride?

Three fresh 100-generation runs, identical seed and maps, isolating one ingredient at a time:

• Full ratchet — evolution plus the inherited tool library.
• Evolution only — the library surgically removed; nothing to inherit.
• Cold sampling — no parents, no library, no evolution at all: a fresh bot every generation, nudged only by 12 rotating style hints. The third arm was my own addition to the design, to check whether the loop mattered at all versus raw model quality.

Nothing about the model or the arena — only which mechanisms were switched on for each arm. This is the first run with a proper control.

Final archive coverage, best Elo, mean Elo, and number of tools extracted, compared across all three arms.

The control matched or edged the full-ratchet arm on every single metric. Evolved arms bred the strongest champions (1070/1081) but explored the least; cold sampling explored the most (+22 percentage points of coverage) but produced the weakest champion — a clean, textbook exploration-versus-exploitation split, reproduced on the first try in a homemade lab.

The tool library did nothing measurable. Not "did less than hoped" — the arm with the library did not outperform the arm without it on coverage, best Elo, or mean Elo. My working theory: the mutation prompt at the time explicitly said "prefer small, targeted changes," which produced timid children that hugged their parent's existing behavior and never actually needed to reach for a new tool. The library never got a fair trial — it just sat there.

One caveat on the cold arm: it used 12 rotating style hints in its prompt, so its diversity partly comes from varied prompting, not a completely bare model — it's a control, not a perfectly clean one.

Rewrite the mutation prompt to be bold and theory-primed instead of conservative, and give the library an actual fair shot. Also move bulk experimentation off the subscription CLI's session limits onto a free, fast engine so more runs fit in a day.

304 API calls · $2.20–$2.60 (Haiku 4.5 via Anthropic API).

Series through Run 2: 413 calls · $3.13–$3.73. Call counts are exact, straight from each run's metrics file. Costs are a range rather than a figure because the provider bills cached input tokens at half price and does not report the cached fraction in the API response, only in the console dashboard, so the upper bound prices every input token as uncached and the lower bound prices all of them as cache hits. A sampled dashboard minute showed 68% of input tokens were cache hits, which puts the truth nearer the bottom of each range. Token counts are measured exactly from Run 8 onward; earlier runs additionally reconstruct tokens from Run 8b's measured per-call profile (3,642 in / 1,348 out), because the usage field was discarded before it was ever written down. A bound is not a measurement and a reconstruction is neither, so both are labelled.