What the Colony Threw Away

This is the only page in the series where that sentence is true, and it is the reason to trust the number at the bottom of it. On 2026-08-30, before a single Run 9 generation had been executed, a pre-registration was written into the vault fixing the hypothesis, the outcome measure, the exact statistical test, the sample size, the stopping rule, and the interpretation of every possible result. It also recorded a prediction, so the prediction could be scored rather than quietly revised afterwards.

Four things were nailed down in advance, each closing a door that had already swung open somewhere earlier in this series:

• One-sided Fisher exact on the pooled 2×2, α = 0.05. Named before seeing anything, so no better-looking test could be substituted later.
• Fourteen colonies, not seven. Seven gives only 53% power against a 4-point effect — the size actually predicted. Registering seven would have been registering a coin flip.
• No stopping for significance. The halfway look was declared descriptive-only in advance, precisely because it would be tempting.
• Run 8b excluded from the primary. Its trials produced the estimate being tested; including them would have been testing a hypothesis on the data that generated it.

No deviations occurred. What follows is what was registered.

Run 7's published conclusion that prompt-injected lessons were "largely placebo for this model" is hereby retracted. They reduce the crash rate by about six points, and the effect is not marginal. This result is specific to gpt-oss-120b — Run 10 replicated the whole protocol on a weaker model and found nothing at all, so nothing on this page should be read as a claim about the method in general.

174 trials drawn from 14 colonies are not 174 independent observations. Trials inside one colony share a tool library, a lineage, and a drifting model state, so a pooled test that treats all 3,480 children as independent overstates its own certainty. That objection deserves an answer before the result is believed, not after someone raises it.

Re-analysed with the colony as the unit of independence — fourteen numbers instead of 3,480:

Thirteen colonies positive, one exactly zero, none negative. The clustered estimate is larger than the pooled one because the two highest-trial colonies — 4 and 5, together 30% of all trials — also carried the highest baseline crash rates and the smallest effects, and they dominate any count-weighted pool. Equal-weighting the colonies removes that pull.

The pre-registration recorded a forecast: 60% that the effect would hold and reach significance but regress to 3–5 points, on the reasoning that Run 8b's 5.6-point estimate came from only 180 children per arm and was probably inflated.

Direction and significance: correct. Magnitude: wrong. The effect did not regress. It came in at 5.89 pooled and 7.05 clustered — at or above the original estimate. The noisy little number from Run 8b was accurate all along, and the regression-to-the-mean reasoning applied to it was the part that needed correcting. Recorded here because a prediction that only gets graded when it is right is not a prediction.

The colony canonized almost none of these lessons. Across Run 8b and Run 9 together, the gate looked at 183 candidate doctrines and adopted a vanishing handful — and every one of those rejections was correct by its own standard. A single trial at twenty children per arm simply cannot resolve a six-point effect; doing so takes roughly 1,200 children per arm. Every individual verdict was properly calibrated. The aggregate of those verdicts threw away something real.

Individually-rational skepticism produces collectively-irrational rejection when true effects are small relative to the evidence available per decision.

This is not a bug in the gate, and tightening or loosening α would not fix it. It is structural. A culture that only adopts what any single member can personally prove will correctly reject nearly everything, and will sit on a genuine transmissible advantage while doing so. The knowledge was available in the aggregate the whole time; no participant was ever in a position to see it.

The series now holds both failure modes, produced by the same lab under the same conditions. Run 5's colony believed everything it was told and died of superstition, canonizing a law about state.arena.walls that pointed at a field which does not exist. Run 9's colony believed nothing it could not personally verify and discarded something true. Neither colony was irrational. Each was defeated by the amount of evidence a single decision could carry.

Fourteen colonies, 4,200 generations, 13,061 API calls, three hours and twenty-one minutes unattended. Every colony verified at exactly 300 generations from its metrics file rather than from the run checkpoint — the checkpoint lags asynchronous completion, and trusting it reported colonies as unfinished that had in fact finished, which on a re-run would have re-executed thirteen of the fourteen and spent the money twice. That bug was found by checking, and is fixed.

Eight runs of argument about whether the colony's lessons meant anything, settled for about the price of lunch.

Read on its own, the finding above is a tragedy with no exit: the colony was right to reject each lesson, and wrong in aggregate, and nothing can be done. That framing is half wrong, and the half that is wrong matters more than the half that is right.

What is genuinely structural is narrow — one agent holding twenty samples cannot resolve a six-point effect. Arithmetic; no design fixes it. But "therefore the knowledge is lost" is not a consequence of that. It is a consequence of what this lab did with a rejected lesson, which was to delete it. A rejection left behind a signature and a counter; the distilled sentence a bot had died to produce was erased rather than annotated.

The effect measured on this page was recoverable only because trial records began being kept partway through the run — a change made for statistical bookkeeping, not for this reason. Under the original design, these verdicts were unrecoverable and the six-point effect would have been published as non-existent.

The loss is not in rejecting. It is in forgetting.

Keep every verdict with the context it was reached in, gate action rather than memory, and the pooled truth reassembles itself for free. Run 10 carries that argument out in full, and gives it the case that makes it undeniable: one lesson, two models, two opposite verdicts, and a design that would have thrown one of them away.

13,061 API calls · $13.71–$17.20 (gpt-oss-120b via Groq). Token counts measured exactly; the range reflects the provider's undisclosed cached-input split.

Series through Run 9: 16,679 calls · $20.13–$25.02. Call counts are exact, straight from each run's metrics file. Costs are a range rather than a figure because the provider bills cached input tokens at half price and does not report the cached fraction in the API response, only in the console dashboard, so the upper bound prices every input token as uncached and the lower bound prices all of them as cache hits. Runs before Run 8 additionally reconstruct tokens from a measured per-call profile, because the usage field was discarded before it was ever written down. A bound is not a measurement and a reconstruction is neither, so both are labelled.