Our planted bugs were blind to the bug that mattered

An outside review found ten defects in the fly's GPU physics engine in one pass. Every stage of that engine had passed MuJoCo parity, and every stage had a planted bug its gate was proven to catch. The review's first finding is still invisible to our gate today: delete the fix, and check.sh tendon_limit prints PASS on all three fixtures.

Part 4 ended on "the engine matches MuJoCo, not a real fly." This is what happened when someone else read the code. The reviewer was Codex, an AI coding agent, working from source and saved profiler files only: no edits, no builds, no GPU runs. It rated correctness 7 out of 10 and gate quality 7 out of 10.

A planted bug measures what we thought of, not what we missed

The first finding (it called it B1) sits in the Newton solver, the loop that finds contact and limit forces by repeated improving steps. Each step uses a matrix, H, that records how the active constraints couple. When a constraint switches on or off, H has to change.

Our code watched contact rows for switches. It did not watch tendon-limit rows. When a tendon limit flipped, the solver kept going on a stale H. Tendon limits had been ported the day before (commit 6273a01, 2026-10-08), with a gate that caught 4 of 4 planted mutants.

Why did the gate stay green? Newton with a line search on a convex cost still converges with a stale H, as long as that H is positive definite. The step direction is worse, so it takes more iterations, but the final answer lands in the same place. Our gates compare the final answer to MuJoCo. A defect that only costs iterations is outside what they look at. The mutant that removes the fix passes on the stiff, soft and fly fixtures.

That makes the severity a judgment call, and the review and we disagree on it. Codex rated it high. We measured it as a cost problem, not a parity problem, unless the iteration cap or the stopping tolerance cuts the solve short. The cap case is not hypothetical on paper. In Codex's two-variable toy (a CPU model of the control flow, not our kernel), the stale H hits the 100-iteration cap about 4e-4 from the optimum. Our parity bar is 1e-5 of a field's scale, so read that as a direction, not a like-for-like comparison.

We also have a bound on the cost from an experiment on October 5, where we staled H on purpose on contact rows. Never refactoring took 389.6 ms per step against 88.9 ms, at 75.6 Newton iterations against a cap of 100. That is the extreme case, not the tendon-limit one.

The other six were found by reading, not by running

None of these came out of a failing gate. The review found them by reading what the exporter and the kernel accept, plus a few CPU calls on the MuJoCo side.

B4 is the one that stung. The export refused joint limits only when the model also had equalities, so the shipped fly lost two limits and nothing said so. B7 is the quiet one: a gate that cannot pass tells you nothing, in the opposite direction from a gate that cannot fail.

The other three findings (B8 to B10) are in the benchmark and traffic scripts and a unit label. They are not covered here.

Four of seven shipped with a gate that fails without the fix

All seven fixes landed in five commits between 09:28 and 10:27 on the same day. We had a rule from part 4: no stage counts as passed until a planted bug fails its gate. Held against the commit messages, it came out like this.

• B3, B4, B5, B6: each came with a new gate or mutant that was caught. B3 got a damping-sign mutant. B4 got joint_limit with two mutants caught. B5 got two new models, both mutants caught. B6 got a tendon-servo case under both implicit integrators with two sign mutants, both caught (the stage caught 8 of 8).
• B2: the commit names no new gate. Only existing stages were rerun.
• B1: the commit says "B1 has no dedicated transition test yet." A real gate would compare Newton iteration counts with and without the rebuild. As of this writing it is not built.
• B7: we deleted the dead stage. Nothing refuses that combination any more, so there was nothing left to gate.

A defect someone else finds becomes a mutant before it becomes a fix.

What counts:

• A mutant that removes the fix and turns a gate red.
• A gate that reads cost (iteration counts) when the defect costs time and not answers.
• Reading the exporter for what it accepts without complaint.

What does not count:

• A PASS on states we chose ourselves.
• Mutants planted before we knew the bug class existed.
• A gate that cannot pass.

What it cost

Five commits and about an hour for the fixes. The unpaid bill is B1's gate: it needs iteration counts recorded per run and a comparison we have not written. One reviewer, one pass, source reading only, so ten is a floor for this snapshot and the review itself marks several kernel-level reproductions Unverified (B1, B3, B5, B6). The new gates are as good as the mutants we thought of, which is the problem this post is about.

How to prove this wrong

Take the limit-flip rebuild out of 20aae03 and keep the sphere fix. Run check.sh tendon_limit on the stiff, soft and fly fixtures. If any fixture fails, the gate was not blind and the claim is wrong. If all three print PASS, B1 is still ungated, and the open question is the one we have not answered: what is the cheapest gate that sees a defect that only shows up as extra iterations?

Provenance: commits 20aae03, 683c491, 33c9a0c, d10aff8, 8d0fa95 in the fly-connectome repo; the review is .work/reviews/codex-engine-review.md, written against 179bfb7. The mutant result for B1 is from 2026-10-09 on the one RX 7900 XTX.