A Gate You Have Never Seen Fail Is Not a Gate Yet

The fused q4x kernels in mojo-baro are gated by kernels/test_q4x_hostops.mojo, which runs the fused path and the unfused path and compares them bit for bit. That gate was in place when the dispatch lane merged, the change that cut GPU dispatches per decode token from 1244 to 992. It passed. It had never been seen to fail.

Commit d181a44 is the first time anyone watched it go red. Three nudges of about 5e-7, one per fused kernel, each tripped the check that covers that kernel and no other. This post is about why the red run was needed before the green runs meant anything.

A passing check reports that it returned zero

A green run tells you the comparison exited cleanly. It does not tell you the comparison can exit any other way. If the test read the wrong buffer, or a tolerance absorbed the very error you care about, the output would look exactly like a correct kernel. Nothing on the screen separates the two cases.

So we broke the code on purpose. Q4X_FUSE_MUTANT is a compile-time switch in kernels/q4x_kernels.mojo, default 0. Value 1 multiplies one output of the hcdown inject by 1.0000005. Value 2 does the same to the hcup cur. Value 3 does it to the gnorm_ffn combine. At 0 the shipped code is unchanged, so the switch adds nothing to the product.

bench/check-fuse-mutants.sh builds one binary per value. It requires the shipped build, mutant 0, to pass. Then it requires mutants 1 to 3 to fail. The GPU runs go through gpu-wait, so the check shares the card with every other job. These are the four run logs:

Each nudge was caught by the check that owns its kernel. That matters more than the count. A gate that fails on everything proves only that it is loud. A gate that fails on each planted path, and names it, shows that each fused kernel has a witness of its own. When someone rewrites the hcup path later and breaks it, the failure will say hcup cur.

The shipped build passing is the other half. A planted-bug script that only checks for failure can hide a gate that fails on correct code, so the script requires both.

The same move on a shared math function

bench/check-shared-math.sh, from commit c24044d, applies the pattern to fexp32 and flog32 in kernels/bexp.mojo, the Float32 exp and log that the CPU and GPU paths share. Its mutant is a sed that removes the k * ln2_lo term from flog32. The script requires three things on 40,017 generated inputs: CPU and GPU return the same bits, both stay within 1 ulp of a correctly rounded mpmath oracle, and the mutant goes above 1 ulp on log. It does. The ulp table is exchange/receipts/shared-exp/flog32-ulp.txt. The script prints FAIL <step>: <reason> and exits non-zero on any miss, and a skipped step is not a pass.

The same move on a feature, and on a threshold

tests/check_brainnet.sh in newera (commit 36b455e) turns the idea on a feature instead of a kernel. The feature is spreading activation through wikilinks. The script builds the tool twice, once with BRAINNET_MUTANT=0 and once with 1, and mutant 1 switches spreading off. On a synthetic vault with notes linked a to b to c, the real build must recall b from a, and the no-spreading build must not. The mutant is the claimed feature with the feature removed, which is the shortest way to ask whether the check notices the feature at all.

The vero language shows a case where the mutant moved the threshold. vero/trec.vero (trace reconstruction) had 4 of 22 generated mutants survive on its first run, recorded in docs/vero-lang-spec.md and added in 27e8e00. Three survived because the claim had only a lower bound: with >= 0.9, turning ok / trials into ok * trials still passed, so the claim now says expects in [0.9, 1.0]. The fourth was real. At a noise rate of 0.95, a decoder that needs a two-thirds supermajority recovered 0.983 of the strings, indistinguishable from the correct majority decoder. At 0.9 the correct decoder scored 0.967 and the supermajority mutant 0.717, and it dies. tests/check_vero.sh keeps two controls that must misbehave on purpose: an impossible bound that must fail the gate, and the old 0.95 claim at 600 trials, where the supermajority mutant must survive.

How this fits the planted-bugs post

Our planted bugs were blind to the bug that mattered argued that a mutant per stage measures the bugs we thought of and misses the ones nobody did. This post argues the step before that. A gate must first be seen to fail at all, or the question of which bugs it catches has no answer.

The two are halves. Seeing a gate fail is necessary and says nothing about the bug nobody planted. A full set of mutants, caught on the one fixture the author chose, is still a sample of the failure space chosen by the person who wrote the check. That post also named the opposite fault: its B7 was a stage that could never pass, which tells you as little as a stage that can never fail. Here we only closed the first of the two, for three kernels.

A gate counts after it has been red once, for a reason you chose.

What counts:

• A mutant that touches only the code the gate claims to cover, caught by the check for that code.
• A shipped build that passes in the same script, so the gate cannot be loud by default.
• A failure message that names the broken path.

What does not count:

• A long history of green runs.
• A gate that fails on every mutant without saying which one.
• A planted bug in a path the gate does not claim to cover.

What it cost

Four builds and four runs, from 02:32 to 02:37 by the log timestamps, 22 lines of shell and a 9-line change to the kernel source. The nudge is an extra branch inside three fused kernels, resolved at compile time, so it is absent from the product at the default value. The unpaid bill is the second gate.

The bench/q4x-ab.sh identity check compares the token ids of 20 prompts at 64 greedy tokens across four runs. It reported IDENTITY PASS: 20 of 20 for the dispatch lane (exchange/receipts/q4x-dispatch/ab-ng64-receipt.txt), and no one has planted a bug in its path. Its green runs are a claim we have not tested. We also have not measured whether a 5e-7 nudge ever flips a greedy token in 64. If it does not, only the bitwise check can see this class of bug, and the token-level check is blind to it by design.

How to prove this wrong

On an RX 7900 XTX, check out d181a44 and run bench/check-fuse-mutants.sh. The claim is wrong if the shipped build fails, or if any mutant passes. To test the open gate, plant one nudge in the q4x-ab path and run bench/q4x-ab.sh with 64 greedy tokens. If it stays green, that identity check is blind to the nudge and the bitwise check is the only witness. If it goes red, it is a second witness we had not yet seen fail. We would like the result either way.

Provenance

Commits in the mojo-baro repo: d181a44 (mutants and bench/check-fuse-mutants.sh), c24044d (bench/check-shared-math.sh, 40,017 inputs), d422318 (the 1244 to 992 dispatch count and the 20 of 20 identity receipt). In newera: 36b455e (brainnet check) and 27e8e00 (vero generated mutants). One RX 7900 XTX (gfx1100), ROCm 7.2.4, Linux 7.2.11, Mojo 1.1.0, all on 2026-10-21. One card proves that this gate can fail on this card, nothing about another GPU.