Spread Does Not Detect an Inert Parameter

For about a week, this project believed it had a hand-written GEMM roughly twice as fast as AMD's own hipBLASLt on an RX 7900 XTX. The number was reproducible. Its spread across runs was tight. It was also completely wrong, and no amount of re-running would ever have said so.

The kernel was fine. The problem was on the other side of the comparison: our shim was calling hipBLASLt without configuring it. Three separate defects, all in our code, all making the vendor look slow — a workspace allocated on every call, blind trust in the heuristic's ordering of returned algorithms, and splitK and wgm that were never actually set. Fixing our own shim took hipBLASLt from 2497 to 5201 GFLOP/s at 512³ and erased the lead entirely. Algorithm selection alone accounted for 3097 → 5201.

That retraction is still in docs/BASELINE.md, under a heading that reads Do not repeat the retracted claim (6fad30f). It stays there on purpose.

The part that should bother you

Every instinct about benchmark hygiene points at variance. Warm the clocks. Run it n times. Report the median and the spread. Discard the round if the spread exceeds ten percent. All of that is correct, and all of it was being done here.

None of it could have caught this.

A misconfigured arm is not noisy. It is consistently misconfigured. The untuned hipBLASLt call did the same untuned work on every iteration and returned a beautifully stable number. Spread measures whether your instrument is repeatable. It says nothing about whether it is pointed at the thing you think it is pointed at.

This is not a hypothetical failure mode dressed up as a lesson. It has happened twice in this repository, and the second one was worse.

The control arm that was not a control

While measuring speculative decoding, we needed a baseline with speculation off. The server accepted "speculative.n_max": 0 in the request. It reported no error. It also ignored the field entirely on that build — draft_n in the response's timings block came back unchanged, which is to say the "no-speculation" arm was still speculating against itself.

Had nobody read that block, the published no-spec bar would have been the speculative number. Every speedup measured afterward would have been computed against a contaminated baseline, and the error would have been inherited silently by every later comparison — including ones made months on, by someone with no idea the flag had never worked. The arm looked clean. It had a tight spread.

Both incidents are now written into the top of the repo's protocol rules (d44dea8) as the reason the rules exist at all.

And once more, from inside our own code

A third case, and the one I find most instructive, because there was no server and no vendor library to blame.

An A/B sweep needed a new comptime parameter, JSPLIT, on an SSM kernel. At its default value of 1 the new code path was algebraically identical to the old one — provably so, by inspection. The control arm was built by generalizing the existing kernel and leaving the parameter at its old hardcoded value.

The first timing showed both arms at roughly 0.09 s. No difference. That would have been written up as inconclusive, within noise.

Checking the ISA receipt against the previous stint's recorded baseline showed what had actually happened: adding the parameter raised VGPR spill count from 603 to 2029 at MR=8, a 3.4x regression caused by the generalization alone, before the parameter's value ever mattered. LLVM's register allocator does not see "this is logically the same as before." It sees new IR. Rebuilding the kernel so that JSPLIT == 1 compiles the byte-for-byte original code restored the spill count exactly, and the real measurement appeared: 0.048 s for the control against 0.088 s for the split arm (af30ac9). The arm was falsified cleanly. The earlier "no difference" had been two broken things cancelling.

The rule that does work

Read the value back from the running system, not from the command you typed.

That is the whole of it. Passing a parameter is not evidence that the parameter took effect, and the only thing that constitutes evidence is the instrument's own report of its state.

What counts as a read-back, in this project:

• Server: GET /props for effective load-time settings; the response's own timings block for per-request settings; stderr at load for what the flags actually resolved to.
• Our engine: values printed by the run itself, and — for compile-time constants — the binary being rebuilt in the same command as the run.
• Vendor calls: the algorithm actually selected and its tuned attributes, dumped from the shim, not assumed from the heuristic.
• Kernels: grid and block dimensions, template parameters, and spill counts echoed by the harness or read out of the ISA.

What does not count: the flag string. The request JSON. The protocol document. A previous run's receipt. Anyone's recollection.

The enforcement is deliberately blunt: no receipt, no arm. An arm whose parameters were not verified before its timed run is void, its numbers may not be recorded or compared, and it is re-run rather than retro-justified — the same standing the spread threshold already had. The verification also has to happen before predictions are frozen, because verifying afterwards lets the number you observed decide which knobs get scrutinised, which is the same defect as not freezing predictions at all.

What this costs, and what it bought

It costs a genuinely annoying amount of time. Every arm now carries a verification step whose entire output is "yes, the thing you asked for is the thing that is happening."

Here is what it bought, and it is not flattering. When the speculative decoding loop landed, it measured 127.96 tok/s, 1.89x over the non-speculative baseline and 1.17x faster than llama.cpp (058dd28). Genuinely measured, receipts and all, on a five-token race prompt with 94% acceptance.

On a set of twenty real prompts, the same engine was 82 tok/s and 0.66x.

Both numbers were correct. Only one of them described the engine. The rule that came out of that — every quoted throughput figure is a median over the full prompt set, with its range, against a competitor measured on the same set in the same session — cost us the best headline number the project had produced.

Where it actually stands today, on the same twenty prompts: 145.6 tok/s on the race prompt, 100.7 median on real text against llama.cpp's 123.5 (41d0361). Ahead by 1.33x on the benchmark everyone runs. Behind by 0.78x on the workload that matters. Both go in the README.

I would rather publish that than the 1.17x.

How to prove this wrong

The load-bearing claim here is the first one: that an unconfigured hipBLASLt call is dramatically slower than a configured one, and that this is what manufactured our 2x. Configure splitK and wgm on your own gfx1100, compare against the defaults at 512³, and if the gap is not there, the central example is wrong and the post should be corrected.

The fp16 GEMM half of this repo runs without model weights — bench/report.sh is one command and emits a machine-readable receipt (6751e4f). Reports from other RDNA3 cards are welcome, especially ones that disagree.

One thing I will not claim: that this rule has since caught a third poisoning. It has not. It has caught inert parameters before they became results, which is not the same thing as a save, and I have no way to prove those runs would have gone wrong.

Provenance

AMD RX 7900 XTX (gfx1100, RDNA3), 24 GB. ROCm 7.2, hipBLASLt from /opt/rocm/lib. Mojo 1.0.0 via max[all]==26.5.0, pinned in pyproject.toml and resolved from PyPI. Every figure above traces to the commit cited beside it; the protocol rules themselves are bench/PROTOCOL-RULES.md in amarbaro/mojo-baro.