The Shape That Actually Matters
The shape a decode step actually runs
The GEMM everyone benchmarks is square. A 4096³ matrix multiply is a compute problem: enough arithmetic per byte that the memory system gets time to keep up, and the number you publish is a fraction of the card's FLOP ceiling.
Single-token decode is not that. Generating one token multiplies one row of activations against every weight matrix in the model, which is the shape M=1, K=4096, N=12288 — a skinny GEMM with almost no arithmetic per byte loaded. The kernel is not waiting on the multiply units. It is waiting for 100 MB of weights to arrive from HBM, and it will do that around thirty times per token, once per weight matrix in the layer stack.
That makes the interesting question a different one. Not "how close to peak FLOPs", but "how close to peak bandwidth", and specifically with the weights not sitting in cache — because during real decode they can't be. The RX 7900 XTX has 96 MB of Infinity Cache and the model is many gigabytes. By the time the next token needs the first layer's weights again, they are long gone.
The instrument came first, and the first one was broken
An earlier version of this measurement timed a single weight buffer in a loop. It reported 2.70 ms on one run and 0.33 ms on the next, from identical builds — an eight-fold swing driven entirely by whether the buffer happened to still be cache-resident. That result was retracted rather than explained, because a number that moves 8x between runs of the same binary is not measuring the kernel.
The replacement rotates eight distinct device buffers per arm. At bf16 that is 8 × 100.7 MB = 805 MB of working set, far past the 96 MB cache, so every launch genuinely streams from memory. One second of clock warm-up on the same rotation, 200 timed launches, and the whole measurement repeated ten times in-process. The first frozen prediction of the round was about the instrument, not the kernel: if per-measurement spread doesn't collapse below 5% of the mean, no ratio may be claimed at all. It came in under 1%.
Four rounds, three wrong predictions
Each round below was written down before it ran — the question, the arms, the predicted range, and the condition that would falsify it. The prediction stays in the file next to the result whether it survived or not.
Round 1 — fewer bytes. Quantized weights are 1.19 bytes per element against bf16's 2. The byte ratio is 0.563, so the predicted speedup was 1.5–1.9x. Measured: bf16 399–403 µs, q8 468–474 µs. Both predictions falsified, and q8 came out slower. The weight layout was coalescing-bound at roughly 250 GB/s — 26% of the card's peak — so it was never limited by bytes in the first place, and halving them only added dequantization work to the arithmetic units. A post-hoc arm that transposed the weights at load time ran 194 µs; it was not preregistered and was disclosed as such.
Round 2 — fewer bytes, laid out properly. The obvious objection to round 1 is that q8 never got a fair layout. So: K-major q8, 56.6 MB against bf16's 100.7 MB, parity-gated before the freeze, plus hipBLASLt as a vendor arm in the same rotation. Predicted 105–145 µs and 1.45–1.85x. Measured 164.5–165.3 µs and 1.18x — faster than bf16 this time, but well outside the predicted band in both directions, missing high on the time and low on the ratio. The vendor ran 123.0–126.2 µs and beat everything on the board. The reading: dequantization ALU plus reloading a scale every 32 elements eats most of the byte win, and hipBLASLt was pulling ~812 GB/s, 85% of peak, against our 517. The gap to close was efficiency, not bytes. No beat-vendor claim was made.
Round 3 — wider loads. Give each thread eight contiguous columns instead of one, so a single vector load covers 16 bytes instead of 2. Predicted the best arm at 130–165 µs, and predicted that CPT=8 specifically would regress against CPT=4, because 64 accumulator lanes should collapse occupancy. The band held; the occupancy prediction did not. CPT=8 was the best arm at 138.8–139.8 µs, 1.40x over the round-1 baseline and 723 GB/s, 75% of peak. Thirty-two extra float registers turned out to be affordable on this kernel. The vendor was still 1.10x ahead.
Round 4 — stop carrying rows that don't exist. The kernel had been built for M=8 and was being used at M=1: eight rows of staging through local memory, 64 accumulator lanes, for one row of actual work. Specializing it for M=1 replaces the staging with scalars and the accumulator with a single SIMD register. Predicted 3–12% over the previous arm, explicitly not via bytes — the traffic is identical between the two — and with a falsifier attached: if the specialized kernel lands within 3% of the general one, the kernel is bound by latency or traffic rather than occupancy, and the whole line of attack should be abandoned in favour of other work.
| arm | µs/launch | predicted | verdict |
|---|---|---|---|
| general kernel, CPT=8, at M=1 | 137.8 – 138.9 | 135–145 | held |
| M=1 specialized, CPT=4 | 123.5 – 124.3 | — | worse than CPT=8 |
| M=1 specialized, CPT=8 | 121.2 – 121.8 | 117–135 | held, 12.3% — top of band |
| hipBLASLt f16 at M=1 | 122.1 – 123.3 | reference | — |
121.8 against 122.1. The two ranges do not overlap, which is the condition the round set in advance for claiming anything at all, and the margin is about 1%.
What that 1% is and isn't
It is a cold-cache win over the vendor library at the exact shape a decode step runs, on this card, reproducible with ./bench/run.py. It is not a large margin, it is not a claim about any other shape, and it is not a claim about any other GPU.
The mechanism is worth being precise about, because it is the part that generalizes even when the number doesn't. Nothing about the memory traffic changed between the losing arm and the winning one — the same bytes cross the same bus. What changed is that the kernel got smaller. Dropping the eight-row staging and shrinking the accumulator freed enough registers and local memory that more waves stay resident on each compute unit, and more resident waves means more outstanding memory requests, which is the only thing that matters when you are bandwidth-bound. Occupancy relief, not byte reduction. Round 1 and round 2 both attacked the bytes and both failed.
Two disclosures that belong next to the numbers: during round 2 a language-model server held 22 GB of VRAM on the same card throughout, which forced two arms to be dropped for memory and is recorded in the protocol file; and in round 4 one vendor measurement out of ten came in at 817 µs against a 122 µs median and was excluded as an outlier, which is stated here because excluding it improves the comparison for us and that is exactly the kind of decision that should not be silent.
The full protocol file — every prediction, every falsification, the arms that were dropped and why — is bench/coldcache-protocol.md in mojo-baro. The rounds are also laid out, with the receipts they came from, on the project page.