Beating llama.cpp at Decode, Without Speculating
Where this stood two days ago
As of the last post in this series, mojo-baro's decode engine ran a full bf16 Qwen-architecture model on an RX 7900 XTX, greedy, no batching, no sampler beyond argmax — a correctness vehicle for the kernels, not a product. On Q8_0 weights, no speculative decoding on either side, it read 68.8 tok/s_gen against llama.cpp's own 74.1 on the same GGUF. 0.93x. Behind, honestly reported, and the number this post starts from.
Two things changed that gap today: a persistent kernel that fuses an entire decode step into one GPU launch, and a drop to 4-bit weights. Stacked together they didn't just close the 7% deficit — they turned it into a lead.
One launch instead of thirty
A decode step at batch size 1 is thirty-odd small GEMMs back to back — one per weight matrix in the layer stack — each one a separate kernel launch. At this shape the arithmetic is trivial and the whole step is bound by streaming weights out of HBM, so launch overhead that would be noise on a training-sized batch is not noise here.
kernels/mega.mojo replaces that with a single grid launch that holds the GPU for the whole token: all 32 layers, the final norm, the head GEMM, and the argmax, with workgroups synchronizing through a software grid barrier instead of returning to the host between kernels. It shipped as the default path today. Same weights, same Q8_0 pack, no speculative decoding:
| arm | tok/s_gen (20-prompt median) | vs llama.cpp Q8_0 (74.1) |
|---|---|---|
| thirty-odd launches per token | 67.13 | 0.91x |
| one persistent launch per token | 81.98 | 1.11x |
+22% from removing launch overhead alone, and the first time this project's non-speculative decode has beaten llama.cpp's non-speculative decode on any quantization. 20 of 20 prompts identical to the reference token stream both before and after.
Then the weights got smaller
Q8_0 packs 8.5 bits per weight. Q4_0 packs 4.5 — blocks of 32 four-bit nibbles around a per-block scale, the same single-block format llama.cpp uses, which matters because it means the comparison stays honest: tools/q4-check.py proves our pack byte-equal to llama-quantize's own Q4_0 output, block for block, the same discipline that made the Q8_0 numbers trustworthy in the first place. The pack shrinks from 10.52 GiB to 6.18 GiB.
Landed on top of the megakernel, same-stint A/B, still no speculation:
| arm | tok/s_gen (20-prompt median) |
|---|---|
| megakernel, Q8_0 | 80.83 |
| megakernel, Q4_0 | 115.02 |
1.42x internally, and against llama.cpp's own Q4_0 bar — measured the same way, on the same twenty prompts, llama-quantize --pure Q4_0 on the same GGUF — 110.0. Already 1.05x ahead.
Two more rounds, today
Two structural changes on the GEMM side closed out the day. Both are their own story — the register-occupancy probe that quietly produced wrong tokens, the LDS-staging trick that reads each activation row once instead of once per wave, the attention kernel that turned out to be latency-bound rather than throughput-bound — and that story is the next post in this series. The number is what belongs here:
| stage | tok/s_gen (20-prompt median) |
|---|---|
| megakernel + Q4_0, re-measured as this round's own baseline | 120.1 |
| + LDS-staged activations | 125.4 |
| + shared, vector-widened attention kernel | 130.7 |
(120.1 against the 115.0 above is the same shipped configuration, re-measured in a fresh stint — a few percent of stint-to-stint spread is normal here and is why every round measures its own before-and-after rather than reusing an older number.)
+8.9% across the two rounds, each gated on bit-exact identity against a numpy reference and against the pre-change binary before it was allowed to count.
The number
| tok/s_gen (20-prompt median) | |
|---|---|
| llama.cpp, Q4_0, no speculative decode | 110.0 |
| mojo-baro, Q4_0, no speculative decode | 130.7 |
| ratio | 1.19x |
Same GGUF, same twenty prompts, same box, both engines run greedy with no speculation. That is the whole claim: on the number every serving stack reports by default — plain decode throughput, no draft model, no acceptance rate to argue about — mojo-baro is ahead.
What this isn't
It isn't a win with speculative decoding turned on. llama.cpp's own Q4_0 MTP run reaches 169.5 tok/s_gen on the same prompts, well clear of anything on this side of the fence — mojo-baro's Q4_0 speculative attempt lands at 129.3 at k=2, 0.76x behind, the same shape of gap this project's MTP work has shown since the first speculative round: ahead on a five-token race prompt, behind on real text, because llama.cpp's draft loop turns a lower acceptance rate into a bigger speedup than ours does. Worth noting in llama.cpp's favor and against it at once: its speculative run only reproduces its own greedy output on 7 of the 20 prompts, at a median 75.5% acceptance. 169.5 tok/s_gen is real, and it is a different, noisier thing than the 110.0 it's being compared against here.
It also isn't a claim about any GPU but this one. Everything in this post runs on a single RX 7900 XTX, gfx1100, RDNA3 — a 32-wide warp, 64 KB of LDS per workgroup processor, occupancy and tile parameters swept for this memory system specifically. None of it should be assumed to transfer to CDNA, to a different RDNA3 card with a different CU count, or to Nvidia hardware without re-measuring there.
Reproduce it: bench/q4-protocol.md and bench/megakernel-protocol.md in amarbaro/mojo-baro carry every stage, every frozen prediction, and the one round that was reverted rather than shipped.