Beating llama.cpp at Decode, Without Speculating

Where this stood two days ago

As of the last post in this series, mojo-baro's decode engine ran a full bf16 Qwen-architecture model on an RX 7900 XTX, greedy, no batching, no sampler beyond argmax: a correctness vehicle for the kernels, not a product. On Q8_0 weights, no speculative decoding on either side, it read 68.8 tok/s_gen against llama.cpp's own 74.1 on the same GGUF. 0.93x. Behind, honestly reported, and the number this post starts from.

Two things changed that gap today: a persistent kernel that fuses an entire decode step into one GPU launch, and a drop to 4-bit weights. Stacked together they didn't just close the 7% deficit. They turned it into a lead.

One launch instead of thirty

A decode step at batch size 1 is thirty-odd small GEMMs back to back, one per weight matrix in the layer stack, each one a separate kernel launch. At this shape the arithmetic is trivial and the whole step is bound by streaming weights out of HBM, so launch overhead that would be noise on a training-sized batch is not noise here.

kernels/mega.mojo replaces that with a single grid launch that holds the GPU for the whole token: all 32 layers, the final norm, the head GEMM, and the argmax, with workgroups synchronizing through a software grid barrier instead of returning to the host between kernels. It shipped as the default path today. Same weights, same Q8_0 pack, no speculative decoding:

+22% from removing launch overhead alone, and the first time this project's non-speculative decode has beaten llama.cpp's non-speculative decode on any quantization. 20 of 20 prompts identical to the reference token stream both before and after.

Then the weights got smaller

Q8_0 packs 8.5 bits per weight. Q4_0 packs 4.5: blocks of 32 four-bit nibbles around a per-block scale, the same single-block format llama.cpp uses, which matters because it means the comparison stays honest: tools/q4-check.py proves our pack byte-equal to llama-quantize's own Q4_0 output, block for block, the same discipline that made the Q8_0 numbers trustworthy in the first place. The pack shrinks from 10.52 GiB to 6.18 GiB.

Landed on top of the megakernel, same-stint A/B, still no speculation:

1.42x internally, and against llama.cpp's own Q4_0 bar (measured the same way, on the same twenty prompts, llama-quantize --pure Q4_0 on the same GGUF) 110.0. Already 1.05x ahead.

Two more rounds, today

Two structural changes on the GEMM side closed out the day. Both are their own story: the register-occupancy probe that quietly produced wrong tokens, the LDS-staging trick that reads each activation row once instead of once per wave, the attention kernel that turned out to be latency-bound rather than throughput-bound. That story is the next post in this series. The number is what belongs here:

(120.1 against the 115.0 above is the same shipped configuration, re-measured in a fresh stint: a few percent of stint-to-stint spread is normal here, which is why every round measures its own before-and-after rather than reusing an older number.)

+8.9% across the two rounds, each gated on bit-exact identity against a numpy reference and against the pre-change binary before it was allowed to count.

The number

Same GGUF, same twenty prompts, same box, both engines run greedy with no speculation. That is the whole claim: on the number every serving stack reports by default (plain decode throughput, no draft model, no acceptance rate to argue about), mojo-baro is ahead.

What this isn't

It isn't a win with speculative decoding turned on. llama.cpp's own Q4_0 MTP run reaches 169.5 tok/s_gen on the same prompts, well clear of anything on this side of the fence. mojo-baro's Q4_0 speculative attempt lands at 129.3 at k=2, 0.76x behind, the same shape of gap this project's MTP work has shown since the first speculative round: ahead on a five-token race prompt, behind on real text, because llama.cpp's draft loop turns a lower acceptance rate into a bigger speedup than ours does. Worth noting in llama.cpp's favor and against it at once: its speculative run only reproduces its own greedy output on 7 of the 20 prompts, at a median 75.5% acceptance. 169.5 tok/s_gen is real, and it is a different, noisier thing than the 110.0 it's being compared against here.

It also isn't a claim about any GPU but this one. Everything in this post runs on a single RX 7900 XTX, gfx1100, RDNA3: a 32-wide warp, 64 KB of LDS per workgroup processor, occupancy and tile parameters swept for this memory system specifically. None of it should be assumed to transfer to CDNA, to a different RDNA3 card with a different CU count, or to Nvidia hardware without re-measuring there.

How to prove this wrong

Run both engines on the same GGUF, greedy, no speculation, over the same twenty-prompt set in one session, and compare the medians. A llama.cpp median above mojo-baro's falsifies the claim above. bench/q4-protocol.md and bench/megakernel-protocol.md in amarbaro/mojo-baro carry every stage, every frozen prediction, and the one round that was reverted rather than shipped.