Posts with this tag: 12
A LoRA patch scored 89.06% on its worst prompt against a frozen 90% floor and looked FAILED. The unpatched base, run against the same floor for the first time, also failed at 89.06%, on a different prompt. The bar was unreachable by construction. Llama.cpp's own f16-KV configuration misses its own f32 reference the same way, at 5 of 7 tested lengths.
ReadA day with 135 GPU jobs and 34 failures, none of which needed the GPU to find. The fix is not a faster GPU, it's treating the card as a shared, budgeted resource with a preflight step before anything touches it.
Readv_wmma_i32_16x16x16_iu8 issues at 1.007x the rate of the bf16 WMMA instruction on gfx1100, both at 133 T-ops/s. An int8 MMQ prefill kernel built on the expected 2x lost to the project's own bf16 kernel, 0.91x at n=1024, and the lane closed negative.
ReadA chunked prefill path for a 35B MoE passes identity on 23 of 23 prompts and runs 3.7 to 7.9 times faster than replaying the decode kernel one token at a time. Two variants that would have been faster still, WMMA attention and a dense SSM scan, reorder floating-point sums and stay off by default because they fail the same identity check.
ReadThe routed experts of a 35B MoE moved to host RAM, VRAM footprint down to 2.68 GB from 21, output identical on 20 of 20 prompts. The first working version ran at 39 tok/s. Pinning the host store and reading misses in-kernel got it to 72.
ReadFour rounds in one session took a 35B MoE from 42.88 to 111.89 tok/s_gen. The first three fixed how experts were dequantized and routed; the fourth removed 490 of 1217 kernel launches per token, and the round that shipped it landed below its own preregistered bar.
ReadAn audit of this project's own optimization loop built four candidates designed to cheat its performance gate. One of them, a one-line deletion of a GPU synchronization call, passed at a fabricated plus sixty-three percent. The fix moved the stopwatch out of the file the candidate is allowed to touch.
ReadThe trunk decode path had been sitting a few percent behind llama.cpp for two days running. Quantizing to Q4_0 and fusing the whole forward pass into one persistent GPU launch closed that gap and passed it: 130.7 tok/s against llama.cpp's 110.0, no speculative decoding on either side.
ReadFive gated rounds in one day took a persistent decode kernel from 120 to 131 tok/s_gen. One of them, tried first, made things worse in a way that produced no error message whatsoever. The fix is now a permanent gate on every run.
ReadSquare GEMM is the benchmark everyone runs; decode is M=1 streaming weights out of HBM. Four frozen rounds, three wrong predictions, and a one-percent win that came from occupancy relief rather than fewer bytes.
ReadA hand-written GEMM that looked twice as fast as the vendor library, and a no-speculation control arm that was quietly speculating: both produced clean, tight, repeatable numbers. The bug was never the math. It was a parameter that was set and never took effect.
ReadA pipelined fp16 WMMA GEMM beats hipBLASLt at ten square sizes on RDNA3, but only once tile shape stops being fixed. Plus a retracted 2x claim and why clock warm-up was worth more than any kernel change.
Read
