Posts with this tag: 5
Kernel parity with llama.cpp, byte for byte, on two Android phones. The IQ1_S kernel is correct and 2.1x to 3.6x faster per matmul than the reference path, at 2.31 bits per weight rather than the 1.56 the format's name implies. The OnePlus is faster on every run and unsteady on three of them.
Readv_wmma_i32_16x16x16_iu8 issues at 1.007x the rate of the bf16 WMMA instruction on gfx1100, both at 133 T-ops/s. An int8 MMQ prefill kernel built on the expected 2x lost to the project's own bf16 kernel, 0.91x at n=1024, and the lane closed negative.
ReadThe routed experts of a 35B MoE moved to host RAM, VRAM footprint down to 2.68 GB from 21, output identical on 20 of 20 prompts. The first working version ran at 39 tok/s. Pinning the host store and reading misses in-kernel got it to 72.
ReadFour rounds in one session took a 35B MoE from 42.88 to 111.89 tok/s_gen. The first three fixed how experts were dequantized and routed; the fourth removed 490 of 1217 kernel launches per token, and the round that shipped it landed below its own preregistered bar.
ReadThe trunk decode path had been sitting a few percent behind llama.cpp for two days running. Quantizing to Q4_0 and fusing the whole forward pass into one persistent GPU launch closed that gap and passed it: 130.7 tok/s against llama.cpp's 110.0, no speculative decoding on either side.
Read
