Systems design
A 35B MoE on one consumer card · part 1
Experts in Host RAM
The routed experts of a 35B MoE moved to host RAM, VRAM footprint down to 2.68 GB from 21, output identical on 20 of 20 prompts. The first working version ran at 39 tok/s. Pinning the host store and reading misses in-kernel got it to 72.
Read