▤ Category

Systems design

Posts in this category: 3

← Alle berichten
Systems design A 35B MoE on one consumer card · part 1

Experts in Host RAM

The routed experts of a 35B MoE moved to host RAM, VRAM footprint down to 2.68 GB from 21, output identical on 20 of 20 prompts. The first working version ran at 39 tok/s. Pinning the host store and reading misses in-kernel got it to 72.

Read