◆projects · mojo-baro

vunreleased (no tag yet) · Apache-2.0 · GitHub

Abstract teal and amber illustration for mojo-baro

mojo-baro runs open-weight language models on one AMD Radeon RX 7900 XTX. It reads a GGUF file, builds an engine for that model's shape, and serves chat over HTTP. Every GPU kernel in it is written in Mojo.

The problem

The RX 7900 XTX has 24 GB of VRAM, more than most consumer cards. The vendor libraries for decode and matrix multiplication are not tuned for these cards, so the question is how much speed they leave out. mojo-baro measures that gap. Each result in the repository has a preregistered protocol in bench/, a check you can run, and a date. Rows measured on an older toolchain say so.

How it works

mojo-baro illustration

Three parts. baro-serve is a Rust HTTP front. It handles the chat template, the tokenizer, tool calls, JSON-schema masks, the request queue, cancellation, per-message checkpoints and /health. It talks over a line protocol (serve/PROTOCOL.md) to an engine process written in Mojo.

The engine is compiled for one model shape. Layer count, hidden size and head counts are compile-time constants, so the loops have fixed trip counts and the compiler can unroll them. The cost is that an engine built for a 9B pack refuses a 27B pack.

Weights are not read from GGUF at run time. tools/baro converts the GGUF once into a pack: one flat pack.bin with weights laid out for the kernels that read them, an index, the tokenizer, and an identity file that names the source by sha256. A pack has q4 or q8 weights (requantised), native K-quant weights (the GGUF's Q4_K and Q6_K blocks, decoded inside the kernel), or split packs for MoE experts held in host RAM.

Every response reports which decode path served it. mega runs one persistent kernel per token for the whole forward pass, the dense default. launch runs one kernel per operation, the reference path and the MoE default. spec is speculative decoding with the model's own MTP draft head, and its output is identical to greedy decoding. prefill runs the prompt in batched chunks through WMMA matrix multiplies.

KV cache is f32 by default, or int8 with BARO_KVQ=int8 on dense models: 33,280 bytes per token on the 9B.

Capabilities

Command line, through tools/baro:

  • serve MODEL.gguf [--port N]: builds the engine and the pack for the file's shape the first time, which takes a few minutes, caches both under ~/.cache/baro/, and serves. Later runs of the same file start in seconds.
  • chat: a terminal client. It builds baro-chat on first use.
  • gui [--model MODEL.gguf] [--port N] [--fps N]: one desktop window with a streaming chat pane, an engine panel, settings tabs, presets, and an Apply step that restarts the engine with new settings. The frame cap defaults to 30 fps.
  • pack-path MODEL.gguf: prints the cached pack path and says PACK FOUND or PACK MISS. It never builds.
  • receipts MODEL.gguf: lists the GGUF receipt for the file.
  • verify: checks an engine file's embedded harness against git at the commit it names.
  • run MODEL.gguf: runs a file from the harness it carries, without the source checkout.
  • eval MODEL-KEY [--quick N] [--dry-run]: a quality run for one key in bench/quality-models.json.
  • eval-all: the quality sweep, bench/quality-sweep.sh.
  • multi-serve MODEL.gguf --devices 0,1: starts one baro-serve node per GPU. baro-router puts the nodes behind one port and routes by model and load.
  • swarm: runs tools/gpu_512_persona_swarm.mojo.

HTTP API, from baro-serve:

  • POST /v1/chat/completions, the OpenAI shape, with stream: true for server-sent events.
  • POST /v1/completions, GET /v1/models, POST /v1/cancel and POST /v1/fork, which copies a session's state to a second engine.
  • POST /tokenize and /detokenize, which match llama.cpp's output on the same GGUF.
  • GET /health, which returns 200 only when an engine can take work, and GET /props for the engine limits.
  • Ollama-compatible routes: /api/chat, /api/generate, /api/tags, /api/ps, /api/show and /api/version.
  • POST /v1/embeddings and /api/embeddings, using bge-small-en-v1.5 (384 dimensions) by default.

Results

mojo-baro illustration

Measured on one RX 7900 XTX. Decode is greedy, with no speculation on either side unless the row says so.

model, weights llama.cpp tok/s mojo-baro tok/s ratio date
Qwythos-9B, Q4_0 both sides, 20-prompt median 111.34 148.89 1.34x 2026-10-12
Qwythos-9B, Q8_0 both sides, same stint 75.46 80.38 1.07x 2026-10-26
Ornith-1.5-9B Q4_K_M, native K-quant pack, megakernel 94.14 99.53 1.06x 2026-10-26
  • Speculative decoding with the MTP head at k=2 gives 1.104x on a 20-prompt median, with identical output.
  • Prefill from 8k to 32k context is within about 1.3x of llama.cpp. On a K-quant pack, 16,384 prompt tokens take 4.54 s against llama.cpp's 5.66 s.
  • At the single-token decode shape (K=4096, N=12288) with the weights rotated out of the 96 MB Infinity Cache, the skinny M=1 kernel takes 121.2 to 121.8 µs per launch, and hipBLASLt f16 takes 122.1 to 123.3 µs. The ranges do not overlap, but the margin is about 1%.
  • An fp16 WMMA GEMM runs at 1.01x to 1.35x hipBLASLt for sizes from 256 to 3584. These sizes fit in the cache for both sides, so this is a warm-cache comparison.
  • Fifteen models across six architectures are listed in the README. Qwen2.5-7B-Instruct Q4_K_M decodes at 130 to 133 tok/s. A 27B model fits in 24 GB with its embedding table in host RAM, at 17.0 GB peak VRAM.

Install and first run

You need an RDNA3 card (gfx1100, 24 GB), Linux with ROCm 7.2, uv, Rust, CMake with a C++ compiler, and jq and curl. 32 GB of RAM is enough for 9B models. No weights ship with the repository.

git clone https://github.com/amarbarosys/mojo-baro && cd mojo-baro
uv sync
(cd serve && cargo build --release)
bench/preflight.sh
tools/baro serve ~/Models/Qwen2.5-7B-Instruct-Q4_K_M.gguf --port 8080
curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' -d '{
  "messages": [{"role": "user", "content": "Name the capital of France in one word."}],
  "max_tokens": 16000
}' | jq -r '.choices[0].message.content'

bench/preflight.sh needs no GPU and no weights. uv sync pins max[all]==26.7.0, which is Mojo 1.2.0. Any OpenAI client works with base_url="http://localhost:8080/v1".

Limits

  • One card is measured. Other RDNA3 cards use the same instruction set, so the kernels should build, but their tuning is not measured.
  • RDNA4 (gfx1200, gfx1201) builds and passes an emulated check on gfx1100. No RDNA4 card has run it, so its numbers are unverified.
  • The 1.34x headline was measured on Mojo 1.1.0. The repository now pins Mojo 1.2.0. On 1.2, the K-quant megakernel reads 87.90 tok/s against the launch path's 98.29 on the same pack. The cause is open. BARO_MEGA=0 selects the launch path.
  • The CPU tier is correct and slow. It decodes at 6 to 7 tok/s with 8 threads, and at 0.48 tok/s beside one busy job. It runs only the dense qwen35 shape.
  • Each model shape needs its own engine build. The first run for a shape takes a few minutes.
  • The GUI's Apply restart has been tested only with a fake command, not on a real engine.
  • The Qwen2.5-0.5B row stays below the agreement bar, and its figures are unverified.

Source and license

Source: https://github.com/amarbarosys/mojo-baro (public, Apache-2.0). Protocols, receipts and the full results are in bench/ and docs/.