◆projects · mojo-neuron

vv1.0.0 · Apache-2.0 · GitHub

Abstract teal and amber illustration for mojo-neuron

mojo-neuron reads the neurons of a GGUF language model and shows what each one does. It computes the logit lens of every neuron in a layer, which is the set of tokens a neuron pushes up and down. It shows where a neuron fires on your text, and it steers and ablates neurons on a GPU. Everything is written in Mojo. Every tool has a gate with a planted defect, so a check that cannot fail is caught too.

The problem

Interpretability results are easy to overstate. A tool that works on one prompt can look like a finding. mojo-neuron was built to fix a bar before each experiment and to publish the result against it, whether it passed or missed. The README lists the misses next to the passes, and the reports in docs/reports/ hold the numbers.

How it works

mojo-neuron illustration

For a chosen layer, the lens maps each neuron onto the output vocabulary and keeps the top and bottom tokens. A quick check runs an f64 brute force on sampled neurons. Rank order must match, and a reversed and a neighbour-neuron mutant must differ.

The CPU forward pass shows where a neuron fires on text. It covers dense llama, qwen2 and granite models, and its output is checked against llama.cpp. The GPU work runs through the mojo-baro engine, which is built and baked separately. The engine handles steering, sweeps, ablation, activation capture, implants and live chat.

The same calls are available as a command line (nx), as an MCP server on stdio, and as a small window app.

What it does

  • lens MODEL LAYER K [CHECK] [--out DIR]: the logit lens of every neuron of a layer, as text or as a lens.bin.
  • nx models, nx lens, nx search_lens: read a saved lens, and search it for a token.
  • nx fires: the activation of one neuron on each token, on the CPU.
  • rank SWEEP.jsonl: word-cluster mass of steered runs, against random-direction controls.
  • rlens MODEL CORPUS: per-layer ridge transport maps, the R-style lens.
  • cvec MODEL L:N:dose OUT: a llama.cpp control vector and the engine's steer file.
  • sweep and reverse: neurons, doses and prompts in one resident engine process, with a steer lens, gain ablation and persona contrast.
  • ablate-scan and surgery: an ablation scan with a noise-floor pass, and an exact zeroing of one neuron's ffn_up row.
  • capture (BARO_CAPTURE): per-token FFN input, FFN output and residual, as f16.
  • tc_train, tc_splice, tcceil, xmap: a TopK transcoder trainer, a spliced loss, a linear ridge ceiling, and a cross-model map.
  • nx concept and nx probe: concept arithmetic, and a live lie score per token.
  • nx lens-diff, nx tlens, nx cast: a contrastive lens, a tuned lens, and a request-level condition gate.
  • nx sound, scan, replay, persona, fingerprint, radar: a WAV capture, a live heat strip, replay, a named persona store, a model fingerprint, and an anomaly digest.
  • implants: a low-rank side FFN per layer, in the engine.
  • .work/mcp and .work/app: the same calls as an MCP server, and a window with a neuron map, steered chat, a dial and a replay view.

mojo-neuron map view

Numbers that passed

  • The lens and its gate took 11 seconds for layer 12 of Qwen2.5-0.5B on a CPU.
  • The steer lens covered 4864 neurons and 64 controls at plus and minus 0.5 in 28.8 to 29.0 s. gguf-neuron-explorer took 119.4 to 128.9 s on one 0.5B model, with that tool's own contexts, a ratio of 4.3.
  • The steering decode cost was measured at 0.7%, with 606.5 and 602.0 tokens per second as the two figures.
  • Against llama.cpp, the steer lens's top-8 overlap was 8/8, 6/8 and 7/8, with KL within 3%.
  • Gain ablation matched a Hugging Face fp32 hook with a relative error of 0.057, against a bar of 0.10.
  • The tuned lens's held-out KL to the teacher was 3.94 down to 1.19, against 6.66 down to 4.36 for the logit lens. It beat the logit lens at all six layers tested, on narrow data of 256 positions.
  • The lie probe reached an AUROC of 0.87 to 0.91 on plain and paraphrased statements.

Numbers that missed

mojo-neuron illustration

  • The lie probe on negated statements reached an AUROC of 0.331, against a bar of 0.70.
  • The transcoder reached a held-out FVU of 0.641 with K=64 on 380,000 tokens. The bar was 0.55, and a dense linear ridge reached 0.520.
  • The CAST condition gate fired falsely on 12.3% of English statements, against a limit of 5%.
  • The MoE router capture matched the llama.cpp router in 198 of 240 cells. The bar was every cell.
  • The logit lens, used as a guide, was at chance up to layer 12 of 24.
  • Three pre-registered model-dream runs failed.

Install and first run

mojo-neuron list view

You need the Mojo toolchain that the sibling mojo-gguf checkout pins, which is Mojo 1.2.0 at the time of writing. You also need mojo-gguf (imported as ggufx) and mojo-json. The window app needs mojo-imkit v0.8.0. mojo-log and mojo-subprocess are needed for the engine tools. LLAMA_BIN points at a llama.cpp build, which serves as the oracle and the tokenizer for the gates.

M=mojo; GG=../mojo-gguf/src; J=../mojo-json/src
mkdir -p .work
$M build src/lens.mojo -I src -I $GG -I $J -o .work/lens
.work/lens qwen2.5-0.5b-instruct-q4_K_M.gguf 12 10 8

The last command reads layer 12 and keeps the top and bottom 10 tokens of each neuron. The eight is the number of neurons checked against the brute force. Add --out DIR to write lens.bin, then ask questions through nx with MNEURON_DIR set to that directory. scripts/check.sh runs the CPU gates in about three minutes and ends with PASS check.

The weights-only tools, lens, rank, rlens, cvec, connections, recipe and fires, need no GPU and no engine.

Limits

  • Tested on one machine with one AMD RX 7900 XTX, 24 GB, through ROCm. Other GPUs are untested.
  • The engine work covers 0.5B to 4B dense models and one 9B hybrid. The 27B build is only partly working.
  • The CPU forward pass exists for dense llama, qwen2 and granite models only.
  • The R-style lens here is a ridge map. It is not the Jacobian R lens, which is not built.
  • The transcoder did not beat a linear map at the data sizes tried, up to about 380,000 tokens on one layer.
  • The lie probe reads a claim inside a sentence, not the sentence itself, and it needs a threshold for each context.
  • Engine capture runs one position per step unless the batched path is used. The two captures differ slightly, so do not mix them in one training set.
  • The steer lens refuses contexts over 96 tokens.
  • nx --help is not a command. The call list is printed on an unknown call.

mojo-neuron is public on GitHub at https://github.com/amarbarosys/mojo-neuron under the Apache-2.0 license. The v1.0.0 release is dated 2026-11-09. The README, skills/mojo-neuron/SKILL.md and docs/experiments.md give the full list of calls and what each gate covers.