Same Bits on the CPU and the GPU

Std.math exp on Float32 does not return the same bits on the CPU and the GPU. Over 40,017 inputs from a seeded generator, the two disagree on 19,437 of them, which is 49%. Any claim that mojo-baro's CPU backend computes the same numbers as the GPU fails at the first softmax.

This post covers closing that gap for one function, what closing it bought, and where it stopped. The gap closed for exp. One attention layer then matched bit for bit. The per-token certificate built on top of that works in a test harness, and nobody has timed it in decode.

The GPU's exp is not reproducible from outside

The first idea was to copy the GPU. On this card the result of std.math exp is v_exp_f32(fl32(x * log2e)), and the hardware unit is undocumented. Modeling it with a correctly rounded exp2 on the CPU matched 96.6% of outputs bit for bit, and the rest came out 1 ulp off (commit 8a7e000). The remainder is the part of the unit we could not model, so that route cannot reach 100%. We dropped it.

The second idea was to stop depending on either side's transcendental. fexp32 is one Float32 exp written in Mojo: Cody-Waite range reduction, a cephes polynomial, and an explicit fma wherever a compiler could otherwise contract a multiply and add differently on the two backends. Both sides run the same source.

Measured against a correctly rounded mpmath oracle, on the softmax range [-30, 0] (newera/.work/f32math/*.split, from tests/check_f32math.sh):

Across the full range (-104, 89), fexp32 is at most 1 ulp on both backends, and CPU and GPU differ on 0 of 40,017 inputs. The GPU's own answer is 28 ulp off on the softmax range, so the shared version is also the more accurate one there. flog32 got the same treatment in commit c24044d, with a planted-bug mutant that must go above 1 ulp.

The first flog32 draft showed how fragile this is. It differed on 62 of 40,017 inputs. Writing 0.5 * f * f as an explicit fma(0.5 * f, f, 0) made it 0 of 40,017. The rule that fell out: a product that feeds a plain add goes through fma.

Per-op identity does not compose on its own

A passing op is not a passing layer. Three things broke the first time through.

The first was the dot product. One q4 row dot, CPU emulation against the launch-path GPU kernel, differed on 0 of 12,288 dots over real rows from layer 3 and four embedding rows. The lane accumulator had to fold in the same order as the GPU's DPP chain. A mutant that folds it as a wave tree by halves fails 7,668 of 12,288, so the test can fail.

The second was the reduction. The GPU contracts warp.sum(v * v): its first DPP level becomes fma(v, v, partner * partner). Without modeling that, head rmsnorm differed on 1,554 and 199 elements in two test sets. The CPU side now models the full xor 1, 2, 7, 8, 16 chain per lane.

The third was the layer. One attention layer runs 14 stages over 8 steps, with positions up to 5,000: rmsnorm, seven q4 GEMVs, the qgate split, head rmsnorm, rope, online-softmax attention over 256-key chunks, gate multiply, residual adds and swiglu. Under -D BARO_SHARED_EXP=1 every stage is bit-equal between the CPU and the GPU. A planted 1.0000001 on the rope sincos argument fails the rope stage and no other.

The default build does not match. In the default-flag run, head rmsnorm has 9,251 mismatches in q and 2,367 in k, rope 2,673 and 670, and attention 5,192 (exchange/receipts/cpu-parity/1-gate-layer-default-flags.txt). The flag swaps exp, the reciprocal square root and sincos together, so identity holds only with all three on the shared routes.

A gate that read an unmutated file

The mutants had their own bug, and it is the failure this post exists to warn about. The first certificate run read "caught on 0 of 7". Both the mutant build and the main build had silently compiled the unmutated matmul_skinny.mojo, because a test file's own directory wins over -I. The earlier layer-gate digest comparison was vacuous for the same reason. It was rerun from a copied tree and still passes with the same digest. The record is in exchange/lane-certificate-report.md.

One source per function, one planted fault per gate, and a build that provably reads the file you broke.

What counts:

• CPU and GPU outputs compared bit for bit on a seeded input set.
• A mutant that makes the same gate fail, built from a copied tree.
• A flag state named in the claim, because the default build does not match.

What does not count:

• A max-ulp bound without a bitwise comparison.
• A stage whose bf16 output hides 1 ulp differences in the f32 scale. That makes the rmsnorm evidence weak.
• A digest compare that cannot tell whether the file was mutated.

The CPU as a checker: a float Freivalds certificate

The reason to want identity is that a bit-identical CPU can check the GPU. The certificate tests each sampled GEMV y = Wx with a random vector r of plus and minus ones: it asks whether |rᵀy minus (rᵀW)x| is below a bound. The host precomputes rᵀW once per matrix, at 8 bytes per input column, and each check takes 15.9 µs on the host.

The bound was calibrated on one seed (commit 998b56f), where the largest honest residual ratio was 0.057 and the frozen constant is 0.125. A second seed passed 336 of 336 honest checks, largest ratio 0.064. A planted +0.05 and a planted +0.005 on one GEMV output were each caught on all seven matrices at every step. The detection floor is about 6e-5 absolute per element. Receipts: exchange/receipts/cpu-parity/2-cert-*.txt, gate bench/check-certificate.sh.

We chose a float bound rather than exact arithmetic mod p, because the production q4 GEMV keeps no integer partial sum to certify. That choice has a known hole. A multi-element fault whose contributions cancel under r escapes with probability 1/2 per matrix. Checking more matrices per token shrinks that, and we have not measured how many it takes.

Prior art covers half the problem. Thinking Machines' batch-invariant kernels stop run-to-run drift on one GPU stack, with no cross-device claim. The LayerCast paper (arXiv 2506.09501) reports divergence across GPU types. TOPLOC (arXiv 2501.16007) hashes activations and verifies by re-running a fraction of the computation, which is a different check from a per-matmul certificate. We read abstracts and the blog page, not the full papers. None of them describes CPU and GPU identity combined with a per-matmul random-projection check on a local engine. That gap may already be filled by something we did not find.

What it cost

The shared exp is slower. In a microbenchmark of 16M threads, each doing 16 dependent exp calls, std.math took 0.168 ms median and fexp32 took 0.609 ms, a ratio of 3.6 (newera/.work/f32math/logs/bench.txt, commit af914d0). The std.math arm is near memory bound and fexp32 is ALU bound.

Pricing that into decode is arithmetic, not a run. At 8k context a token needs about 8.4M exp calls (32 heads, 32 layers). At the measured per-call difference that is about 14 µs of extra work, roughly 0.1% of a 10 to 20 ms token. We have not measured it end to end, and we will not quote it as a decode result until we do.

Also unpaid:

• The certificate is not hooked into the engine. It runs in a test chain that reads GPU buffers. Hooking it means async copies in step_window on the launch path.
• The megakernel keeps x and y inside one persistent kernel. Certifying it needs kernel changes.
• Decode cost with certificates on is unmeasured. This boot's PCIe link trained down to 2.5 GT/s, and device-to-host copies at that speed would make any timing void. The number waits for a reboot.

The layer covers one attention layer. SSM layers, embedding, final norm, LM head, sampler and MoE are not covered. There is no end-to-end token or logit identity yet.

How to prove this wrong

Run bench/check-shared-math.sh at 04ab7dc on an RX 7900 XTX. A single input where CPU and GPU bits differ, or where either side passes 1 ulp against the oracle, defeats the claim for exp and log. For the layer, run bench/check-cpu-layer.sh at ada7420 with -D BARO_SHARED_EXP=1; any stage with a mismatch defeats the layer claim. The strongest test is the one not yet run: a teacher-forced pass over 20 prompts against the GPU engine. One logit or one greedy token that differs between the CPU path and the GPU path means the per-op results do not compose, and the CPU is not a trustworthy reference for the GPU. Send us the input.

Provenance

One RX 7900 XTX (gfx1100), ROCm 7.2.4, Linux 7.2.11, Mojo 1.1.0, measured on 2026-10-21. One card shows these results hold on this card, and says nothing about another architecture's exp unit. Commits in newera: 176f817 (the 19,437 of 40,017 difftest), 8a7e000 (fexp32), af914d0 (microbenchmark). In mojo-baro: d88764f and 04ab7dc (shared exp), c24044d (flog32), 8a10649 (shared rsqrt, sin and cos), 998b56f (certificate bound), ada7420 (CPU parity merge).