The fly engine waits on memory, not math

One launch of the fly engine at 65,536 animals reads 40.0 GB from memory and writes 15.9 GB. We timed the launch at 66.7 ms. That is 838 GB/s, 87 percent of the card's 960 GB/s, and it is not a measurement: the bytes were counted in one clock mode and the 66.7 ms was timed in another. Divide the same bytes by the 99.2 ms the profiler recorded while it counted and it is 563 GB/s, 59 percent; that time carries the slower clock mode and the profiler's own overhead.

So the honest headline is a range, and the explanation we wrote down for the bytes was wrong. This is the sequel to part 7, and the same outside review is the reason there is a correction.

55.9 GB per launch is 59 to 87 percent of the card, and the clock mode picks which

The numbers, for the stage-7 bench (a standing, actuated fly, one thread per animal):

• Footprint: 35,309 float32 rows, 141 KB per animal, about 9.3 GB for 65,536 animals.
• Reads: 611 KB per animal per step, 4.3 times its own footprint.
• Writes: 242 KB per animal per step.

The counters only work in profile_standard (the next post, The profiler crashed because the CPU has a GPU, is the story of why). In that mode the kernel runs slower, and counting adds its own overhead on top, so neither time is the true one: 66.7 ms is too fast for these bytes, 99.2 ms too slow. The 87 percent pairs bytes from one mode with time from the other. We have not taken the same-mode measurement that would settle it.

Our first write-up said 52.6 GB and 790 GB/s, 82 percent, and called the step bandwidth bound. The review caught two errors in it: the write bytes did not follow the vendor's formula for combining two overlapping counters, and MiB were labeled MB. I recomputed from the saved counter files: 40.02 GB read, 15.87 GB written, 55.89 GB total. The conclusion that the step is heavy on memory survives the correction. The "82 percent" did not.

This is the stage-7 bench. It is a different benchmark from part 4's 104.4 ms figure, and I have not reconciled the two here.

Our first explanation of the bytes was wrong

The write-up said each animal reads 4.3 times its footprint because "the Newton loop re-reads J, D and H rows every iteration". It then ranked the fixes by that: put solver state in fast memory first, store the Jacobian more narrowly second.

The review counted accesses instead of allocations, from the source. Three results contradict the first explanation:

• A one-pass read of the live Jacobian is about 1.6 KB. There is not much there to halve.
• The Newton working set of about 9 KB is a capacity, not a read budget. A sparse factorization reads each entry many times.
• One factor of the Hessian stores 3.6 KB of entries and reads about 54 KB to build them, because every update reads both the target entry and the pivot-row entry.

Its ranked read budget, per animal per step (source-level estimates):

The review's total comes to 630 to 690 KB against 611 KB counted, so this is a budget, not an attribution of the hardware counters. It overshoots, and the review says so. The saved profile also reports zero scratch and zero LDS (the GPU's on-chip shared memory) for the kernel, so register spilling is not the story either.

Moving the state into fast memory has already lost once

Our own history said the same thing before the byte budget did. The probe in proto_factor.mojo factorized the fly's 74-dof mass matrix several ways. Microseconds per factorization:

A wave per animal is 10 times faster at 1,024 animals, where one thread per animal leaves the card idle. At 65,536 it is about 2 times slower: each row waits on two barriers, and packing several animals into a wave multiplies the LDS per wave until occupancy collapses. The whole step needs about 34 KB of LDS per animal, ten times the probe's share.

Then the byte-budget section ranked "per-animal solver state in LDS or registers" as lever one. That contradicts the probe above it in the same document, and the review pointed it out. Two other cuts bear on the title's other word:

• Incremental Hessian assembly (add and subtract only the rows that flipped): 86.25 ms against 88.9, 2.9 percent, reverted under our 3 percent bar. Our write-up concluded the sums for J'DJ were never the cost.
• A rank-1 update of the factor in place: 88.9 to 70.6 ms, 20.6 percent, cutting rebuilds from 4.33 to 1.0 per step.

Neither separates bytes from arithmetic cleanly, and we have not read an arithmetic counter. The "not math" half of the title is an inference from these two results, and the "memory" half rests on the byte count.

Count the accesses, not the allocation.

What counts:

• Bytes read and written per animal per launch, from counters, in the same mode as the timing.
• Per-function access counts derived from the source, labeled as a budget.
• A before-and-after on a prototype.

What does not count:

• The footprint (141 KB), which is capacity.
• A working-set size (9 KB), which is not a read bill.
• Dividing bytes from one clock mode by time from another.

What it cost, and what is not measured yet

The mistake cost us a ranking. We would have spent the next round narrowing Jacobian storage and building an LDS design that our own probe had already beaten at this batch size.

The planned fix is not measured. The review's first-ranked change keeps the solve vectors in registers and writes the result once. It estimates about 58 KB fewer reads and 28 KB fewer writes per step, worth 7 to 10 ms if bandwidth is the whole story. That is a prediction. Cache hits, instruction cost and occupancy can shrink it or turn it negative: the 74-dof solve wants a lot of registers, and the saved baseline already uses 112 vector registers per thread.

How to prove this wrong

Run the byte counters and the kernel time in one clock mode on one launch of the 65,536-animal bench. If bytes over time is under 480 GB/s, half of the card's 960, the bandwidth-bound claim fails. We currently have 563 GB/s on the profiled side, which is above that line, but only by 17 percent. Then build the register-resident solve and see whether the 7 to 10 ms shows up. If it does not, the budget is wrong in a way the counters cannot show.

Provenance: one RX 7900 XTX (gfx1100), the standing-fly stage-7 bench at 65,536 animals; bytes under profile_standard with the integrated GPU unbound, time under auto; counters and review in the fly-connectome repo (docs/plans/BODY_ENGINE_OPT.md, .work/reviews/codex-engine-review.md, agent/body_engine/traffic.sh). The 563 and 838 GB/s figures are my recomputation from the saved counter files.