The Bottleneck Was the PCIe Link, Not the Code

A decode trace of mojo-baro's engine said the GPU sat idle for 20.5 ms of every token, 60 percent of wall time. The largest slice of that idle was expert-miss DMA: 11.3 ms per token spent copying weights for experts that were not resident. Those copies ran at about 3.1 GB/s. The link they crossed had trained down to 2.5 GT/s. The kernels were not what the trace was waiting on.

We spent an hour on the host side before finding that out, and the link was already a logged fault.

The trace pointed at copies, not at compute

The lane traced 32 decode tokens with rocprofv3, kernel and memory-copy records together, and bucketed every gap between consecutive kernel dispatches by the kernel before it. The window was 1091 ms and the GPU was idle for 656 ms of it. Kernel busy time was 14 ms per token (commit 99709a9 on the unmerged lane-q4x-host branch, receipt exchange/receipts/q4x-host/gaps-base-ng33.txt, engine from main d181a44).

Each miss issued three SDMA copies back to back on the main stream (gate, up, down, 460,800 bytes each), about 150 us apiece. One missed expert cost 451 us of idle, two cost 840 us, three cost 1413 us. The host was not the limit: enqueue work came to about 3.3 ms per token. The tracer stretches launch gaps, so the absolute numbers carry that caveat. The bucket shares match an earlier 63-token trace (exchange/lane-q4x-rows-report.md), which is why we trusted them.

Three copies at 3.1 GB/s is a link number

Each copy moved 460,800 bytes in about 150 us, which is 3.1 GB/s. By the same arithmetic the same copy at the 28.5 GB/s this card reached on a healthy boot takes about 16 us. The copy code was not slow. The wire was.

Raw probes agree. Pinned host-to-device copies of 460,800 bytes, 4 MB and 64 MB all ran between 3.1 and 3.6 GB/s (device-to-host 2.7 to 2.8), with the GPU busy or idle and with SDMA on or off (exchange/receipts/q4x-host/pcie-probes.txt). The kernel log said why:

pci 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x16 link at 0000:00:01.1 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)

0000:00:01.1 is the CPU root port, and it negotiated 2.5 GT/s where it is capable of 16. The card's own endpoint at 0000:03:00.0 reports 16 GT/s, but that is the internal link behind the card's switch. Read the root port or you read the wrong number.

The root port does not train low on every boot, and it has done this before:

The software settings were clean: the PCIe generation and lane caps, ASPM, and the kernel command line, with zero AER correctable errors on both ports. So the link is not erroring. It negotiated low at boot. Why it does that on some boots is not established, and our fault note still says unverified.

The same trap, one level up

A reboot is where this fault gets cured, which makes it a cousin of A Reboot Is Not a Health Check. That post asked whether the services came back. This one asks whether the hardware came back at the speed every timing assumes. A GPU that is present, bound and answering is not a GPU on a 16 GT/s link, and nothing in a service check reads the root port.

Our own tooling had the same blind spot. The fault had been logged since 2026-09-11, and our recall for "q4x host idle synchronize" did not surface it, because the note says "PCIe", "link" and "downtraining". The timing lanes also never read the link before they timed anything. That gap was ours, and bench/pcie-link.sh now writes the root-port link into the A/B receipts and flags the PASS line when it is not 16 GT/s.

Read the link before you time anything that crosses it.

What counts:

• current_link_speed of the root port at 00:01.1, read in the same boot as the timing.
• A receipt that carries the link speed beside the number.
• A same-boot ratio between two arms that both ran on the slow link.

What does not count:

• The endpoint's own 03:00.0 speed, which always says 16 GT/s.
• A millisecond figure from a boot with the link down, quoted against a healthy one.
• A kernel conclusion drawn from a trace whose gaps are copies.

What it cost

Every timed comparison that moves expert or host-device bytes on that boot is void. That is the MTP window A/B (B/A 1.069 at K=2 and 1.095 at K=4), the MoE expert-capacity A/B, the idle shares from the rows trace, and a quick-speed arm that read 47 to 49 ms per token against a 20 ms receipt. A Q4X_DEVARG A/B read A 49.0 and B 46.0 ms per token, B/A 0.938, with per-run spread up to 2378 percent, so we recorded it as not a claim.

One result survives. The q4x dispatch fusion cut dispatches per decode token from 1244 to 992, and its same-boot ratio was B/A 0.9536 with B below A on 20 of 20 prompts (exchange/receipts/q4x-dispatch/ab-ng64-receipt.txt, commit d422318). Both arms ran on the same slow link, so the ratio is still a fair comparison. The milliseconds from that boot (A 42.5 ms per token against about 20 ms warm) are not comparable to a healthy boot, and we are not quoting them.

The fix is not in yet. Moving the BIOS x16 slot from Auto to Gen4 gave one boot at x4 with host copies at 7.06 GB/s, then one boot at x16 with a pinned 40 MB copy at 28.23 GB/s. One good boot is not the several the fix needs, so the fault stays open. A reboot usually clears it and it recurs. On 2026-10-21 the reboot was postponed, and the root port still reads 2.5 GT/s x16 on a boot that started 2026-10-19 06:10:57.

How to prove this wrong

The claim is that the link, not the code, set the idle time. Reboot, read /sys/bus/pci/devices/0000:00:01.1/current_link_speed, and confirm 16.0 GT/s. Run a pinned 40 MB host-to-device copy and require 25 GB/s or more. Then repeat the 32-token rocprofv3 trace with bench/q4x-rows-trace.sh and bench/q4x-gaps.sh. If the miss-DMA bucket still sits near 11 ms per token on a full-speed link, the copies were not the whole story and the kernel question reopens. If the link trains low again on the next boot with the same settings, tell us your board and BIOS.

The scope is narrow: one card, one box, one prompt of 32 tokens. The dispatch ratio is the only comparison from the slow boot we are keeping.

Provenance

RX 7900 XTX (gfx1100) behind root port 0000:00:01.1, ROCm 7.2.4, rocprofv3 1.1.0, Mojo 1.1.0. Boot of 2026-10-19 06:10:57, trace of 2026-10-21. Commits in mojo-baro: 99709a9 (lane report, probes, trace), d422318 (dispatch ratio), d181a44 (engine under test). The fault note records the 2026-09-09 and 2026-09-11 boots, with the same kernel log line.