The profiler crashed because the CPU has a GPU

For four days we believed that hardware counters were unusable with our Mojo kernels. The profiler crashed with a C++ std::out_of_range, then hung in a signal handler that SIGTERM could not kill. The verdict went into our notes: --pmc is dead for our kernels. It was wrong. A plain C++ kernel crashed the same way, and the cause was the integrated GPU inside the processor.

With that GPU unbound from the driver, SQ_WAVES read 32,768 for a 1,048,576-thread kernel. That is exact: 1,048,576 threads at 32 per wave.

The crash looked like a Mojo problem

On October 5 we wanted bytes per launch for the fly engine's memory budget. The profiler lists two derived counters for this, FETCH_SIZE and WRITE_SIZE. Both failed:

• Both together: Could not construct profile cfg failed with error code 38: Request exceeds the capabilities of the hardware to collect, then SIGABRT.
• FETCH_SIZE alone: terminate ... std::out_of_range ... unordered_map::at inside rocprof.
• After either, the process hung. timeout did nothing, because the hung handler survives SIGTERM. Only timeout -s KILL ends it.

Then even a raw counter, SQ_WAVES, crashed on a Mojo binary. Every crash was on a Mojo binary, so the conclusion wrote itself: our toolchain, our kernels. --kernel-trace kept working, so we had a fallback, and the verdict stood.

A plain kernel crashed the same way

On October 9 we ran a plain hipcc vector-add for gfx1100 under --pmc SQ_WAVES. It crashed identically, and the binary ran fine without the profiler. No Mojo in it. The box was the common factor.

Every crash log carried one line right before the out_of_range: rocprofiler_iterate_agent_supported_counters failed for agent 2 (gfx1036) :: Agent HW architecture is not supported. gfx1036 is the Ryzen's integrated GPU, a second GPU agent that the profiler walks and cannot handle. ROCR_VISIBLE_DEVICES=0 did not hide it from the profiler.

Unbinding it from the amdgpu driver at runtime removed the agent (echo <iGPU PCI address> | sudo tee /sys/bus/pci/drivers/amdgpu/unbind). The same plain kernel then passed: no crash, no hang, 32,768 waves.

A second failure was hiding behind the first

With the crash gone, a derived byte counter, FETCH_SIZE, read 0 for a kernel that reads 4 MB. So did every L2 counter, until the card's performance level was set to profile_standard (power_dpm_force_performance_level). The profiler's documentation says the auto level gates off the performance-counter clock in some blocks on RDNA3. In that mode the vector-add read 4 MB and showed:

• GL2C_MC_RDREQ: 32,769
• GL2C_EA_RDREQ_128B: 32,772

4 MB over 128 bytes is 32,768 requests, so the counters were right to within a handful.

We had tested this idea on October 8 and the note says "FALSIFIED." Under both auto and profile_standard the crash persisted, so the test said "performance level is not the cause." That was true of the crash. It said nothing about whether auto zeroes the counters, because a profiler that crashes never gets to return a value. Our October 9 ledger entry still lists "perf level falsified" among the things already ruled out.

We also debugged it a second time without reading our own note

This part is ours. On October 9 we re-debugged the crash from scratch: three GPU jobs, about twenty minutes, and then a proposal to set ROCR_VISIBLE_DEVICES=0. The note already held the verdict, the open lead about the integrated GPU, and the fact that ROCR_VISIBLE_DEVICES=0 does not hide it. We had not searched it first. The correction was caught when someone said we had dealt with this before. We logged it as a mistake in the lab ledger: recall before a profiler run, every time.

A crash in the instrument is not a result about the thing measured. Run the instrument on a kernel whose answer you know before you read anything into its failures.

The plain vector-add is that kernel. Its answer is known to the digit: 32,768 waves, 32,768 128-byte requests for 4 MB.

What counts:

• An exact count on a known kernel, from the same recipe you will use on the real one.
• A bisect that changes one thing: Mojo out, plain C++ in.
• A note that says what a test could and could not see.

What does not count:

• A counter that reads 0, or a crash, taken as "no effect."
• A "falsified" label that covers more than the test could reach.
• A verdict about the toolchain from a failure that never left the profiler.

What it cost

• The unbind has a side effect: the lactd service (which applies the RX 7900 XTX's power caps) segfaulted in libdrm_amdgpu and hit its start limit, so the caps were no longer applied. A reboot or a write to the driver's bind file undoes the unbind.
• The recipe is: unbind, set profile_standard, one counter per pass, timeout -s KILL, restore auto afterward. The two root steps have to happen before the job enters our GPU queue, because queued jobs cannot sudo.
• Times under profile_standard are not comparable with auto. That cost the fly engine's bandwidth number a clean answer: The fly engine waits on memory, not math has the range.
• Four days with a wrong verdict in the notes, and about twenty minutes repeating the work.

What this does not show

One machine, one ROCm (7.2.4), one profiler version (rocprofv3 1.1.0), one plain kernel. Writes read near zero on a 4 MB kernel, because the L2 is write-back and 4 MB never leaves it inside the kernel, so the write counters here prove nothing yet. FETCH_SIZE, the derived counter, has not been tested under profile_standard. We did not test turning the integrated GPU off in the BIOS, so we cannot say whether unbinding at runtime and disabling it are the same fix for the profiler. And we do not know whether the fault is in the profiler or the driver.

How to prove this wrong

On any box with a processor graphics unit and a discrete card, compile a plain hipcc kernel and run rocprofv3 --pmc SQ_WAVES first with the integrated GPU bound, then with it unbound from amdgpu, with timeout -s KILL around both. If the unbound run still crashes, or reads anything other than threads divided by 32, the claim is wrong for your box. If the bound run does not crash, the claim does not generalize past ours, and we would like to know which driver and profiler versions you have.

Provenance: RX 7900 XTX (gfx1100) with the Ryzen 7800X3D's integrated gfx1036, ROCm 7.2.4, rocprofv3 1.1.0, notes dated 2026-10-05, 2026-10-08 and 2026-10-09; the probe scripts (a HIP counter run and a perf-level run) are in the fly-connectome repo.