◆projects · wavewait

vv1.0.0 · Apache-2.0 · GitHub

Abstract teal and amber illustration for wavewait

wavewait shows where a GPU kernel waits. For each instruction of a gfx1100 kernel (RDNA3, such as the RX 7900 XTX), it reports how many cycles each wave spent waiting before that instruction issued, and it gives a cause for each wait. The input is the thread trace the GPU emits, not a sampled estimate.

The problem

A profile that reports busy time can hide the reason a kernel is slow. An s_waitcnt that waits 670 cycles for a memory load looks like one cheap line in a listing. It is the cost that matters. wavewait charges each wait to the instruction that held the wave, and sorts the costs by cause: memory, LDS or scalar memory, barrier, dependency, or instruction issue.

How it works

wavewait illustration

There are two capture paths.

  1. rocprofv3 ATT, for any HIP program. AMD's thread trace is taken from compute unit 3 in shader engine 0. tools/att-capture.sh wraps it with a 256 MB buffer cap, a timeout, and a watcher that kills the run when ROCm reports a failed packet buffer or a resource deadlock.
  2. tinygrad's KFD runtime, for one workgroup of a kernel. tools/sqtt-capture runs it. This path has more limits, listed below.

wavewait decodes the packets itself, in Mojo. The nibble state machine comes from tinygrad and is generated into src/sqtt_table.mojo. The walk matches each packet to an instruction by its class, and charges the cycles. The walk is checked against AMD's own decoder: tools/amd_oracle.py compares every ATT fixture with AMD's code.json, and each s_waitcnt must agree within one cycle per wave.

With --causes, the cost of each wait moves onto the s_waitcnt or s_barrier that held the wave. Each row then gets a label: wave start, memory, LDS or scalar memory, barrier, dependency, and so on. On the saxpy example the causes split 27% wave start, 25% LDS or scalar memory, 21% memory, and 21% memory plus LDS.

What it does

  • wavewait DIR [--top N] [--causes]: the per-instruction stall table, with the cause summary. --top N cuts it to the costliest rows.
  • wavewait rank RUNDIR: every kernel of a whole process, from a rocprofv3 kernel trace, with its share of time and dispatch count.
  • wavewait diff A B: aligns two captures by instruction text and shows the per-row change, the cause shift, and the total cycles.
  • wavewait diff RUN_A RUN_B --kernel K: the same for one dispatch of two ATT run directories.
  • --by-class: cycles, share and issues per instruction class. With --watts and --tok-s it adds an energy-per-token view, labelled as an upper bound.
  • --exposed: cycles the traced SIMD sat idle, shared among the waves alive at the time, with occupancy.
  • --loops: charged cycles per innermost loop, with polling loops flagged as spin.
  • --hints: fix hints from the measured gfx1100 timing, and a count of waves that took a branch a different number of times.
  • --html OUT: a self-contained per-wave timeline page.
  • occ CODEOBJ: waves per SIMD the code object allows, and which resource limits them.
  • budget FILE DIR: a CI gate on cycles N and cause NAME PCT rules. It exits 1 and names each broken rule.
  • app RUNDIR: every dispatch of an ATT run, one row per kernel.
  • --json: rows and totals with stable keys, for scripts.

Numbers

wavewait illustration

  • The walk on an 11 MB trace peaks at 24.0 MB of memory for the table. A 37 MB trace walks in 294 ms at 48.4 MiB, down from 579 ms, after a per-packet allocation was removed.
  • Diffing a kernel of 14,000 rows takes 10 s and 88 MB with the linear-memory alignment. The full alignment table for the same kernel took 1.2 GB.
  • The saxpy example, traced through rocprofv3, totals 38,342 cycles.
  • On 2026-10-30, the audit re-walked 136 dispatches from ATT runs, and all 136 agreed with AMD's decoder.
  • On mojo-baro, --exposed pointed at a grid barrier in the delta-scan kernel, with 64 of 96 blocks idle. A change in mojo-baro commit 1d89111, which moved the z projection, gave +1.7% tokens per second on Mojo 1.1 and +2.7% on Mojo 1.2. The token ids matched in 20 of 20 checks.

The CPU gate is pixi run check, which runs check.sh. It checks the saxpy table against a frozen golden table, conserves total cycles, and checks each budget rule.

Install and first run

The repository pins Mojo 1.2.0 and Python 3.13 in pixi.toml. It also pins a tinygrad commit, which is needed only to unpickle the fixtures. A capture needs hipcc, rocprofv3, and AMD's rocprof-trace-decoder library. tools/doctor.sh reports what the machine still lacks, with the fix for each item.

hipcc --offload-arch=gfx1100 -O3 examples/saxpy.hip -o .work/att/saxpy
rocprofv3 --att --att-target-cu 3 --att-shader-engine-mask 0x1 \
    --att-library-path <decoder dir> -d .work/att/run -- .work/att/saxpy
tools/from-att.sh .work/att/run .work/att/saxpy_att
build/wavewait .work/att/saxpy_att --causes --top 10

The launch needs enough workgroups that CU 3 gets some. The example launches 192. A single workgroup traces nothing on that path.

Limits

  • Only RDNA3 (gfx1100). Tested on one RX 7900 XTX.
  • The ATT path traces only the workgroups that land on CU 3.
  • The tinygrad path supports only CU mode, pointer arguments and a one-dimensional grid. Use the ATT path for real programs.
  • SQTT on gfx11 has no execution mask. Divergence is counted per wave, not per lane.
  • --exposed sees one SIMD. AMD's viewer ranks pipes instead, so the two numbers differ.
  • A walk whose packets do not match the code exits 3. --allow-mism shows it anyway, with a warning.
  • The --by-class energy split is a cycle-share model, not a measured split of power.
  • On one torch matmul, a few s_waitcnt rows after ds_store_b128 differ from AMD's decoder by about 2.4 cycles per hit. The totals stay within 0.12%.
  • sim is not built. It needs a per-SIMD scheduler model, and it exits with status 2.
  • wavewait --help is read as a file name and fails. The subcommand table in the README is the reference.

wavewait is public on GitHub at https://github.com/amarbarosys/wavewait under the Apache-2.0 license. The v1.0.0 tag is dated 2026-11-07.