vv1.0.0 · Apache-2.0 · GitHub
wavewait shows where a GPU kernel waits. For each instruction of a gfx1100 kernel (RDNA3, such as the RX 7900 XTX), it reports how many cycles each wave spent waiting before that instruction issued, and it gives a cause for each wait. The input is the thread trace the GPU emits, not a sampled estimate.
A profile that reports busy time can hide the reason a kernel is slow. An s_waitcnt that waits 670 cycles for a memory load looks like one cheap line in a listing. It is the cost that matters. wavewait charges each wait to the instruction that held the wave, and sorts the costs by cause: memory, LDS or scalar memory, barrier, dependency, or instruction issue.

There are two capture paths.
tools/att-capture.sh wraps it with a 256 MB buffer cap, a timeout, and a watcher that kills the run when ROCm reports a failed packet buffer or a resource deadlock.tools/sqtt-capture runs it. This path has more limits, listed below.wavewait decodes the packets itself, in Mojo. The nibble state machine comes from tinygrad and is generated into src/sqtt_table.mojo. The walk matches each packet to an instruction by its class, and charges the cycles. The walk is checked against AMD's own decoder: tools/amd_oracle.py compares every ATT fixture with AMD's code.json, and each s_waitcnt must agree within one cycle per wave.
With --causes, the cost of each wait moves onto the s_waitcnt or s_barrier that held the wave. Each row then gets a label: wave start, memory, LDS or scalar memory, barrier, dependency, and so on. On the saxpy example the causes split 27% wave start, 25% LDS or scalar memory, 21% memory, and 21% memory plus LDS.
wavewait DIR [--top N] [--causes]: the per-instruction stall table, with the cause summary. --top N cuts it to the costliest rows.wavewait rank RUNDIR: every kernel of a whole process, from a rocprofv3 kernel trace, with its share of time and dispatch count.wavewait diff A B: aligns two captures by instruction text and shows the per-row change, the cause shift, and the total cycles.wavewait diff RUN_A RUN_B --kernel K: the same for one dispatch of two ATT run directories.--by-class: cycles, share and issues per instruction class. With --watts and --tok-s it adds an energy-per-token view, labelled as an upper bound.--exposed: cycles the traced SIMD sat idle, shared among the waves alive at the time, with occupancy.--loops: charged cycles per innermost loop, with polling loops flagged as spin.--hints: fix hints from the measured gfx1100 timing, and a count of waves that took a branch a different number of times.--html OUT: a self-contained per-wave timeline page.occ CODEOBJ: waves per SIMD the code object allows, and which resource limits them.budget FILE DIR: a CI gate on cycles N and cause NAME PCT rules. It exits 1 and names each broken rule.app RUNDIR: every dispatch of an ATT run, one row per kernel.--json: rows and totals with stable keys, for scripts.
--exposed pointed at a grid barrier in the delta-scan kernel, with 64 of 96 blocks idle. A change in mojo-baro commit 1d89111, which moved the z projection, gave +1.7% tokens per second on Mojo 1.1 and +2.7% on Mojo 1.2. The token ids matched in 20 of 20 checks.The CPU gate is pixi run check, which runs check.sh. It checks the saxpy table against a frozen golden table, conserves total cycles, and checks each budget rule.
The repository pins Mojo 1.2.0 and Python 3.13 in pixi.toml. It also pins a tinygrad commit, which is needed only to unpickle the fixtures. A capture needs hipcc, rocprofv3, and AMD's rocprof-trace-decoder library. tools/doctor.sh reports what the machine still lacks, with the fix for each item.
hipcc --offload-arch=gfx1100 -O3 examples/saxpy.hip -o .work/att/saxpy
rocprofv3 --att --att-target-cu 3 --att-shader-engine-mask 0x1 \
--att-library-path <decoder dir> -d .work/att/run -- .work/att/saxpy
tools/from-att.sh .work/att/run .work/att/saxpy_att
build/wavewait .work/att/saxpy_att --causes --top 10
The launch needs enough workgroups that CU 3 gets some. The example launches 192. A single workgroup traces nothing on that path.
--exposed sees one SIMD. AMD's viewer ranks pipes instead, so the two numbers differ.--allow-mism shows it anyway, with a warning.--by-class energy split is a cycle-share model, not a measured split of power.s_waitcnt rows after ds_store_b128 differ from AMD's decoder by about 2.4 cycles per hit. The totals stay within 0.12%.sim is not built. It needs a per-SIMD scheduler model, and it exits with status 2.wavewait --help is read as a file name and fails. The subcommand table in the README is the reference.wavewait is public on GitHub at https://github.com/amarbarosys/wavewait under the Apache-2.0 license. The v1.0.0 tag is dated 2026-11-07.