← Blog
✎post del blog

The slow part was a memory layout, not a kernel

A cinematic photograph of a long warehouse aisle of shelving where neatly stacked crates sit beside scattered unstacked crates, in teal and steel with a warm amber light, the AMARBARO mark and wordmark centred over it.

Wan 2.2 attention on AOTriton was slow because ComfyUI passed strided q, k and v views, and making them contiguous fixed it without a new kernel.

We were about to write our own attention kernel for video. Then we found that the slow part was not the math at all: the data was laid out badly in memory, and one small node that fixed the layout did the job.

Wan 2.2 is a video model. We use its image to video mode (i2v): give it a still, get a short clip. On our 7900 XTX a clip of 81 frames at 480 by 832 took about four minutes, and attention was 75 % of the time inside each step. This is the story of how we nearly built a kernel and then did not need to.

Where a clip's time went

We measured four clips of 81 frames with baked 4-step lightx2v experts (more on that below), launched with --use-pytorch-cross-attention --reserve-vram 2 --disable-pinned-memory.

Clip Wall Sampling Everything else
c1 (cold) 306 s 190 s 116 s
c2 239 s 190 s 49 s
c3 (fresh process) 253 s 190 s 63 s
c4 237 s 190 s 47 s

Sampling was 78 to 80 % of a warm clip. Two of four clips also fell back to tiled VAE encode after an out-of-memory error, and the process memory reached 46 GB before the kernel killed it at clip 3. The two Wan 14B experts are about 14 GB each and stay in system RAM for the high noise to low noise swap.

A wrong first answer

Wan has 40 blocks of width 5120. At 832 by 480 by 81 frames each step handles 32,760 tokens. We did the arithmetic: 9.2e14 FLOP for the matrix multiplies and 8.8e14 for attention, about 1.8e15 per step. At 48 seconds per step that is 37.5 TFLOPS. We wrote down a card peak of 61 TFLOPS and concluded that kernels could give at most 1.6 times.

That was wrong, and we said so in the note. The real fp16 peak is about 123 TFLOPS, and a profile showed attention was three quarters of step time, not half.

The profile

We ran torch.profiler over one low-noise step (a small custom node, step_profiler.py, switched on by an environment variable):

Kernel Device time Share Rate
attn_fwd (AOTriton flash attention), 80 calls 35.4 s 74.9 % about 25 TFLOPS
GEMM Cijk_..._HHS_BH_MT128x128x16, 406 calls 11.1 s 23.6 % about 83 TFLOPS
rope, norms, gelu, adds 0.8 s 1.6 %
dequantize_fp8 0.043 s 0.09 %

The last row killed one idea for free. We had planned a fused fp8 dequantize inside the matrix multiply, but dequantizing cost 0.09 % of the step. There was nothing to win.

The matrix multiplies ran at 67 % of peak. Attention ran at 20 %. That looked like the lever: a flash attention kernel for gfx1100 at head size 128 and 32,760 tokens, written in Mojo.

The cheapest falsifier

Before building anything, we benchmarked the same attention call on the same data two ways. ComfyUI passes data to the attention function in non-adjacent slices rather than one continuous memory block. We called .contiguous() first.

  • Strided input: 1701 ms per call.
  • The same data made contiguous: 278.5 ms per call.

So AOTriton was never slow. It was being fed a layout it handles badly. The fix was a custom node, contig_attn.py, that makes q, k and v contiguous before the call. In place, it costs 299.7 ms per call including the copies.

Result on a full clip:

  • Sampling: 226.7 s down to 122.6 s (1.85 times).
  • Clip total: 156 s.
  • Output against the stock run: Video quality was identical to the unmodified pipeline: the difference between outputs was smaller than the random variation between two stock runs.

Contiguous AOTriton runs at about 79 TFLOPS, or 63 % of peak. The "20 % of roof" number was the strided layout all along.

Kernels we did build, and what they proved

A custom kernel now had to beat 278.5 ms. Two attempts, both useful as negative results.

mojo-vattn. A Mojo attention kernel of the same design measured 294 ms against 279 ms for SDPA (torch's built-in scaled dot product attention). Our gate said 232 ms, so it failed. A tie is not a win.

Comfy Kitchen int8 attention. For the TRELLIS work (see the props article), the CK backend's int8_attn_kernel<128,64> took 9.4 ms per call against 14.0 ms for the other. We read its source (public, Apache-2.0, tag v0.2.40, built for gfx1100 only with setup.py --hip build_ext --hip-archs=gfx1100 bdist_wheel, about 3 minutes). Static facts: 204 VGPR (vector registers), 19 KB LDS (on-chip shared memory), no spills, 64 int8 WMMA (matrix multiply) instructions per tile, 2 barriers per tile.

Hardware profiling showed the bottleneck was on-chip memory access and instruction latency, not fetching data from VRAM (which was only 6% of stalls). Hypothesis one: prefetching K and V into registers with a double buffer would hide memory time. Hypothesis two: four accumulator chains instead of two would hide latency.

Median milliseconds, bit identical output on every arm:

Kernel 4096 tokens 19004 33153
shipped v0.2.40 2.20 32.76 97.06
K/V double buffer, 1 barrier per tile 2.12 32.28 97.30
plus scale loads first 2.15 32.56 97.57
four accumulator chains 2.21 33.09 99.52

The prefetch did work: memory stalls fell from 6 % to 0. It bought nothing, because the kernel is bound by LDS operand traffic and the latency of WMMA results. More chains cost registers (VGPR 204 to 222, so 7 to 6 waves per SIMD) for no gain. What is left is a different design, with K held in registers, which needs a register budget the head size 128 kernel does not have. We did not start it.

We also saw the AOTriton kernel attn_fwd warm: barrier 42 %, LDS queue 16 %, issue 15 %. Same story: an LDS-fed matrix pipeline, not a memory problem.

Step distillation: the real 4x

The biggest Wan win was not a kernel at all. The lightx2v LoRAs let Wan run in 4 steps at cfg 1 instead of 20 steps at cfg 3.5. Generating a 33-frame video clip dropped from 460 seconds at 20 steps down to 1 minute 50 seconds at 4 steps. We watched all three clips (baseline, int8 high-noise, 4-step) and judged all good, so 4-step became the default recipe.

We then baked the LoRA into both expert files with bake_lora.py, using ComfyUI's own patcher on the CPU. Naive fp8 requantizing loses the LoRA, since its delta is about 0.1 % of the weight norm, 20 to 70 times below rounding noise. Check: decoded frames from the baked files with no LoRA were bit identical to the LoRA at runtime (same rgb24 hash, infinite PSNR).

Other Wan lessons

  • Never run the low-noise model alone. Without the high-noise base, the generated video dissolved into blurred static. Pair the experts: KSamplerAdvanced for high noise with return_with_leftover_noise on, then low noise with add_noise off. Cuts fell from 8 to 0.
  • --reserve-vram 2.5 for long Wan batches. At the default 0.5, one clip in 11 ran out of memory inside VAEDecode. PYTORCH_HIP_ALLOC_CONF=expandable_segments:True made it worse. With 2.5, 16 of 16 clips finished.
  • Lower resolution is the other big lever: 640 by 368 with the same seed took 116 s total against 236 to 275 s at 832 by 480. Quality look was pending in that note.
  • Wan is not bit-exact across server restarts. The floor is 41 to 56 dB PSNR.

What we learned

A kernel is a hypothesis about where time goes. Before writing one, hand the existing kernel the data in the shape it likes and measure again. It took one .contiguous() call.

Commenti

Ancora nessun commento.

Accedi per commentare.