Seamless textures that really tile, and the crack prior that fought us
Circular padding makes a texture tile but changes what SDXL draws. Our default is now a plain render, a half-roll and a seam-band inpaint, with a checker that fails real seams.
Wan 2.2 attention on AOTriton was slow because ComfyUI passed strided q, k and v views, and making them contiguous fixed it without a new kernel.
We were about to write our own attention kernel for video. Then we found that the slow part was not the math at all: the data was laid out badly in memory, and one small node that fixed the layout did the job.
Wan 2.2 is a video model. We use its image to video mode (i2v): give it a still, get a short clip. On our 7900 XTX a clip of 81 frames at 480 by 832 took about four minutes, and attention was 75 % of the time inside each step. This is the story of how we nearly built a kernel and then did not need to.
We measured four clips of 81 frames with baked 4-step lightx2v experts (more on that below), launched with --use-pytorch-cross-attention --reserve-vram 2 --disable-pinned-memory.
| Clip | Wall | Sampling | Everything else |
|---|---|---|---|
| c1 (cold) | 306 s | 190 s | 116 s |
| c2 | 239 s | 190 s | 49 s |
| c3 (fresh process) | 253 s | 190 s | 63 s |
| c4 | 237 s | 190 s | 47 s |
Sampling was 78 to 80 % of a warm clip. Two of four clips also fell back to tiled VAE encode after an out-of-memory error, and the process memory reached 46 GB before the kernel killed it at clip 3. The two Wan 14B experts are about 14 GB each and stay in system RAM for the high noise to low noise swap.
Wan has 40 blocks of width 5120. At 832 by 480 by 81 frames each step handles 32,760 tokens. We did the arithmetic: 9.2e14 FLOP for the matrix multiplies and 8.8e14 for attention, about 1.8e15 per step. At 48 seconds per step that is 37.5 TFLOPS. We wrote down a card peak of 61 TFLOPS and concluded that kernels could give at most 1.6 times.
That was wrong, and we said so in the note. The real fp16 peak is about 123 TFLOPS, and a profile showed attention was three quarters of step time, not half.
We ran torch.profiler over one low-noise step (a small custom node, step_profiler.py, switched on by an environment variable):
| Kernel | Device time | Share | Rate |
|---|---|---|---|
attn_fwd (AOTriton flash attention), 80 calls |
35.4 s | 74.9 % | about 25 TFLOPS |
GEMM Cijk_..._HHS_BH_MT128x128x16, 406 calls |
11.1 s | 23.6 % | about 83 TFLOPS |
| rope, norms, gelu, adds | 0.8 s | 1.6 % | |
dequantize_fp8 |
0.043 s | 0.09 % |
The last row killed one idea for free. We had planned a fused fp8 dequantize inside the matrix multiply, but dequantizing cost 0.09 % of the step. There was nothing to win.
The matrix multiplies ran at 67 % of peak. Attention ran at 20 %. That looked like the lever: a flash attention kernel for gfx1100 at head size 128 and 32,760 tokens, written in Mojo.
Before building anything, we benchmarked the same attention call on the same data two ways. ComfyUI passes data to the attention function in non-adjacent slices rather than one continuous memory block. We called .contiguous() first.
So AOTriton was never slow. It was being fed a layout it handles badly. The fix was a custom node, contig_attn.py, that makes q, k and v contiguous before the call. In place, it costs 299.7 ms per call including the copies.
Result on a full clip:
Contiguous AOTriton runs at about 79 TFLOPS, or 63 % of peak. The "20 % of roof" number was the strided layout all along.
A custom kernel now had to beat 278.5 ms. Two attempts, both useful as negative results.
mojo-vattn. A Mojo attention kernel of the same design measured 294 ms against 279 ms for SDPA (torch's built-in scaled dot product attention). Our gate said 232 ms, so it failed. A tie is not a win.
Comfy Kitchen int8 attention. For the TRELLIS work (see the props article), the CK backend's int8_attn_kernel<128,64> took 9.4 ms per call against 14.0 ms for the other. We read its source (public, Apache-2.0, tag v0.2.40, built for gfx1100 only with setup.py --hip build_ext --hip-archs=gfx1100 bdist_wheel, about 3 minutes). Static facts: 204 VGPR (vector registers), 19 KB LDS (on-chip shared memory), no spills, 64 int8 WMMA (matrix multiply) instructions per tile, 2 barriers per tile.
Hardware profiling showed the bottleneck was on-chip memory access and instruction latency, not fetching data from VRAM (which was only 6% of stalls). Hypothesis one: prefetching K and V into registers with a double buffer would hide memory time. Hypothesis two: four accumulator chains instead of two would hide latency.
Median milliseconds, bit identical output on every arm:
| Kernel | 4096 tokens | 19004 | 33153 |
|---|---|---|---|
| shipped v0.2.40 | 2.20 | 32.76 | 97.06 |
| K/V double buffer, 1 barrier per tile | 2.12 | 32.28 | 97.30 |
| plus scale loads first | 2.15 | 32.56 | 97.57 |
| four accumulator chains | 2.21 | 33.09 | 99.52 |
The prefetch did work: memory stalls fell from 6 % to 0. It bought nothing, because the kernel is bound by LDS operand traffic and the latency of WMMA results. More chains cost registers (VGPR 204 to 222, so 7 to 6 waves per SIMD) for no gain. What is left is a different design, with K held in registers, which needs a register budget the head size 128 kernel does not have. We did not start it.
We also saw the AOTriton kernel attn_fwd warm: barrier 42 %, LDS queue 16 %, issue 15 %. Same story: an LDS-fed matrix pipeline, not a memory problem.
The biggest Wan win was not a kernel at all. The lightx2v LoRAs let Wan run in 4 steps at cfg 1 instead of 20 steps at cfg 3.5. Generating a 33-frame video clip dropped from 460 seconds at 20 steps down to 1 minute 50 seconds at 4 steps. We watched all three clips (baseline, int8 high-noise, 4-step) and judged all good, so 4-step became the default recipe.
We then baked the LoRA into both expert files with bake_lora.py, using ComfyUI's own patcher on the CPU. Naive fp8 requantizing loses the LoRA, since its delta is about 0.1 % of the weight norm, 20 to 70 times below rounding noise. Check: decoded frames from the baked files with no LoRA were bit identical to the LoRA at runtime (same rgb24 hash, infinite PSNR).
KSamplerAdvanced for high noise with return_with_leftover_noise on, then low noise with add_noise off. Cuts fell from 8 to 0.--reserve-vram 2.5 for long Wan batches. At the default 0.5, one clip in 11 ran out of memory inside VAEDecode. PYTORCH_HIP_ALLOC_CONF=expandable_segments:True made it worse. With 2.5, 16 of 16 clips finished.A kernel is a hypothesis about where time goes. Before writing one, hand the existing kernel the data in the shape it likes and measure again. It took one .contiguous() call.
Comments
No comments yet.
Log in to comment.