Game-sized TRELLIS.2 settings, OPENBLAS_NUM_THREADS=1 and --use-ck-attention took one prop from 325 s to 84 s without a worse mesh.
Our first 3D apple took 325 seconds. The same apple now takes 84, with a mesh as good as the slower runs made.
TRELLIS.2 is a model that turns a single picture into a textured 3D mesh. We run it inside ComfyUI to make props for a game: an apple, a crate, a tea cup. The game wants an 8000 triangle mesh and a 1024 pixel texture. The ready-made workflow template wanted far more than that, and it took 325 seconds per prop.
Here is how the time came down, and what did not work.
A fair warning on the numbers
The machine was often busy with other jobs. One asset took 115 to 245 s under load against 94 to 102 s on an idle box. So wall times below are upper bounds, and a full job A/B on this box has about plus or minus 9 % noise. The stage shares hold. We say "one run" where it was one run.
The pipeline in one picture
image -> Pixal3D + TRELLIS.2 (int8) -> shape, structure, texture samplers
-> RemeshMesh -> DecimateMesh -> UnwrapMesh -> BakeTexture -> BakeAO -> SaveGLB
A sampler is the loop that refines the shape step by step. The mesh nodes after it run mostly on the CPU.
Cut 1: size the settings for the game
The template targeted film quality: 1536 upsampling, 4096 textures, 64 ambient occlusion samples, and 700,000 polygons. We were going to bake down to 8000 triangles anyway.
New defaults, all environment settings in our driver script:
ASSET_UPSAMPLE=1024 ASSET_TEXRES=1024 ASSET_AO_SAMPLES=16
ASSET_SIGN=udf ASSET_REMESH=512 ASSET_DECIMATE=150000
ASSET_STEPS=0.75
At the upsample stage, the apple went from 43,020 tokens at 1536 to 19,004 at 1024, and the step time from 20.3 s to 4.1 s. Attention cost grows with the square of the token count, so this one setting was most of the gain.
Result: the apple went from 325 s to 128 s on a cold server (model loads included). A batch of 10 food props averaged 142 s against 217 s for earlier ones, and all 15 meshes passed our checker. We compared the base mesh with the fast one: the shape difference was less than 0.4%, which matches the variation between two runs using the exact same settings. Side by side the silhouette and detail matched, with the fast one a shade duller in tone from the lower AO count.
Earlier, on one prop, a fast arm with ASSET_STEPS=0.5 finished in 61 and 69 s but lost 42 to 59 % of texture sharpness. Dropping the steps to 0.75 while shrinking the mesh settings gave 72 to 77 s with sharpness inside the normal spread. The newest note, from 2026-11-05, says steps between 0.5 and 0.75 were noise for time on four props. So the older quality claim rests on one prop only.
Cut 2: a math library that spins
Under load, the GPU was only 5 % busy during mesh post-processing while the server burned 366 % CPU. We pointed corewait, our CPU sampler, at the live server (this is safe on a running server: it only samples).
The answer was a surprise. 96.5 % of the server's CPU sat in the OpenBLAS library bundled with numpy. 47% of CPU time was threads burning cycles in spin locks, and 46% was wasted on tiny matrix calls. Actual math took only 0.2%.
What happens: UnwrapMesh solves many tiny linear systems with np.linalg.solve. Each goes to a 16-thread pool that spins while it waits for the next tiny job. A ranked table that puts a library's thread-pool symbols above its compute symbols is the signature of an over-threaded pool.
Our first guess a day earlier had been wrong: we blamed "BLAS contention on the main thread" with no evidence. The profile said what it really was. The fix is one line: OPENBLAS_NUM_THREADS=1. Note that OMP_NUM_THREADS does not reach this copy, because numpy's bundled OpenBLAS is built on plain threads.
Result: the apple at the template's 1536 setting went from 320 s to 96 s. Both passed the checker, one run each, fresh server per arm. UnwrapMesh fell from 30 s of lock wait to 0.8 s. After the fix, corewait showed libhsa-runtime at 63.8 % (the CPU waiting politely for the GPU, which is normal) and scipy's _ckdtree at 26.9 %.
Cut 3: a different attention backend
Attention is the part of a transformer where every token looks at every other. On this job it was 47 % of GPU kernel time. ComfyUI has several backends, and we ran each on the same apple, one run each:
| Backend flag |
Wall |
Check |
--use-pytorch-cross-attention (AOTriton) |
98.5 s |
pass |
--use-ck-attention (Comfy Kitchen) |
84.1 s |
pass |
--use-quad-cross-attention |
147.0 s |
pass |
--use-split-cross-attention |
error at the upsample sampler after 33 s |
no mesh |
A repeat gave 104.1 s for pytorch and 91.5 s for CK. CK's int8 attention kernel took 9.4 ms per call against 14.0 ms for the AOTriton one. The CK mesh stayed within 0.24 % surface distance of the AOTriton mesh (floor 0.23 %). --use-ck-attention is now the default. We lost the reason for the split failure: our A/B script cut error messages at 300 characters and the next arm overwrote the server log. Keep one log per arm.
The summary, with caveats
| Step |
Apple |
| Template |
325 s |
| Game-sized settings |
128 s |
OPENBLAS_NUM_THREADS=1 |
96 to 98 s |
--use-ck-attention |
84 and 92 s (two runs) |
These runs were taken at different times and box loads, so do not multiply the ratios.
Where the time goes now
For the 98.5 s run before CK: the shape upsample sampler 34.8 s, the texture sampler 17.8, the shape sampler 9.3, structure 7.4, RemeshMesh 6.5, BakeAO 4.1, Decimate 3.1, BakeTexture 2.9, UnwrapMesh 0.8.
One thing for trees
A tree took 193 s per step and never finished, because it produced too many tokens. We capped MAX_HR_TOKENS at 49152 in Trellis2UpsampleStage. The tree then ran at 8.45 s per step and finished in 364 s with 43,500 tokens and 7997 faces. The limit came from memory of the official code and was not checked by eye.
What did not help
--highvram: about 1.4 s per asset, inside the noise.
- MIOpen: 87.9 s with no gain and 334 s of tuning per conv shape.
- Overlapping the GPU sampler of prop N+1 with the CPU work of prop N: 599 s sequential against 608 s overlapped. GPU idle time fell, wall time did not.
- A HIP port of the mesh ray tracing for AO: matched the CPU result and gave no win.
- PNG level 1 inside SaveGLB: 4.19 to 1.2 s, with 18 % larger files. One run, no side by side.
- Memory is not a lever: heaptide found 0 B leaked, with a 4.4 GB peak GPU block.
Checking a prop
We never say faster until asset_check.py passes (faces, bytes, texture present) and glb_compare.py reports distance and sharpness against the same-settings floor. Then we open a side by side render, because a sharpness of 6.59 against 7.43 is a number, not a look.
Reacties
Nog geen reacties.
Inloggen om te reageren.