ComfyUI reports success for jobs that did the wrong thing, so every job needs a cheap guard that checks the output itself.
One of our jobs returned "success" and handed us pure noise. ComfyUI says success for many jobs that did not do what you meant. Here are eight we hit on our 7900 XTX, with the guard each one earned. At the end is a small experiment, training a style LoRA inside ComfyUI, that shows why we look at every output.
1. The fastest arm was pure noise
We benchmarked Flux.1-dev: the flux1-dev-fp8 all-in-one checkpoint (17.2 GB), 1024 by 1024, euler with the simple scheduler, cfg 1.0, FluxGuidance 3.5.
| Arm |
Steady time |
| 24 steps |
34 s |
| 20 steps |
28 s |
| 8 steps with Hyper-FLUX LoRA at 0.125 |
14 s |
| 8 steps with Hyper-FLUX LoRA at 1.0 |
12 s |
The 12 s arm looked like the winner. Every image from it was RGB static. The job exited 0, wrote a file and logged no warning. The Hyper-FLUX 8-step LoRA wants strength 0.125, not 1.0. We caught it by looking, one step after writing the number in a table.
Guard: a cheap statistic. Pixel variance on pure noise was five times higher than on any real render, giving an easy automated check. Run it on any unattended render batch.
2. The cache that times itself
ComfyUI hashes each node's inputs and reuses an output when nothing upstream changed. A lightning-fast 2-second repeat run was not speed: ComfyUI simply returned cached images because the runs reused the same random seed. On one server, two A/B arms that differ only after the sampler also share the sampler's run. A lanczos arm at 4 s against an 8 s arm in our texture test was cache, not speed.
Guard: a different seed per repetition, or one server per arm.
3. The timer polled every 2 seconds
The full story is in A Z-Image picture in 1.7 seconds: our client waited 2 s between checks, which added up to 2 s to every timing. Guard: poll at a small fraction of what you measure, or report the server's own number.
4. One bad node, the rest still runs
For a style LoRA prototype we used a LoadImage pointing at symlinked tiles and typed algorithm: "lora" where the node wants "LoRA". When a node has an invalid parameter, ComfyUI drops that whole section of the graph and executes whatever is left. Our client saw status success, copied the plain renders and printed DONE. The first run reported 18 s for a training that never happened.
Guards: the client asserts that every expected output file exists and fails otherwise, and we read server.log for Failed to validate prompt after any new graph. Symlinks inside input/ are refused, so copy files. Case matters in combo values.
5. A node pack that vanished
After we added a new node to our custom pack, the BaroSeamless node disappeared from /object_info and the runs failed. The pack exposed both a v3 entry point (comfy_entrypoint) and a legacy module-level NODE_CLASS_MAPPINGS. That loads with no error, and ComfyUI drops the whole v3 node list.
Guard: register everything through get_node_list, and grep -c BaroSeamless server.log after boot, or fetch /object_info and assert the id, before trusting the job.
6. A rejected graph shipped the last run's mesh
In our TRELLIS driver script, a graph the server rejected let the previous run's GLB file be picked up as the new result. Fixed in our driver. The common pattern with number 4: a client that asks "is there an output file" instead of "did this run make it".
7. Attaching a profiler to a live server
We proposed attaching heaptide, a memory profiler, to a ComfyUI server holding 56 GB during a Wan batch. gdb loaded the library into the torch process, which died 83 seconds later with a segmentation fault. Six of 10 queued clips were lost. heaptide then printed a message about ptrace settings that hid the fact that the attach had happened.
Rule: never attach an interactive memory profiler to an active GPU server. Profile from launch on a dedicated test run. The 56 GB turned out to be the two Wan 14B experts kept in RAM, visible in the log: "Unloaded partially: 13410 MB". We also misread a heaptide report once: we read only the first 30 lines, which list children, and blamed the launcher, while the root Python process held the data. A traced server needs SIGTERM, since kill -INT on a backgrounded job is ignored.
8. Judging at the wrong size
A pack of 15 redone TRELLIS renders went through our review and 5 of them passed that the user then rejected: warped crate walls, a torn stack, a blur patch, a chrome keg. We judged a contact sheet. Rule: judge each 320 pixel view on its own. A tear, shard, blur or wrinkle is a fail. Prompts need care too: bare metal draws chrome, black or open-top containers break, and a "solid block" drew a second object.
The LoRA that taught us to look
ComfyUI v0.43.0 ships a trainer: TrainLoraNode, SaveLoRA, LoraModelLoader and dataset nodes including LoadImageTextDataSetFromFolder, ResizeImagesByShorterEdge, RandomCropImages and MakeTrainingDataset. There is no separate environment to build. We trained a style LoRA from 34 of our own albedo tiles, then rendered plain against LoRA on the same seed in one graph.
Settings: RealVisXL, rank 8, bf16, gradient checkpointing, 512 pixel crops. It ran 0.67 s per step (1.48 it/s), so 60 steps took 40 s and 400 steps 4.5 min.
| Run |
Data |
Steps |
LR |
Result |
| proto |
16 tiles, one shared caption |
60 |
5e-4 |
collapse: uniform maroon grain on both prompts |
| proto2 |
34 tiles, caption per tile, tile downscaled to 512 |
90 |
1e-4 |
near no-op |
| proto3 |
same |
400 |
2e-4 |
uniform fine grain, cracks gone |
| proto4 |
shorter edge 1024, then random 512 crops |
400 |
2e-4 |
stones at the right scale, plaster rough brown, colour drifts |
Rules from the failures: a caption per sample (a shared caption trains the mean image), and data at the render scale (crops, never a downscaled tile). Loss moved 0.13 to 0.55 on the collapsed run and 0.22 to 0.17 on the working one. The number does not say which is which. The sheet does.
Then we tried colour captions and a strength sweep of 0.4, 0.7 and 1.0 (40 tiles, 418.8 s). It was not kept. Plaster grew cracks with strength until it looked like dry earth at 1.0, and wet asphalt came out as coarse gravel at every strength. The LoRA had learned the crack prior from the set, the very thing we built it to remove. The texture check on the 0.7 tiles also failed the seam (5.76 and 5.33), since that arm renders a plain sampler, not the seamless graph. Whether routing it through the seamless graph helps is untested, and not worth it while the LoRA itself adds cracks.
The one habit
Exit code zero and a file on disk are not a result. Every render arm gets looked at, every expected output gets asserted, and every timing is checked against the server's own clock.
Comentarios
Aún no hay comentarios.
Iniciar sesión para comentar.