← Blog
✎post de blog

Eight ways a ComfyUI job succeeds and is wrong

A cinematic photograph of a factory conveyor belt carrying identical sealed metal boxes, one box slightly dented, in teal and steel with a warm amber light, the AMARBARO mark and wordmark centred over it.

ComfyUI reports success for jobs that did the wrong thing, so every job needs a cheap guard that checks the output itself.

One of our jobs returned "success" and handed us pure noise. ComfyUI says success for many jobs that did not do what you meant. Here are eight we hit on our 7900 XTX, with the guard each one earned. At the end is a small experiment, training a style LoRA inside ComfyUI, that shows why we look at every output.

1. The fastest arm was pure noise

We benchmarked Flux.1-dev: the flux1-dev-fp8 all-in-one checkpoint (17.2 GB), 1024 by 1024, euler with the simple scheduler, cfg 1.0, FluxGuidance 3.5.

Arm Steady time
24 steps 34 s
20 steps 28 s
8 steps with Hyper-FLUX LoRA at 0.125 14 s
8 steps with Hyper-FLUX LoRA at 1.0 12 s

The 12 s arm looked like the winner. Every image from it was RGB static. The job exited 0, wrote a file and logged no warning. The Hyper-FLUX 8-step LoRA wants strength 0.125, not 1.0. We caught it by looking, one step after writing the number in a table.

Guard: a cheap statistic. Pixel variance on pure noise was five times higher than on any real render, giving an easy automated check. Run it on any unattended render batch.

2. The cache that times itself

ComfyUI hashes each node's inputs and reuses an output when nothing upstream changed. A lightning-fast 2-second repeat run was not speed: ComfyUI simply returned cached images because the runs reused the same random seed. On one server, two A/B arms that differ only after the sampler also share the sampler's run. A lanczos arm at 4 s against an 8 s arm in our texture test was cache, not speed.

Guard: a different seed per repetition, or one server per arm.

3. The timer polled every 2 seconds

The full story is in A Z-Image picture in 1.7 seconds: our client waited 2 s between checks, which added up to 2 s to every timing. Guard: poll at a small fraction of what you measure, or report the server's own number.

4. One bad node, the rest still runs

For a style LoRA prototype we used a LoadImage pointing at symlinked tiles and typed algorithm: "lora" where the node wants "LoRA". When a node has an invalid parameter, ComfyUI drops that whole section of the graph and executes whatever is left. Our client saw status success, copied the plain renders and printed DONE. The first run reported 18 s for a training that never happened.

Guards: the client asserts that every expected output file exists and fails otherwise, and we read server.log for Failed to validate prompt after any new graph. Symlinks inside input/ are refused, so copy files. Case matters in combo values.

5. A node pack that vanished

After we added a new node to our custom pack, the BaroSeamless node disappeared from /object_info and the runs failed. The pack exposed both a v3 entry point (comfy_entrypoint) and a legacy module-level NODE_CLASS_MAPPINGS. That loads with no error, and ComfyUI drops the whole v3 node list.

Guard: register everything through get_node_list, and grep -c BaroSeamless server.log after boot, or fetch /object_info and assert the id, before trusting the job.

6. A rejected graph shipped the last run's mesh

In our TRELLIS driver script, a graph the server rejected let the previous run's GLB file be picked up as the new result. Fixed in our driver. The common pattern with number 4: a client that asks "is there an output file" instead of "did this run make it".

7. Attaching a profiler to a live server

We proposed attaching heaptide, a memory profiler, to a ComfyUI server holding 56 GB during a Wan batch. gdb loaded the library into the torch process, which died 83 seconds later with a segmentation fault. Six of 10 queued clips were lost. heaptide then printed a message about ptrace settings that hid the fact that the attach had happened.

Rule: never attach an interactive memory profiler to an active GPU server. Profile from launch on a dedicated test run. The 56 GB turned out to be the two Wan 14B experts kept in RAM, visible in the log: "Unloaded partially: 13410 MB". We also misread a heaptide report once: we read only the first 30 lines, which list children, and blamed the launcher, while the root Python process held the data. A traced server needs SIGTERM, since kill -INT on a backgrounded job is ignored.

8. Judging at the wrong size

A pack of 15 redone TRELLIS renders went through our review and 5 of them passed that the user then rejected: warped crate walls, a torn stack, a blur patch, a chrome keg. We judged a contact sheet. Rule: judge each 320 pixel view on its own. A tear, shard, blur or wrinkle is a fail. Prompts need care too: bare metal draws chrome, black or open-top containers break, and a "solid block" drew a second object.

The LoRA that taught us to look

ComfyUI v0.43.0 ships a trainer: TrainLoraNode, SaveLoRA, LoraModelLoader and dataset nodes including LoadImageTextDataSetFromFolder, ResizeImagesByShorterEdge, RandomCropImages and MakeTrainingDataset. There is no separate environment to build. We trained a style LoRA from 34 of our own albedo tiles, then rendered plain against LoRA on the same seed in one graph.

Settings: RealVisXL, rank 8, bf16, gradient checkpointing, 512 pixel crops. It ran 0.67 s per step (1.48 it/s), so 60 steps took 40 s and 400 steps 4.5 min.

Run Data Steps LR Result
proto 16 tiles, one shared caption 60 5e-4 collapse: uniform maroon grain on both prompts
proto2 34 tiles, caption per tile, tile downscaled to 512 90 1e-4 near no-op
proto3 same 400 2e-4 uniform fine grain, cracks gone
proto4 shorter edge 1024, then random 512 crops 400 2e-4 stones at the right scale, plaster rough brown, colour drifts

Rules from the failures: a caption per sample (a shared caption trains the mean image), and data at the render scale (crops, never a downscaled tile). Loss moved 0.13 to 0.55 on the collapsed run and 0.22 to 0.17 on the working one. The number does not say which is which. The sheet does.

Then we tried colour captions and a strength sweep of 0.4, 0.7 and 1.0 (40 tiles, 418.8 s). It was not kept. Plaster grew cracks with strength until it looked like dry earth at 1.0, and wet asphalt came out as coarse gravel at every strength. The LoRA had learned the crack prior from the set, the very thing we built it to remove. The texture check on the 0.7 tiles also failed the seam (5.76 and 5.33), since that arm renders a plain sampler, not the seamless graph. Whether routing it through the seamless graph helps is untested, and not worth it while the LoRA itself adds cracks.

The one habit

Exit code zero and a file on disk are not a result. Every render arm gets looked at, every expected output gets asserted, and every timing is checked against the server's own clock.

Comentarii

Niciun comentariu încă.

Conectare pentru a comenta.