← Blog
✎blog post

One-Shot Quality Claims Did Not Survive a Second Seed

A steel vernier caliper on a grey slate surface with one jaw open around a small brass gear, cold teal light with a thin amber rim on the scale, empty space above the caliper.

Two quality claims about LTX clips, four steps costing 0.03 alignment and 12 frames per second dropping it to 0.014, came from single shots and did not survive: one seed per shot moves the metric by 0.04 or more.

We wrote that four diffusion steps cost about 0.03 alignment on a video clip, and that 12 frames per second dropped alignment to 0.014. Both numbers came from a single shot each. A second seed moved the same metric by 0.04 or more, which is larger than the gap between the step counts we were trying to rank. We retracted both claims.

Alignment here means one number: how closely a clip's frames match a short written caption, scored by a vision-language model. It is a proxy. It is not a judgment of whether a clip looks right.

The three shots

Our test used three shots, each with its own prompt and seed, and every arm used the same prompts and seeds. The table reads per shot, then the mean of the three.

Arm Alignment, shots 1 / 2 / 3 Mean
8 steps 0.121 / 0.105 / 0.031 0.086
6 steps 0.116 / 0.089 / 0.025 0.076
4 steps 0.093 / 0.119 / 0.064 0.092

Read shot 1 alone and four steps look 0.028 worse than eight, which is the 0.03 we first wrote. Read the three-shot mean and four steps score higher than eight. The shot-one reading and the mean disagree, and neither is strong enough to rank the arms.

The 12 frames per second claim has the same shape. The shot-two reading was 0.014, and the same arm's other shots scored 0.131 and 0.057 in the same test. The low reading was one shot, not the arm.

The seed swing

Within one shot, changing the seed moves alignment by 0.04 or more. The gap between the 8-, 6- and 4-step means is 0.016 at most, well under that swing. So three shots cannot rank the step counts, and a single shot cannot say anything about one arm against another.

That is why we retracted both claims rather than qualifying them. The claim was a ranking, and the test could not rank.

What we kept

Two checks in the test held up. The flicker metric, which measures brightness changes between frames, was calibrated on known cases: exposure flicker scored 13.5 and smooth motion scored 0.06, so it separates the two. The second check dropped a second metric that could not separate smooth motion from noise, scoring 1.57 for smooth and 1.73 for noise.

The runs were also scored one scorer at a time, after an earlier pass had run two scorers on one directory at once. The clean re-score is the table above.

What we did not judge

We have not looked at the contact sheets for these clips. The alignment numbers say nothing about whether a clip is sharp, whether the subject holds, or whether the motion is what the prompt asked for. Nothing in this post makes a visual quality claim.

Clips at different frame counts are also different clips, so alignment across 33, 65, 97 and 121 frames is not a like-for-like comparison.

What a ranking needs

Nine shots per arm, the same seeds in every arm, one scorer, and then a person looking at the clips. The table would need nine columns, not three, and the seed-swing reading would need to be taken from repeats of the same shot, so the noise floor is measured and not assumed.

How to prove this wrong

Pick one shot and run it with two seeds in each arm. Score both runs with the same scorer. If no shot's two readings differ by 0.04 or more, the seed-swing claim is wrong, and the retraction was too cautious. If they do differ by that much, the retraction stands for three shots.

The open question is whether alignment tracks what a viewer sees at all. We have not tested that, and the retraction does not depend on the answer.

Provenance

  • /home/mario/Projects/AssetsHQ/docs/lanes/video-fast-report.md, lines 85 to 98 (the clean re-score table, the retraction, the seed-swing sentence, the frame-count caveat, and the calibration line at 56 to 57).
  • /home/mario/Brain/AssetsHQ/2026-11-17-video-fast.md, lines 27 to 28 (the retracted claims as stated in the written record).

Comments

No comments yet.

Log in to comment.