One-Shot Quality Claims Did Not Survive a Second Seed
Two quality claims came from one shot each. A second seed moves the metric by more than the gap between the step counts, so we retracted both and kept the test that can rank them.
Our final recipe reaches 0.4188 pooled panoptic quality in 5-fold validation and 0.36 on the public board.
Solar filaments are long, dark threads that lie across the bright disk of the Sun. In a photograph taken in one narrow band of red light, called H-alpha, they show up as dark ribbons against the glow. A Kaggle competition asked for a model that finds each filament and draws its outline. This article is the story so far: what the competition scores, what we built, what helped, what did not, and where our best validated number sits against the leaders.
The honest headline: our final model scores 0.4188 in 5-fold validation and 0.36 on the public board, rank 730 of 1,549. Validation sits about 0.06 above the board, and that gap, not the model, is the main lesson.
Each input is a 2048 by 2048 pixel grey image of the Sun from GONG, a network of six ground stations. The training set has 707 observations, each with one or more human annotations. The test set has 180 observations and no annotations. The output is a set of instance masks: one mask per filament, each stored as run-length counts.
Annotation is the hard part of the data. Of the 707 training observations, 411 have one annotator, 145 have two and 151 have three. Across them there are 8,199 filaments. The median filament covers 1,228 pixels, and filaments together cover 0.33 percent of every image. A model that predicts nothing is right 99.67 percent of the time, which is why the training method matters so much.
The metric is Panoptic Quality, or PQ, from a 2019 paper on panoptic segmentation. The host's version works like this. Every pair of one ground-truth mask and one predicted mask whose overlap, measured as intersection over union (IoU), is above 0.5 counts as a true positive (TP). A predicted mask with no partner is a false positive (FP). A truth with no partner is a false negative (FN). The score is:
PQ = (sum of IoU over the TPs) / (TP + FP/2 + FN/2)
In plain words, a good match earns its overlap. Each miss and each false alarm adds half a match to the denominator, so it lowers the score without adding anything to the top. Two things follow. The score rewards boundaries that overlap well, not just detection. And it is computed against every annotator's entry separately, so a model is scored against each human's version of the same image.
That second point is the main trap. Two humans disagree. When we scored one annotator's masks against the others on the same observation, the PQ was 0.33. So a model that learns the consensus will score better than any single annotator under this metric, and a model that copies one annotator's style may score worse. The host has not said whether the test set pools several annotators per image, as the training set does. If it has one annotator per image, the consensus target may lose points.
Resolution costs score too. A consensus mask built at 1024 pixels and enlarged to 2048 kept an IoU of 0.91 with the full-size truth on one sample. So a model that works at 1024 pixels starts about 9 percent of IoU down before any error of its own. Our measure of the best case, using the consensus target itself as the prediction, was PQ 0.697. That number is inflated by the 411 single-annotator images, where the target is one person's mask.
The host also said that compute efficiency is a filter, not a race. A slow pipeline or one that leaks memory can be cut. A few seconds faster does not win. So speed buys us more experiments, and inference needs a leak check rather than a sprint.
We cache each image resized to 1024 pixels. The training target is soft: for each pixel, the fraction of annotators who marked it. The model is a ConvNeXt-tiny encoder with weights pretrained on ImageNet-22k, followed by a U-Net style decoder with full-resolution skip connections, about 32 million parameters in all. Training uses binary cross-entropy plus a soft Dice loss, AdamW, a one-cycle learning-rate schedule, half-precision arithmetic, flips and 90-degree rotations, and per-image standardisation of intensities over the solar disk.
Instances come from a flip-averaged probability map, upsampled to 2048 pixels and cut at a threshold of 0.7. Connected blobs under 300 pixels are dropped, and fragments within 10 pixels are joined. Because the output is one label map, instances cannot overlap, which the host's scorer rejects.
Validation uses five folds by observation day, so no two frames from one day sit on both sides of a split. We score at 2048 pixels against every annotator entry of each held-out observation, as the host does. Our PQ port matches the host's maths to 1e-6 on a check case.
With the output bias at zero, the model starts at a probability of 0.5 everywhere. On a task where 0.33 percent of pixels are foreground, it spends about ten epochs predicting almost nothing before it drops sharply. Setting the output bias to the logit of 0.0033 removes that plateau. After eight epochs at 512 pixels, PQ was 0.347 with the bias and 0.165 without. A higher learning rate alone did not fix it, reaching 0.101.
The fix also cut the noise. Without it, two seeds of the same recipe differed by 0.026 in untuned PQ. With it, the same two seeds gave 0.4036 and 0.4042. A difference smaller than that is not a result, so we set our bar for each claim at the seed spread.
Every number below is tuned PQ, the best of several thresholds, on the half-length proxy schedule, unless the row says full. We used the proxy for A/B tests and the full schedule to confirm.
| change | PQ | against |
|---|---|---|
| 1 epoch at 1024 pixels | 0.3742 | base |
| 2 epochs at 1024 pixels | 0.3846 | +0.010 |
| 4 epochs at 1024 pixels | 0.3948 | +0.010 (seed 1: +0.0136) |
| 8 epochs at 1024 pixels | 0.4040 | +0.009 |
| 512-pixel crops at equal compute | 0.4020 | +0.0072 (seed 1: +0.0036) |
| crops plus double the epochs | 0.4091 | best half-proxy arm |
| full recipe, fold 0 | 0.4176 | best full run |
Training longer at the higher resolutions paid about 0.010 per doubling, and we have not seen it flatten. Cropping helped at equal compute because the model stopped spending its steps on the dark sky corners. In our frames, 23 percent is sky corner, and 512-pixel crops are 97 percent disk. The crops also cut peak video memory from 8.68 to 6.21 GB.
Most of what we tried tied or lost. The losses are as useful as the wins.
| arm | PQ | verdict |
|---|---|---|
| ensemble of 2 same-recipe models | 0.4142 | no gain |
| ensemble of 5 same-recipe models | 0.4183 | no gain: the error is bias, not variance |
| wider encoder (enc 1.0) | +0.0042 at 4 epochs | +0.0017 at 8 epochs: does not stack |
| larger encoder (convnext_small) | 0.3932 | tie |
| three input channels (raw, flattened, high-pass) | 0.3914 | lose |
| Adam with atan2 update | 0.3840 | tie |
| clDice loss | 0.3690 | lose |
| exponential moving average of weights | 0.3885 | lose |
| mean teacher | 0.3868 | lose |
| soft targets from an in-sample teacher | 0.3969 | tie |
| copy-paste dreams | 0.3907 | lose, small false positives 67 to 91 |
| loss at full 2048 resolution | 0.3945 | tie |
Two post-processing ideas also lost. A hysteresis rule that keeps a blob only if it holds a strong pixel did not beat the plain threshold. A learned grouper, which merges fragments by pair features, scored 0.3945 and 0.4136 on the two halves of the data, against 0.3997 and 0.4304 for the plain rule. The grouper had only 71 and 52 training pairs, so it may need more data before it can win.
The pattern across the losers is consistent. Every arm that showed the model more filaments, whether by crops centred on filaments or by dreams, raised the false positives. We now treat "show more" as a suspect idea by default.
A training step at 1024 pixels went from 152.5 milliseconds per image in bfloat16 to 35.0 milliseconds in half precision, with the Mojo fused kernels, a tuned matrix-multiply search and compilation on a warm cache. Loading the data at 2048 pixels went from 24 GB of RAM to 4.4 GB, by reading in chunks through memory mapping.
The fused kernels were first written in Mojo. Our own rule for the public code allows only PyTorch and Triton, so the public path runs a Triton port and the Mojo backend stays a local comparison. The port is a little slower: with the smaller encoder, compiled, it takes 29.2 milliseconds per image against 26.2 for Mojo.
The last night added two things. The first was test-time augmentation over all 8 rotations and flips of each image, instead of one flip. It added 0.0036 PQ on both seeds of fold 0, for 41 seconds per evaluation, so it was a free gain. The second was the full 5-fold run. Folds 1 to 4 trained on Kaggle's GPUs while fold 0 trained locally. Pooled over all 707 images, the 5-fold score is 0.4188, and a threshold of 0.7 was best on every fold.
The final submission averages six models: one trained on all the data, plus the five fold models, each with 8-view augmentation, cut at 0.7. Averaging two seeds beat the mean single seed by 0.003 on fold 0. The all-data model has no held-out images, so the ensemble itself cannot be scored on validation. We chose it on that evidence alone.
| submission | model | public PQ |
|---|---|---|
| 1 | U-Net with spine head, fold 0 model, flip average | 0.35 |
| 2 | a public detector plus refiner notebook, reproduced | 0.34 |
| 3, final | six-model ensemble, 8-view augmentation | 0.36 |
The public notebook in row 2 advertised 0.70. That number came from the competition's old Dice metric, before the switch to PQ on 7 August. The same file scored 0.34 under PQ. Check the date of the metric before you trust a public score.
At submission time our final model ranked 730 of 1,549 teams on the public board. Each step that raised validation moved the board by little or nothing: from 0.35 to 0.36 in total. The board shows only two decimals and scores a small part of the test set, so a change of 0.01 is close to its noise.
The gap between validation and the board is about 0.06, from 0.42 to 0.36. Two facts bound it. On the training set, PQ across annotators runs from 0.281 to 0.406, so the mix of annotators in the test set alone could move a score by about 0.12. And our threshold and blob filters were tuned on validation, which inflates validation a little. We have not tested which of these causes the gap.
The private board is published after the competition closes on 16 November at 11:59 UTC. We will add the private score here when it is out. The next experiment is to compare the test images with the training images (brightness, scale, filament count), to find the cause of the gap before we train anything else.
If your own run of the host's self-evaluation disagrees with the number above, the falsifier applies, and we want to hear it.
Komentarze
Brak komentarzy.
Zaloguj się aby skomentować.