blog›contrails
Segment Anything, stripped down for contrails
On this page
Contrails are the thin white lines aircraft leave behind at cruising altitude. Some linger for hours and spread into cirrus cloud, trapping heat. Google Research turned finding them into a Kaggle competition: take a short sequence of GOES-16 infrared satellite frames and mask every contrail pixel.
I teamed up with Iafoss and Théo Viel, and we finished 2nd out of 954 teams. Each of us built a completely different model. Meta had just released Segment Anything (SAM) a few weeks earlier, so I bet on it, quickly ran into why SAM as released couldn't touch two-pixel satellite lines, and ended up stripping away everything except its image encoder.
Six real records from the training set. Each one is eight frames taken ten minutes apart, drawn here in the same false colours the labellers saw, in which contrails show up as dark lines. The movie stops on the fifth frame and shows its label in white. Hover or tap a small frame to hold it, or hover or tap the large image to see the label. One pixel is about 2 km.
The problem
Jet engines dump water vapour and soot into air colder than −40 °C, where the vapour freezes onto the soot into a thin line of ice crystals. In dry air it vanishes within seconds. In air already saturated with respect to ice, the trail can hang around for hours, drift, and spread from a hundred metres wide to several kilometres. Those lingering contrails are thin cirrus clouds that let sunlight through while trapping heat leaving the ground.
One widely cited assessment puts contrail cirrus warming in 2018 at 57 mW/m², against 34 mW/m² for all the CO₂ aviation has ever emitted [1]. Most of it comes from a tiny slice of flights: a study of Japanese airspace estimated that nudging just 1.7% of flights by about 2,000 feet would cut contrail warming by 59% [2]. Testing flight diversions like that in practice requires checking satellite imagery afterwards to verify whether a contrail actually formed.
The dataset comes from GOES-16, a geostationary weather satellite over the Americas. Each sample is a 256 × 256 pixel crop of eight frames, with only the fifth frame labelled and the surrounding frames (four before, three after) providing motion context.
On disk, one record is a folder of arrays:
| file | what it is |
|---|---|
band_08 … band_16 | Nine infrared bands. Each is a stack of eight frames: four before the labelled one, the labelled one, three after. |
human_ | The mask each labeller drew for the fifth frame. Four or more people per record. Training set only. |
human_ | The final label: a pixel is contrail if more than half of the labellers marked it. |
One record. There are 20,529 in the training set and 1,856 in the validation set.
Human labellers didn't inspect all nine infrared bands; they stepped through an ash colour composite built from three of them, watching contrails form and drift across frames. Training on all nine bands actually scored worse, likely because the ground-truth masks only capture what was visible in those ash images.
Submissions were scored by the global Dice coefficient pooled over every pixel of the hidden test set at once (about 1,900 images), and the whole ensemble had to finish inside a nine-hour Kaggle notebook.
Two difficulties
First, the labellers frequently disagree. Switch the hero figure above to "Show each labeller" and the noise jumps right out: in the first record alone, 1,976 pixels were marked by all four people while 7,520 were marked by just one. Because the ground truth is a majority vote across drawings that messy, no model can guess which side of 50% a borderline pixel landed on. Training on the continuous average of the individual masks helped a lot, even though the final score still rounds everything to yes or no.
Second, contrails are razor-thin, typically one to three pixels wide. To see how unforgiving that is, I took the ground-truth masks of the six records above, nudged them one pixel to the right, and scored them against themselves:
A mask that finds every single contrail, shifted by one pixel, drops to 0.77 Dice, when the winning team in this competition scored 0.724. Almost everything separating a medal model from an average one comes down to sub-pixel edge placement on a two-pixel line. Standard segmentation decoders that predict at quarter resolution and bilinearly upsample at the end smear those edges away, so every model we built was designed to avoid that step.
Five steps with Segment Anything
When the competition opened, Meta had only just released Segment Anything: trained on 1.1 billion masks across 11 million images, able to carve out objects zero-shot. My teammates leaned on convolutional nets and hierarchical transformers. I figured an encoder that had seen a billion crisp object outlines might shine on hair-thin boundaries, so I started there.
Each step below includes a diagram highlighting what changed in colour (hover or tap any block for details).
Step 1: SAM as released
Out of the box, SAM isn't an automatic segmenter. Its heavy image encoder turns an image into a 64 × 64 feature grid once, and a lightweight decoder takes a prompt (say, a click on an object) and outputs a mask for whatever sits under that click.
With no human clicking through 1,900 test images, you have to fall back on SAM's automatic grid mode: lay down 32 × 32 prompt points, run the decoder at every point, and collect the masks. That works nicely for cars and coffee mugs. On two-pixel contrails, it misses constantly:
Across 20 training records I checked, the true masks contain 200 separate contrail pieces. The default 32 × 32 grid drops a point onto just 120 of them; the other 80 slip neatly between the dots, so SAM never even gets asked about them. Even quadrupling to 64 × 64 points still misses 39 pieces (161 of 200). The only grid guaranteed not to skip a two-pixel line puts a prompt on every single pixel: 65,536 decoder passes with three candidate masks each, per image, across about 1,900 test images, all sharing a nine-hour notebook with the rest of our ensemble.
Even with infinite compute, prompting SAM this way fails for three deeper reasons:
- No classes. Clicking a contrail gives a mask, and so does clicking a cirrus cloud, ocean or coastline. Sorting which of those thousands of masks are actually contrails puts us right back where we started.
- Coarse masks. The mask decoder predicts at 64 × 64 and upsamples 4×, fine for a cat, fatal for a line two pixels wide.
- Wrong domain. SAM saw natural photographs, nothing like false-colour infrared satellite frames. Fine-tuning through prompts means synthesising fake clicks during training only to go back to a brute-force grid at test time.
So I threw out the prompt encoder and mask decoder altogether, kept only SAM's pretrained image encoder, and treated it like an ordinary segmentation backbone.
Step 2: the encoder and a simple decoder
The first working version was as straightforward as possible: take the encoder's 64 × 64 feature map and apply two learned upsampling convolution stages (64 → 128 → 256) down to a single logit per pixel. We trained with the team's shared loss, binary cross-entropy plus a Lovász term.
Because SAM's position embeddings expect a 1024 × 1024 input while our crops are 256 × 256, I enlarged every image 4× with bicubic interpolation. SAM's ViT cuts its input into 16 × 16 patches, so after that 4× enlargement each token spans 4 × 4 original pixels instead of 16 × 16, keeping contrails from disappearing inside a single patch.
Step 3: four scales and a U-Net
Drawing everything from one 64 × 64 map ignores what makes a U-Net sharp: skip connections back to finer, earlier features. Convolutional backbones shrink resolution in stages, giving you those skip maps for free. A plain ViT keeps the same 64 × 64 grid all the way through all twelve blocks.
So I built a four-scale pyramid by hand. SAM has eight windowed-attention blocks interleaved with four global-attention blocks. I tapped those four global outputs plus the neck to construct maps at 128, 64, 32 and 16 pixels on a side (block 3 and the neck each doubled and summed for 128², the neck alone for 64², block 6 for 32², and the average of all four tapped blocks for 16²; the diagram places each map at its source block's height rather than sorted by size). On top went the decoder Iafoss wrote for his CoaT model: three U-Net blocks upsampling via pixel shuffle, an FPN collecting every scale, and a final pixel-shuffle head to 256 × 256, with zero interpolation between the encoder and the mask.
Step 4: freeze, then unfreeze
A freshly initialised decoder fires large, noisy gradients straight back into the pretrained encoder and can scramble its weights before anything settles. Keeping the encoder frozen for the first 10 epochs let the new decoder adapt safely, after which I unfroze everything for 20 epochs with the encoder learning rate ten times lower than the decoder's, running inside Iafoss's training pipeline.
Step 5: more than one frame
Both teammates gained 0.005 to 0.01 by feeding multiple frames, so I tried bringing time in too: encoding the labelled frame and its two neighbours and mixing features with a 1D temporal convolution (the repo also has a five-frame LSTM variant, plus a UNet++ decoder with channel and spatial attention and several alternative scale setups). Pushing three 1024 × 1024 frames through ViT-B burned too much memory both during training and inside the Kaggle test budget, and none of the variations beat Step 4. The single-frame model went into the ensemble.
My final model
Here is the full model from Steps 3 and 4 opened up (SAM_U in the repository).
Model details
- Input. Single labelled ash-colour frame, bicubically resized from 256 × 256 to 1024 × 1024.
- Encoder. SAM ViT-B (the smallest released checkpoint): 12 blocks, width 768, 64 × 64 token grid. Even the base size pushed memory limits in training and Kaggle inference.
- Four scales. 32 channels at 128 × 128, 256 at 64 × 64, 512 at 32 × 32, and 768 at 16 × 16 from blocks 3, 6, 9, 12 and SAM's neck.
- Decoder. Three U-Net blocks (384, 192, 96 channels), a 32-channel FPN across scales, and a pixel-shuffle head, using LayerNorm and GELU throughout for small batches.
- Targets and loss. Continuous average of the individual labeller masks; binary cross-entropy plus Iafoss's Dice Lovász loss.
- Optimiser. AdamW, 4e-5 decoder / 4e-6 encoder, 10 epochs frozen then 20 unfrozen.
- Augmentation. Shift, scale, small-angle rotation, gamma, brightness/contrast and blur. No flips and no 90-degree rotations, for a reason we caught too late (below).
- Cost. 16 hours per seed on two A6000s (five seeds total); inference for all five takes 48 minutes on Kaggle's P100.
My teammates' models
CoaT with an LSTM over time, by Iafoss
Contrails stretch across the whole crop while staying two pixels thin, so you need both global attention to track them and fine feature maps to draw their edges. I got the first from SAM and had to hack together the second. CoaT has both natively across four scales. Iafoss upscaled the input 2× to 512 pixels and topped it with the exact pixel-shuffle U-Net decoder my model borrows.
To use five frames without tripping over moving clouds (which drift 10 to 30 pixels every ten minutes, breaking naive frame alignment and 3D convolutions), he kept time mixing strictly on the two coarsest maps at 32 × 32 and 16 × 16, where a 20-pixel shift is just a cell or two. A one-layer LSTM runs over time at each coarse location, while the sharp high-resolution skip maps come purely from the labelled frame, adding 0.005 to 0.01 Dice. He also built a variant mixing across all frame tokens with a transformer, and paired both temporal heads with a NeXtViT backbone.
Averaged over five seeds, that single CoaT + LSTM hit 0.718 on the private leaderboard, good enough for fourth place on its own.
EfficientNetV2-S, by Théo
Théo's convolutional U-Net was by far the fastest and cheapest model of the three. Instead of enlarging the 256 × 256 input, he changed the stem convolution's stride from 2 to 1, keeping every downstream feature map twice as sharp for two-pixel lines. He trained it for 100 epochs on the official set mixed with extra GOES-16 frames Iafoss gathered and labelled with our models, decaying the weight of those pseudo-labels down to zero by the end of training. A four-frame bidirectional LSTM variant existed too, though the single-frame model trained with pseudo-labels beat it and went into the ensemble in three configurations.
Results
Here is where the team's single models landed, each averaged across five or six seeds. We scored exclusively on the official validation split rather than cross-validating the training set, because overlapping crops across training regions artificially inflate fold CV.
| model | built by | frames | input | validation | public | private |
|---|---|---|---|---|---|---|
| CoaT, LSTM over time | Iafoss | 5 | 512 | 0.7039 | 0.712 | 0.718 |
| CoaT, transformer over time | Iafoss | 5 | 512 | 0.7055 | 0.722 | 0.713 |
| NeXtViT, LSTM over time | Iafoss | 5 | 512 | 0.7014 | 0.713 | 0.708 |
| EfficientNetV2-S | Théo | 1 | 256 | 0.699 | 0.704 | 0.699 |
| SAM ViT-B with U-Net decoder | me | 1 | 1024 | 0.694 | 0.698 | 0.704 |
All three architectures landed close together, yet SAM was the lone plain ViT in the mix and the only one running at 1024 × 1024. On the private test set it jumped to 0.704 (beating both its validation and public scores) and made genuinely different errors from the CNNs and CoaTs, earning a full weight in the final blend of eight models:
| model | built by | weight | validation | public | private |
|---|---|---|---|---|---|
| CoaT, LSTM over time | Iafoss | 1 | 0.7039 | 0.712 | 0.718 |
| CoaT, transformer over time | Iafoss | 1 | 0.7055 | 0.722 | 0.713 |
| NeXtViT, LSTM over time | Iafoss | 0.5 | 0.7014 | 0.713 | 0.708 |
| NeXtViT, LSTM, pseudo-labels | Iafoss | 0.5 | 0.7025 | 0.719 | 0.714 |
| EfficientNetV2-S, three variants | Théo | 1 | 0.699 | 0.704 | 0.699 |
| SAM ViT-B with U-Net decoder | me | 1 | 0.694 | 0.698 | 0.704 |
| Weighted average | all three | 0.7140 | 0.72574 | 0.72304 |
Our blend finished at 0.72304 on the private leaderboard, 2nd place, just 0.00128 behind 1st place (0.72432).
What we missed
Early in the competition we noticed something odd: horizontal flips, vertical flips and 90-degree rotations all hurt validation. That makes little sense for satellite imagery (a contrail is a contrail whichever way north points), yet we shrugged it off as prevailing winds or preferred flight corridors, switched flips off, and moved on.
Only after the deadline did we learn why: the ground-truth masks are offset from the satellite images by half a pixel. Flip a half-pixel-shifted crop and image and label no longer sit on the same grid, which is devastating when a one-pixel shift costs a quarter of the Dice score. Correct that offset first and all eight flips and rotations work cleanly at both training and test time. We had the clue staring straight at us and talked ourselves past it; fixing that alone would very likely have closed that 0.00128 gap.
Team and thanks
We worked as a three-person team. Iafoss (Maxim Shugaev) built the CoaT and NeXtViT temporal models, wrote the decoder blocks and Dice Lovász loss my model uses, put together our training pipeline and pseudo-labelled GOES-16 set, and authored our Kaggle writeup. Théo Viel built the EfficientNet pipeline, ran its pseudo-label pretraining, and squeezed all eight models into the nine-hour inference notebook. My focus was the SAM model in this post.
The competition was run by Google Research with Kaggle. Thank you to the organisers, and to the labellers who drew more than twenty thousand masks by hand, several times each.
Links
- Lee et al. The contribution of global aviation to anthropogenic climate forcing for 2000 to 2018. Atmospheric Environment, 2021.
- Teoh et al. Mitigating the climate forcing of aircraft contrails by small-scale diversions and technology adoption. Environmental Science & Technology, 2020.
- Ng et al. OpenContrails: benchmarking contrail detection on GOES-16 ABI. The dataset paper, 2023.
- Google Research: Identify Contrails to Reduce Global Warming. Kaggle competition, 2023.
- Théo Viel, Iafoss, DrHB. 2nd place solution. Kaggle writeup, August 2023.
- DrHB/2nd-place-contrails. Training code for Iafoss's models and mine. Théo's is at TheoViel/kaggle_contrails.
- Kirillov et al. Segment Anything. ICCV 2023.