blog›icecube
Tracking neutrinos in a cubic kilometre of ice
On this page
The IceCube collaboration put more than a hundred million simulated neutrino events on Kaggle, asking where in the sky each particle came from. I joined hoping for a trip to Antarctica. Three months later, my teammate Iafoss and I finished second out of 812 teams, and our model ended up in a paper alongside the organisers and the other two winning teams.
Along the way I logged roughly ninety experiments, starting from a 1.182 baseline and grinding down to 0.963 (lower is better). Most of what I tried went nowhere, including three stubborn weeks spent trying to make the wrong kind of model work. What follows is that log in order, built around real sensor positions, real events and my actual validation scores.
The detector, drawn from the real sensor positions: 86 strings of 60 sensors, between 1.45 and 2.45 km under the surface. Press the button and a particle flies through along the true direction of one simulated event, and the sensors switch on in the order they did in the data. The table on the right fills with the same pulses as rows, which is all the input the model is given. The data only has the direction, so the particle and its exact path are my drawing. Drag to rotate.
The problem
Neutrinos barely interact with matter at all. Every once in a while one collides with an atom in the Antarctic ice, kicking off charged particles that travel faster than light does in ice and leave behind a faint blue cone of Cherenkov light. IceCube catches those flashes with 5,160 light sensors suspended through a cubic kilometre of deep ice.
Each event arrives as a list of pulses, four numbers per row:
| column | what it is |
|---|---|
sensor_id | Which of the 5,160 sensors saw it. A second file gives the position of every sensor in metres. |
time | When, in nanoseconds. Only the differences between pulses mean anything. |
charge | How much light, in photoelectrons. One photon gives roughly one. |
auxiliary | A flag. If true, the pulse wasn't fully read out and is more likely to be noise. |
One row per pulse. This is all the input the model is given.
No energy, no interaction point, no particle type: just that pulse list. From it the model predicts two angles, azimuth and zenith, scored by mean angular error in radians (a random guess lands around 1.57). From here on I'll call pulses with the flag off trusted and the rest noise, keeping in mind that the flag is really only a hint.
Events fall into two rough shapes, both shown in the 3D view above. In a track, a muon cuts across hundreds of metres and leaves a clean line of light, making the direction easy to read. In a cascade, everything dumps its energy in one spot and the light expands in a ball, which is much harder. In the final paper, the winning models get within 0.5 degrees on tracks and within 5 degrees on cascades above 10 TeV. On top of that, event lengths vary wildly:
Half of all events have 63 pulses or fewer, and roughly two thirds of a typical event is flagged as likely noise, while a handful of huge events run a thousand times longer. On paper, a variable-length cloud of noisy 3D points in space and time looks much more like a point cloud than a sentence, which is why the organisers' baseline was a graph neural network.
Ninety experiments
Every scored experiment from my log sits on the timeline below, with numbered squares marking the nine turning points I walk through next. Each step has a small diagram showing what changed in colour (hover or tap any block to inspect it), along with the formula when the change was mathematical.
Step 1: a plain transformer
- squared error
- L = mean [ (az − âz)² + (zen − ẑen)² ]
- log-cosh
- L = mean log cosh(ŷ − y)
I started with the simplest thing I could get training: a six-layer transformer encoder reading the first 100 pulses, averaging its outputs and regressing azimuth and zenith directly. That gave 1.182. Masking out padding from the average brought it to 1.169, swapping squared error for log-cosh (gentler on huge misses) reached 1.144, and giving each sensor its own learned embedding nudged it to 1.140 after two runs that blew up to NaN. One full week of work for 0.04.
Worse, both losses treat angles like ordinary scalars. Azimuths of 0.01 and 6.27 point in nearly the same direction, yet predicting one when the truth is the other triggers a massive penalty.
Step 2: a molecular graph transformer
- attention
- a = softmax(QKᵀ / √d) + 0.5 · A + 0.5 · e−D
where A is 1 for two sensors that are close neighbours and 0 otherwise, and D is the distance between them.
Next I borrowed a molecular graph transformer, mixing standard attention with an adjacency matrix and a distance matrix so neighbouring sensors acted as if they were chemically bonded. It scored 1.286, then 1.275, comfortably worse than the plain model, so I shelved it. I gave it one more shot six weeks later and didn't even get a number worth logging.
Step 3: the loss
- guess
- μ = v / |v|
- confidence
- κ = |v|
- density
- p(y) = κ / (4π sinh κ) · exp(κ μ·y)
- loss
- L = −log p(y) = −κ μ·y + log(4π sinh κ) − log κ
where v is the model's output and y is the true direction as a unit vector. The first term of the loss rewards pointing at y, multiplied by the confidence. The other two are the price of confidence, and they grow with κ.
The biggest jump in the entire competition came from changing nothing except the output head and the loss. Until then, every miss was penalised equally, even though plenty of events are genuinely unsolvable: if all you get is fifteen trusted pulses along a single string, nobody on Earth can pin down where that neutrino came from.
The von Mises-Fisher loss solves both problems at once. Instead of two raw angles, the model outputs a 3D vector whose direction is the prediction (no wrap-around seam) and whose length is its confidence. On a crisp track, it can stretch the vector long and collect a big reward; on an ambiguous mess, it keeps the vector short so a miss barely hurts. Neither the loss nor the head was my invention: both came straight out of the organisers' GraphNeT library, and switching to them dropped the score from 1.142 to 1.043 in a single step.
Two days later, scaling to 148 pulses and the full dataset pushed the exact same model to 1.008 (1.006 on the public leaderboard). Blending in a cosine term, L = ½ (LvMF + 1 − cos(v, y)), squeezed out a little extra on the smaller training split.
Step 4: three weeks of graph networks
- graph layer
- h′i = Σj ∈ N(i) MLP(hi, hj − hi)
where hi is the vector of pulse i and N(i) is its 8 nearest pulses. No information from outside those 8 reaches pulse i in one layer.
Because the organisers' baseline was a graph network, I convinced myself GNNs had to be the right inductive bias and sank the next three weeks into them. I ran their DynEdge architecture and an equivariant graph network, linked pulses by position and then by spacetime, rebuilt the kNN graph after every layer, and engineered neighbour-similarity features. Almost every single run stalled between 1.00 and 1.04, topping out at 0.999 (while one variant that separated coordinates from the other features cratered to 1.535, basically random guessing).
Meanwhile, a bit of extra fine-tuning had already carried the plain transformer to 1.000. Three weeks of graph engineering had gained me 0.001.
Step 5: a better transformer
- encoding
- γ(x) = [ sin(s · x · ωk), cos(s · x · ωk) ]
- frequencies
- ωk = 10000−k / K, k = 0 … K − 1
where x is one input number and s is a scale that sets the finest detail the model can tell apart. It was 100 in this run and 4,096 in the final models.
Roughly halfway through the competition I teamed up with Iafoss. He had skipped graph networks altogether and gone straight to transformers, bringing two things I was missing: a fast data pipeline capable of getting through the whole dataset, and a Fourier input encoding where I only had a linear layer. My very first full run on his setup scored 0.9854, gaining more in one shot than all three weeks of graph experiments combined.
In our writeup, Iafoss describes a transformer as "a GNN on steroids": every pulse connects to every other pulse, and attention decides which links matter. Passing messages across only eight neighbours keeps reasoning local, whereas reconstructing a neutrino's trajectory is fundamentally a whole-event problem.
Even so, I couldn't quite shake the feeling that local geometry had to help somewhere. From that point on, the transformer became the backbone, and I spent the next few weeks trying different ways to bolt local structure onto it.
Step 6: a graph network in front
First attempt: take a pretrained 0.99 graph network, freeze its weights, and feed its features into the transformer. That regressed to 0.995 and 0.994. Swapping in two layers of the equivariant network gave 0.988. A few days later I tried a graph front-end yet again (the log entry literally opens with "retrying one last time") and landed at 0.994. Every single combination lost to the bare transformer.
Step 7: local attention
- local attention
- aij = qi·kj / √d if j ∈ N8(i), −∞ otherwise
where aij is the attention score from pulse i to pulse j before the softmax, and N8(i) is the 8 nearest pulses to i. Minus infinity sets the attention weight from pulse i to pulse j to zero.
Next I restricted the first few attention layers to each pulse's eight nearest neighbours before handing off to full global attention. That actually worked: it started at 0.983 and reached 0.9787 after fine-tuning on the full dataset. Ten days of work, and my best model so far.
The catch was speed: gathering neighbours shuffles tensors all over the GPU. Once the plain full-attention model trained longer, it caught right up. We also tried masking patterns, alternating local and global blocks, and parallel attention layers, and the plain transformer beat all of them.
Step 8: two branches
- interval
- ds²ij = c² (ti − tj)² − |xi − xj|²
- attention
- aij = qi·kj / √d + f(dsij)
where t and x are the time and position of a pulse and c is the speed of light. The interval is near zero for two pulses that one flash of light could have caused. f is a small learned function, so the model learns how much weight to give the interval.
Third idea: stop forcing local and global layers to fight in series. Run a graph branch over each pulse's nearest neighbours alongside a full-attention branch over the whole event, then merge their outputs into the main transformer. Version one scored 0.9806; adding Iafoss's spacetime bias (more on that below) to the global branch and running three rounds of fine-tuning brought it down to 0.9752.
Step 9: the final shape
- loss
- L = arccos(μ·y) + 0.05 · LvMF
where the first term is the angle between the guess and the truth, which is exactly what the leaderboard measures. The small second term keeps the length of the vector meaningful as a confidence, which the ensemble needs.
One last simplification dropped the separate global branch. The graph network became just another stream of input features sitting alongside the Fourier encodings, feeding straight into the same transformer. At the small (S) size, that started at 0.9743.
Then the loss delivered its second major breakthrough. Von Mises-Fisher is great for getting training off the ground, but it isn't the competition metric. Switching the final passes directly to angular error, keeping just 0.05 of von Mises-Fisher so vector length stayed calibrated for ensembling, jumped one fine-tuning run from 0.9725 to 0.9669. Additional passes, finishing on longer sequences, carried the S model to 0.9636, while the larger B variant reached 0.9647. On the very last day, a pure B transformer with zero graph features hit 0.9634.
After six weeks of trying to prove local structure mattered, my small graph-feature model and a pure transformer four times its size finished within 0.001 of each other. We put both into the ensemble.
The final model
Here is the architecture I trained for our final ensemble, built in two sizes: S and B. Hover or tap any block to see how pulses turn into a 3D direction vector.
Model details
- Inputs. Position is divided by 500 m, time is scaled by (t − 10,000) / 30,000, charge becomes log10(charge) / 3, and the model also sees the log of the pulse count in the event.
- Which pulses. Training caps each event at 192 pulses and inference at 768. On longer events, trusted pulses are sampled at random first and noise fills the rest.
- Graph features. Each pulse connects to its 8 nearest neighbours (by position in S, by position and time in B). Those layers take nine inputs: position, time, charge, the auxiliary flag, whether the sensor sits inside DeepCore, and the scattering and absorption of the ice at that depth.
- Sizes. S is 384 wide with 4 + 8 blocks; B is 768 wide with 4 + 12 blocks. My graph variants put the spacetime bias only on the first of the four early blocks, while a later pure transformer gives it to all four.
- Training. AdamW (weight decay 0.05), an effective batch of 4,096 events, cosine schedule with warmup, and 4 to 5 passes over the data. Peak learning rate starts at 5e-4 for S and 1e-4 for B, dropping near 1e-5 in later passes. From pass two onward, the weights are averaged.
- Loss. Von Mises-Fisher for the first two or three passes, then angular error plus 0.05 times von Mises-Fisher (as in Step 9).
- Hardware. Trained on two A6000s, taking about 56 hours per pass for the B model with graph features.
The spacetime bias
This trick came from Iafoss, and it's pure physics. If the same particle triggered two pulses, light had just enough time to get from one sensor to the other, so the distance between them roughly matches the time gap times the speed of light. In relativity that's written as the spacetime interval, ds² = c²dt² − dx² − dy² − dz², which sits near zero for two pulses from the same particle. A random noise pulse has no such link, so its intervals scatter everywhere.
Computing that interval for every pair of pulses and adding a learned function of it straight into the attention scores lets the model spot which hits belong together from the very first layer. In our tests, this bias alone was worth about 0.004.
Results
Here are the single-model scores from our writeup. T, S and B are the usual tiny, small and base transformer sizes, and the number after "d" is the width of each attention head.
| model | parameters | validation | public | private |
|---|---|---|---|---|
| T, d32 | 7.6M | 0.9704 | 0.9693 | 0.9698 |
| S, d32 | 29.3M | 0.9671 | 0.9654 | 0.9659 |
| B, d32 | 115.6M | 0.9642 | 0.9623 | 0.9632 |
| B, d64 | 115.6M | 0.9645 | 0.9635 | 0.9629 |
| B + graph, d48 | – | 0.9643 | 0.9624 | 0.9627 |
| S + graph, d32 | – | 0.9639 | 0.9620 | 0.9628 |
| B + graph, d32 | – | 0.9633 | 0.9609 | 0.9621 |
The graph-feature rows sit slightly ahead of the pure transformers, though I trained them on different hardware with a larger real batch, so take that gap with a grain of salt.
One extra pass on longer sequences pushed the B + graph model to 0.9608 public and 0.9618 private, our strongest single model. Our final submission was a weighted average of six models. Because each model outputs a vector whose length reflects its confidence, averaging the vectors automatically lets whichever model is more certain on an event speak louder. That ensemble scored 0.9594 public and 0.9602 private, good for second place.
Ensembling added surprisingly little on top of the best single model. Even the tiny 7.6M-parameter T model, taking roughly half a day per pass, would have landed in the top five on its own.
What else didn't work
Beyond the graph and local-attention dead ends in my log, a few other surprises and failed ideas stood out across both of our runs:
- Where graph networks lose. Splitting events by confidence showed that on the hardest 70% of events, graph networks and transformers tied. The entire margin came from the clean, high-confidence 30% where there's enough light to read the whole event at once.
- Ice properties as inputs. No measurable gain. With over a hundred million events, the network infers the ice's scattering and absorption directly from sensor depth. We kept both feature sets anyway for ensemble diversity.
- Smarter pulse selection. On long events, keeping the brightest pulses or taking a time window around the peak charge both lost to picking pulses at random.
- Predicting a map of the sky. Iafoss tried framing the task as classification over sky patches; it stalled near 1.0.
- Bigger models. Each 4× jump in parameters bought roughly 0.003. We had to stop at B because a driver bug kept two 4090s from training together until the final two weeks; our rough guess is the next two sizes up would have reached around 0.958 and 0.955.
Time zero in simulated events
One big caveat for anyone taking these models outside Kaggle: the competition data is simulated, and time zero carries a cue a real detector never gives you. Models quietly learn from when the first pulse arrives, and stripping that reference out in one test cost roughly 0.015. Most teams leaned on it without realising, us included. On real IceCube data, you would need to train without it.
Credits
We ran as a two-person team. Iafoss (Maxim Shugaev) built the transformer backbone the solution rests on: the caching pipeline with length-matched batches, the Fourier encoder, the spacetime bias and the BEiT architecture. He also wrote our Kaggle writeup, which covers his experiments and robustness checks in detail. My side was the early exploration, the graph and local-attention trials, and the graph-feature models in the final ensemble.
The competition was put together by the IceCube collaboration and TU Munich. Afterwards, Philipp Eller and Rasmus Ørsøe from the organising team co-authored a paper with all three winning teams comparing the top solutions head-to-head on the same data.
Links
- Bukhari, Chakraborty, Eller, Ito, Shugaev, Ørsøe. IceCube – Neutrinos in Deep Ice: the top 3 solutions from the public Kaggle competition. European Physical Journal C 84, 571 (2024). arXiv:2310.15674.
- IceCube – Neutrinos in Deep Ice. Kaggle competition, 2023.
- Iafoss, DrHB. 2nd place solution: Neutrino direction prediction with transformers. Kaggle writeup, April 2023.
- DrHB/icecube-2nd-place. Training and inference code for the final models.
- DrHB/icecube-journal. My experiment log, with every script and notebook.
- GraphNeT. The organisers' library. The graph layers and the von Mises-Fisher loss come from it.