blog›ribonanza
One matrix multiply for RNA folding
On this page
Protein structure prediction works as well as it does because researchers have roughly 200,000 experimentally solved 3D structures to learn from. RNA has nowhere near that many. The Stanford Ribonanza competition, run by the Das lab on Kaggle, tried a different kind of supervision: measure chemical reactivity along about 800,000 RNA sequences (one value per nucleotide), and predict that profile from sequence alone. Any model that nails reactivity has to learn how RNA folds.
I competed with Iafoss and slime, finishing 7th out of 755 teams, and the work contributed to the Ribonanza paper. Everyone received a matrix of base-pair probabilities from a folding program, and it quickly became clear that transformers need it. Across sixty logged experiments (forty with validation scores), I tried feeding that matrix in as a graph, via a pretrained RNA foundation model, and through convolutional nets. In the end, the dumbest possible operation won: simply multiply the node features by the matrix.
An RNA chain folding into the shape ViennaRNA, one of the packages we used, finds most likely. Each square is one nucleotide, and a thin line joins two nucleotides that pair. Then the colours switch from nucleotide identity to exposure: positions that are likely to stay unpaired turn orange and grow, and paired ones turn blue and shrink. That second picture is roughly what the two chemicals measure. The first example is one of the 457-nucleotide test sequences. The other four are well-known RNAs from the Protein Data Bank, not competition data, and they are here because they are small enough to see every nucleotide. All of it is a prediction and the drawing is flat, so it shows which positions pair and not where they sit in space. Hover or tap a square to see its partner.
The problem
RNA is a single chain of four nucleotides (A, C, G, U) that folds back onto itself: G pairs with C, A with U, and weakly G with U. Paired stretches form double-stranded stems separated by unpaired loops, and that pairing pattern (the secondary structure) packs into the 3D shape that governs what the molecule does.
Solving a 3D RNA structure directly via crystallography, NMR or cryo-EM can take a whole project of its own, and floppy RNAs often refuse to resolve at all. Sequence-based folding packages like ViennaRNA and EternaFold offer a fast shortcut by scoring secondary folds under an energy model, yet they leave out pseudoknots and long-range tertiary contacts, and frequently get the fold wrong.
Chemical mapping sits neatly in between. Treat folded RNA with a reactive probe and read out the modified positions by sequencing (where modifications show up as mutations): nucleotides locked inside a stem stay protected, while exposed loop nucleotides react strongly. Because the readout is sequencing, one experiment yields profiles for hundreds of thousands of molecules at once.
The dataset measures each sequence with two complementary reagents: DMS (which attacks base-pairing atoms on A and C) and 2A3 (which reacts wherever the backbone of any nucleotide is flexible):
| column | what it is |
|---|---|
sequence | The RNA, a string of A, C, G and U. |
experiment_type | Which chemical was used, DMS or 2A3. |
reactivity_0001… | One number per position. Scaled so that most values fall between 0 and 1. |
reactivity_error_… | The estimated error of each of those numbers. |
signal_to_noise | How clean the whole profile is. Only clean profiles are used for scoring. |
The columns of the training file that matter here.
Predictions were scored by mean absolute error against measured reactivity clipped to [0, 1].
Two things made the dataset tricky. First, it is noisy, with a large share of the 806,578 training rows failing the organisers' signal-to-noise filter. Second, the test set didn't match training lengths: training sequences run 115 to 206 nucleotides, whereas the private test sequences deciding final rankings run 207 to 457 nucleotides, and most of them were literally being synthesised and probed in the wet lab while the competition was live.
As a computed helper, the organisers provided EternaFold's L × L base-pair probability matrix for every sequence, where entry (i, j) is the chance over the thermodynamic ensemble that nucleotides i and j pair (those thin arcs in the figure above). I'll call it the pairing matrix from here on.
Forty scored experiments
Every scored experiment from my log appears below, with numbered squares marking the eight steps I walk through next (hover or tap any coloured block in the diagrams to see what changed).
Step 1: a plain transformer
I jumped in nine days after launch with twelve transformer layers over a learned nucleotide embedding. Before tuning anything, I built a leak-free validation split: because the dataset contains swarms of near-identical sequences, I clustered them by sequence similarity so whole clusters stayed on one side.
That first plain transformer scored 0.1392 on my split and 0.16144 on the public leaderboard (the two cover different sequence subsets, though they tracked each other well most of the way). Adding a single 1D convolution right after the embedding so each base sees its neighbours before attention brought both down to 0.1347 and 0.1559. Remember, though, that both my split and the public leaderboard only cover training lengths, not the longer sequences scored at the end.
Step 2: the pairs as a graph
Coming straight off three months of pairing graphs with transformers on IceCube, a graph layer was my reflex choice. I thresholded the pairing matrix at 0.5 to build edges and inserted a graph attention layer after every fourth transformer block, dropping validation from 0.1347 to 0.1304 (0.1514 public).
Then I burned an entire week trying ten variations. Plain graph convolutions did no better than having no graph; edges from one MFE structure gave 0.1316; putting the graph layers before the transformer gave 0.1312; doubling width gave 0.1310 and hurt the public score; running two parallel graphs (one on structure, one on pairs) plus node position encodings scraped down to 0.1299; and going 4× wider died with a segfault. Ten experiments for 0.0005.
Step 3: multiply by the matrix
- mix
- hi = Σj Pij xj
- gate
- g = sigmoid(W [h, x, h − x])
- output
- x′ = g · h + (1 − g) · x
where xj is the vector of position j and Pij is the chance that positions i and j pair. The gate g is one number per position, between 0 and 1.
Thresholding at 0.5 throws most of the matrix away: 0.45 and 0.02 both vanish to zero, while 0.55 looks identical to 0.99, wiping out all the information about competing alternative folds. So I threw out the threshold and simply multiplied the sequence features by the raw pairing matrix P, letting a learned gate blend each position's features with a probability-weighted average over its candidate partners (picture below).
That dropped validation from 0.1299 to 0.1271 (0.1481 public), our biggest leap since adding pairs at all. Moving the multiplication deeper into the stack barely budged the score; my log entry notes that timing had minimal impact and "its mere presence" was doing all the work.
Step 4: no graph at all
If a single matrix multiply delivers the fold information, why keep graph layers around at all? I ripped out the GNN, ran two small three-layer transformers (one multiplied by the single predicted structure, one by the pairing matrix), and concatenated them. Validation jumped around wildly at first until an exponential moving average of the weights stabilised it at 0.1268 (0.14621 public). Adding a second matrix averaged across three folding packages nudged that to 0.1267.
Step 5: a detour through a foundation model
Next I spent a week trying to squeeze signal out of RNA-FM, a foundation model pretrained on millions of non-coding RNA sequences. Fine-tuning it directly scored poorly; using frozen embeddings gave 0.1293; unfreezing hit 0.1277 before overfitting hard; running it alongside my embedding hit 0.1307 and overfitted too; using its predicted pairing matrix gave 0.1333. Even the best setup (feeding its pairing matrix as the extra channel) stalled at 0.1271, behind Step 4.
Step 6: the mixing block
- mix
- h = P x
- block
- x′ = relu(x + conv(conv(h)))
where conv is a 1D convolution over seven positions, followed by a normalisation and a relu. The gate from step 3 is gone.
Until here, the matrix entered once near the bottom. Next I moved it inside every block: a transformer layer, a matrix multiply, then two width-7 convolutions so partner features spill onto neighbouring bases (natural, since base pairs come in stems). Eight blocks plus four plain transformer layers hit 0.1259 and 0.1459.
Cycling three rounds inside each block (one for EternaFold, one for the multi-package average, one for ViennaRNA's single structure) with a block-level residual pushed this to 0.1247 validation and 0.14436 public. That became exp_32 in my log, the blueprint for everything I trained over the final six weeks.
Step 7: convolutions
Once exp_32 worked, I spent a week checking whether attention was even necessary. A pure 1D-convolutional EfficientNet bombed at 0.1623 (the worst entry in my log). Putting convs before mixing blocks gave 0.1332, and EfficientNet blocks feeding mixing blocks peaked at 0.1272 before overfitting after epoch 28. Meanwhile, a run where I patched a padding bug and swapped a folding package at the same time tanked, earning a sheepish log note ("perhaps sampler or mask") before I reverted straight back to exp_32.
Step 8: labels from our own models
From there the architecture stayed frozen. Making it deeper (six mixing blocks vs four) scored 0.1249, and fine-tuning on the external RMDB archive (the only set with measurements right at sequence ends) gave 0.1245.
What actually worked was self-training with pseudo-labels. Because the test sequences are longer than anything in training and completely unlabelled, we ran our ensemble over them and trained on its predictions. Three rounds took validation from 0.1247 down to 0.1227 (and the public leaderboard to 0.14074), with a final clean-sample pass reaching 0.121 on validation, though I trust that last number less since the public leaderboard plateaued around 0.1403.
The final model
Here is exp_32 opened up, as it went into our final ensemble.
Model details
- Inputs. Sequence plus three matrices: EternaFold pairing probabilities, the average of ViennaRNA, CONTRAfold and RNAformer, and ViennaRNA's binary predicted structure (RNA-FM came out after proving too noisy).
- Sizes. Width 384 with 32-wide attention heads; 4 or 6 mixing blocks (each holding three transformer layers), followed by 4 plain transformer layers.
- Transformer layers. Shared across the team: Iafoss swapped the standard MLP for a masked 1D convolution of width 5, calling it the decisive piece of his own model.
- Loss. Mean absolute error divided by √(1 + reported error), downweighting noisy measurements.
- Training. AdamW (lr 5e-4, weight decay 0.05), batch size 64, 64 epochs with a short warmup.
- Flipping. Every sequence also runs backwards at training and test time (recomputing the pairing matrices in reverse), worth 0.001 to 0.0015.
What the multiplication does
Each row of the pairing matrix holds mostly zeros plus the probabilities for a position's candidate partners. Multiplying the sequence features by that matrix replaces every nucleotide with a weighted sum across its pairing row: a locked-in base pair pulls its partner's features straight across, an ambiguous position blends its options, and an unpaired loop position gets zero and keeps its own state via the residual connection.
Standard self-attention can learn the same routing on its own, just slowly: early in the competition we trained transformers to match the pairing matrix in their attention maps, and it took 200 to 250 epochs to converge where handing the matrix straight in matched the score in a few dozen.
All three of us found different ways to inject the matrix. Iafoss added it to attention logits before the softmax (and later fed attention maps from one block to the next); slime passed it through a stack of 2D convolutions to produce an attention bias; my model left the attention weights alone and routed features directly along the predicted pairs.
Results
Here are all four model families from our writeup:
| model | built by | public | private |
|---|---|---|---|
| Matrix added to attention | Iafoss | 0.13973 | 0.14292 |
| Matrix through convolutions | slime | 0.14066 | 0.14509 |
| Dual stream | Iafoss | 0.13697 | 0.14296 |
| Matrix multiplied in | me | 0.1403 | – |
| Ensemble of about 20 | all three | 0.13604 | 0.14189 |
Iafoss discovered his dual-stream architecture only days before the deadline, too late to run pseudo-labelling on it, yet it still posted our best single-model public score (0.13697) and hit 0.14296 private, top-ten pace by itself.
Blending roughly twenty models across all four families gave 0.14189 on the private leaderboard and 7th place.
What else didn't work
Alongside my own detours through thresholded graphs, RNA-FM and 1D conv nets, a few bigger negative findings came out across the whole team:
- More folding packages. Computing extra pairing matrices with ViennaRNA, CONTRAfold, RNAformer and RNA-FM barely improved over the single EternaFold matrix the organisers gave everyone.
- The external dataset. Fine-tuning on RMDB helped validation and public scores by 0.0005 to 0.001, yet gave zero gain on private and mysteriously pulled predictions near sequence ends toward zero.
- Pretraining on 30 million sequences. Masked-nucleotide pretraining on a huge outside RNA corpus didn't help, whereas running the same pretraining directly on the competition sequences helped slime's model.
- Graph distances. slime computed all-pairs shortest paths through the predicted secondary structure, only for the transformer to learn those distances on its own.
- 2D models. Published U-Net architectures operating on full L × L pair matrices trailed the 1D transformers.
Batch size and noisy labels
Two quirks never got a clean explanation: training with batches of 16 sequences beat larger batches across our runs, and noisy training targets dragged down convergence until, in the final rounds, we blended each training label with our own ensemble prediction weighted by inverse error.
Team and thanks
We competed as a three-person team. Iafoss (Maxim Shugaev) built the attention-bias and dual-stream models, the conv-equipped transformer layer, and the pseudo-labelling pipeline, and wrote our writeup. slime (Eduard Martynov) built the conv-biased model along with its own masked pretraining and training pipeline. My part was the mixing model, the sequence-similarity validation split Iafoss and I shared, and the detours through graphs, RNA-FM and convolutions.
The Das lab at Stanford ran the competition on Kaggle with sequences largely designed by Eterna players and most of the test set synthesised and measured while the clock was ticking. Thank you to the organisers for that, and for releasing the pairing matrices this whole post leans on.
Links
- Ribonanza: deep learning of RNA structure through dual crowdsourcing. bioRxiv 2024.02.24.581671.
- Stanford Ribonanza RNA Folding. Kaggle competition, 2023.
- slime, DrHB, Iafoss. 7th place solution. Kaggle writeup, December 2023.
- DrHB/rna-stanford. My code and experiment log, with every notebook.
- arnie. The Das lab's wrapper around the folding packages, which I used to compute the extra matrices.
- ViennaRNA. Used for the figures in this post and as one of the folding packages in the model.
- RNA-FM. The pretrained RNA model from step 5.