5×5 cross validation · two data sets

Which foundation model, and does it replicate?

Seven modelling approaches, fifteen ADME and physicochemical endpoints across two unrelated data sets, 25 replicate models each, every one scored on a held-out test set it never saw. Four of the seven are pre-trained foundation models, and only two of those four win anything at all. The model that wins most does no downstream training whatsoever — it freezes its encoder and predicts in context. And the neatest pattern in the first data set does not survive the second.

How often is each method best?

Tukey HSD across the 25 folds, counted over every endpoint × metric combination. Tied means the method could not be distinguished from the best one at α = 0.05 — a distinction a bolded maximum would hide.

Of the 27 combinations, 19 have a single method at the top and 8 have two or three that cannot be told apart. Monroe is alone at the top on 12 and shares it on 6, CheMeleon alone on 7 and shares on 5. Nothing else is ever at the top, alone or otherwise. The split between those two is not noise and it does not follow data volume — it follows the assay. Monroe leads on LogD, both microsomal stability endpoints and both Caco-2 endpoints. CheMeleon leads on all three tissue-binding endpoints, plus LogS.

The one place a from-scratch model keeps up is LogD, the endpoint with the most measurements, where single-task ChemProp shares the top with Monroe on all three metrics. Given enough data, the model learns a representation as good as the one it would have been handed.

The other end of the table is just as clean. LightGBM, MEGA-CL and Mol-JEPA are best on nothing and tied with the best on nothing, across all 27 combinations. Two of those three are pre-trained foundation models. Pre-training is not by itself worth anything here.

R² by endpoint

Mean ± standard deviation over 25 folds.

Question 1 — does pretraining help?

CheMeleon² against the identical multitask D-MPNN¹ trained from scratch. Same molecules, same folds, same hyperparameters; the only difference is how the message-passing block was initialised.

Question 2 — does multi-task training help?

The same from-scratch architecture, trained one model per assay family versus one model per endpoint. This is the control that separates multi-task transfer from the architecture itself, and it produces the cleanest gradient in the study: sort the endpoints by how much data they have and the benefit falls away almost monotonically. LOG_MGMB, with 222 training molecules, gains 0.34 R² from being trained alongside its assay family. LogD, with 5,039, is actively hurt.

Question 3 — graph network or fingerprints?

LightGBM on Morgan count fingerprints against the single-task D-MPNN — both single-task, so this is the like-for-like comparison, and it is a split decision. The graph network wins on the two endpoints with the most data (LogD, LogS) and loses on five of the remaining seven. Stripped of pretraining and multi-task transfer, a D-MPNN is not a general improvement on fingerprints for this data set.

Question 4 — does every foundation model help?

MEGA-CL⁹ is a graph contrastive-learning model pre-trained on roughly 100 million molecules. It is single-target by construction, so the like-for-like comparison is against single-task ChemProp: same one-model-per-endpoint setup, same folds, pre-trained backbone against random initialisation. It loses that comparison on five endpoints, wins on three and ties on one. Against LightGBM it is a coin flip, three wins, three ties, three losses. Against CheMeleon it loses nine out of nine.

Where it does help is where the data runs out. On LOG_MGMB, the smallest endpoint at 222 training molecules, MEGA-CL beats LightGBM by 0.187 R². On the two Caco-2 endpoints it collapses, and not only on R²: its Spearman ρ of 0.225 on LOG_Caco_AB is half of what LightGBM manages. The lesson is not that pre-training fails, since the only two methods that win anything here are both pre-trained. It is that the benefit belongs to a particular pre-trained model, not to pre-training as an idea.

Question 5 — does a foundation model need fine-tuning at all?

Monroe⁵ is a 58.5 M-parameter graph transformer pre-trained on 81 million PM6 molecules and 1,089 PCBA assays. What separates it from everything else here is that nothing is trained downstream. The encoder is frozen, each molecule becomes a single 720-dimensional vector, and TabPFN⁸ predicts the endpoint in context: it is handed the training embeddings together with their labels and returns the test predictions in one forward pass. No weight updates, no epochs, no per-endpoint hyperparameters. Embedding all 7,608 molecules took 32 seconds on one GPU and the 225 fold predictions took 14 minutes. On the same card the CheMeleon arm took about six hours and MEGA-CL took twenty-one.

Caco-2 is where the gap is widest. Monroe reaches R² 0.471 on permeability and 0.275 on efflux; nothing else clears 0.16 and four of the other five land below zero on both. Its Spearman ρ on permeability is 0.706 against CheMeleon's 0.564 and LightGBM's 0.469, and it wins all 25 folds on all three metrics. Microsomal stability goes the same way, if less dramatically.

Protein binding reverses it. On LOG_MPPB, CheMeleon wins all 25 folds on both R² and Spearman, 0.541 against 0.300, and it takes LOG_MBPB and LOG_MGMB as well. Whatever the frozen embedding captures about passive permeability and microsomal turnover, it does not capture what fraction of a compound stays unbound in plasma — and there, fine-tuning the representation on the assay still wins.

One more thing separates it. Monroe's fold-to-fold standard deviation is the smallest of the six on all three metrics: 0.052 against CheMeleon's 0.088 on R², averaged over the endpoints. That follows from the design. With no training loop there is no initialisation, no early-stopping epoch and no optimiser trajectory to vary. The only thing that changes between folds is which molecules are in the support set.

Question 6 — does adding modalities beat adding scale?

Mol-JEPA⁶ is the same shape of arm as Monroe and a different bet about what a molecule is. Instead of augmenting a structure and asking for matching views, it collects fourteen modalities of the same molecule — graph, ECFP, MOE descriptors, xTB and DFT calculations, embeddings borrowed from five other pre-trained models, and experimental ChEMBL, PCBA and TDC label vectors — masks whole modalities out, and trains a transformer to predict the missing latents from the ones that remain. The idea is that a molecule is defined by the company it keeps, not by a perturbation of its own graph. Only structure is needed at inference. The 45.4 M parameter encoder is frozen here, every molecule becomes one 512-d CLS token, and TabICL⁷ predicts in context, which is what the authors recommend.

It is best on nothing and tied with the best on nothing. Monroe beats it on all nine endpoints and all three metrics, and on most of them it wins all 25 folds. The mean R² gap is 0.18.

The obvious objection is the head, since Monroe uses TabPFN and this arm uses TabICL. So the same embeddings were run through TabPFN at Monroe's settings. Across the 225 folds that moves Mol-JEPA by 0.013 R² and 0.013 Spearman ρ — detectable, at p = 0.003 and 3 × 10⁻¹⁰, and about a fourteenth of the distance to Monroe. The gap is the representation, not the predictor bolted onto it.

None of which makes it a bad model. On Caco-2 permeability it reaches R² 0.135 where LightGBM manages −0.014 and MEGA-CL −0.747, and on LOG_MGMB, the smallest endpoint in the study, it reaches 0.503 against CheMeleon's 0.642. It is a respectable model that happens to be in a study with two better ones.

Ranking is healthier than R²

Several endpoints post R² at or below zero while ranking test compounds perfectly respectably. The test set is genuinely shifted from the training set — for LOG_Caco_Efflux the test mean is 0.92 against 0.51 in training — so a model that hedges toward the training mean is punished by R² while its ordering survives. Spearman ρ tells the more useful story for triage.

Mean absolute error

MAE is the metric a chemist reads closest to directly: it is in log units of the measurement, and it does not depend on how the test set happens to be spread. That makes it the fairest of the three here, because it is unmoved by the label shift that drags R² below zero. The ordering barely changes. Monroe has the lowest error on five of nine endpoints and CheMeleon on the other four, split the same way as before: metabolism and permeability against binding. LightGBM, MEGA-CL and Mol-JEPA are significantly worse than the best on all nine.

The gaps are worth reading in absolute terms rather than as ranks. On LOG_MGMB CheMeleon is at 0.175 against LightGBM's 0.302, so the fingerprint model is wrong by roughly 73% more per compound. On LOG_HLM the six methods span 0.293 to 0.377, a spread of 0.084 log units, narrow enough that the assay noise probably matters more than the choice.

---

Does any of it replicate?

Everything above is one data set. A comparison run once is a hypothesis, so the same seven methods, the same protocol and the same statistics were run again on Biogen's public ADME set³: 3,521 commercially sourced compounds on six endpoints, unrelated to the first collection in chemistry, in provenance and in who measured it.

The two files are not the same kind of thing. ExpansionRx is a lead optimisation campaign — 7,608 molecules in 651 clusters, 170 of them singletons. The Biogen set is a diverse commercial selection: 3,521 molecules in 1,905 clusters, 1,123 singletons. It carries no train/test split, so one was built by holding out whole clusters to the same 30% the ExpansionRx file uses. That makes the Biogen holdout cluster-pure, which the ExpansionRx one is not: 59 of its 651 clusters straddle the boundary. Biogen is the harder test of the two, so absolute numbers do not transfer between the halves of this page. Rankings within each half do.

Eighteen combinations this time, six endpoints by three metrics. Monroe takes all eighteen. Nothing else is best on one, and nothing else is so much as tied with the best on one. Against CheMeleon it wins every fold of every endpoint on every metric — 450 out of 450.

So the headline replicates and the interesting part does not. On ExpansionRx the wins split by assay, with CheMeleon taking the three tissue-binding endpoints. That pattern is gone. Biogen's two plasma protein binding endpoints are precisely where CheMeleon ought to have been strong, and they are where it loses by the widest margin on the page: R² 0.478 against 0.230 on human, 0.415 against 0.154 on rat.

What shows through instead is data volume, and the clearest sight of it is the one comparison where the two frozen-encoder models disagree. Mol-JEPA, best at nothing in the first half, beats CheMeleon here on the three endpoints with the least data — solubility, and the two protein binding assays with 128 and 109 training molecules — and loses to it on the three with the most. In-context prediction from a frozen encoder degrades gently as the labels run out. Fine-tuning falls off a cliff: on those two endpoints single-task ChemProp posts R² of −0.064 and −0.005, worse than predicting the training mean, and MEGA-CL manages −0.292 and −0.426.

Three things do carry across. Monroe is the most accurate method on both collections and also the steadiest, with the smallest fold-to-fold standard deviation on all three metrics in both. MEGA-CL is the worst on both. And a from-scratch D-MPNN is still not reliably better than a fingerprint baseline: on Biogen, single-task ChemProp beats LightGBM on the four larger endpoints and loses to it on the two smallest.

MAE says the same thing, which is worth checking rather than assuming: R² is sensitive to how the test set happens to be spread, and a verdict that held on only one metric would be a verdict about the split. Monroe has the lowest error on all six endpoints, by 0.029 log units on rat microsomal stability at the narrowest and 0.091 on rat protein binding at the widest. The runner-up changes with the amount of data, and splits the six endpoints the same way R² did: CheMeleon is second on the three with the most training molecules, Mol-JEPA on the three with the least.

The Mol-JEPA pre-training check was run again, and it matters more on this side: the Biogen set is public and reaches that pre-training table through Therapeutic Data Commons, which carries a column for each of the six endpoints. 59 of the 3,521 molecules do appear by exact InChIKey, 1.7% of them, through nabla-DFT, PubChem BioAssay and TDC, with 132 more sharing a connectivity block. Not one carries a Biogen measurement, and those six label columns are empty across all 4.66 million rows. Structures overlap slightly. Labels do not.

---

How this was run

The measurements are the OpenADMET–ExpansionRx blind challenge set, contributed by Expansion Therapeutics and released by OpenADMET under CC BY 4.0. Every endpoint was put on a log10(x+1) scale except LogD, which is already a log quantity. The +1 is there because six endpoints report zeros, which plain log10 cannot take. LogS and Caco-2 Papp carry a further −6, which moves them from µM and 10⁻⁶ cm/s into molar and cm/s. That shift is a constant, so it cancels in every metric here. The 651 chemical clusters come from BitBIRCH-Lean at its default settings.

The second collection is Biogen's public ADME set, 3,521 compounds on six endpoints, released under MIT with the paper of Fang et al.³ Its values arrive already log transformed, so nothing was done to them. It carries no train/test split, so 00b_prepare_biogen.py builds one: BitBIRCH-Lean clusters the 3,521 molecules into 1,905 groups, and whole clusters are held out in a seeded order until the test set reaches the same 30% the ExpansionRx file uses. Everything downstream of that point is shared between the two collections — the same fold construction, the same seven arms, the same statistics.

The ds column fixes the train/test split (5,326 / 2,282 molecules after dropping unparseable SMILES and rows with no measured endpoint). The 25 replicates come from five repeats of a five-fold GroupKFold over the training molecules, grouped by cluster so no chemotype straddles the train/validation boundary. Each fold fits on four fifths, early-stops on the held-out fifth where there is a training loop to stop, and is scored on the same untouched test set — so every method sees identical training molecules in every fold and a difference in the metrics is a difference in the method.

• LightGBM + Morgan — count fingerprints, radius 2, 2048 bits; library defaults; one model per endpoint.
• ChemProp ST / MT¹ — chemprop 2.3.0 D-MPNN, 50 epochs, batch 64, one network per fold. Multitask groups are the three assay families: physicochemical and tissue binding, microsomal stability, Caco-2 permeability.
• CheMeleon² — the same multitask model with message passing initialised from the CheMeleon foundation model, then fine-tuned.
• MEGA-CL⁹ — the authors' pre-trained checkpoint and their own fine-tuning recipe (100 epochs, batch 32), one model per endpoint. Only the split was overridden, so it reads the same folds as everything else.
• Monroe + TabPFN⁵ — the authors' pre-trained encoder, frozen. Every molecule becomes one 720-d embedding, computed once for the whole data set, and TabPFN⁸ predicts each endpoint in context from that fold's training embeddings. Nothing is fitted by gradient descent, so the held-out fifth goes unused, exactly as it does for LightGBM.
• Mol-JEPA + TabICL⁶ — the authors' pre-trained encoder, frozen. One 512-d CLS token per molecule, then TabICL⁷ in context, which is what their model card recommends. A TabPFN head was run over the same embeddings as a control.

Every script, the data set and all 2,483,250 predictions are at model-validation-central/studies/expansion-ml-comparison. The figures and tables on this page rebuild from the stored predictions in about a minute, with no retraining.

Statistics follow the approach in “Even More Thoughts on ML Method Comparisons”: distributions of fold-level metrics, Tukey HSD corrected for multiple comparisons, and paired tests using the folds as the pairing. Every individual prediction is retained in predictions_all.parquet.

References

1. Graff, D. E.; Morgan, N. K.; Burns, J. W.; et al. Chemprop v2: An Efficient, Modular Machine Learning Package for Chemical Property Prediction. J. Chem. Inf. Model. 2026, 66 (1), 28–33. 10.1021/acs.jcim.5c02332
2. Burns, J. W.; Zalte, A. S.; Abreu, C. R. A.; et al. Deep Learning Foundation Models for Low-Data Regimes from Classical Molecular Descriptors. J. Chem. Inf. Model. 2026, articles ASAP. Introduces CheMeleon, the descriptor-pretrained D-MPNN that chemprop ships as --from-foundation CHEMELEON; preprint at arXiv:2506.15792. 10.1021/acs.jcim.6c01546
3. Fang, C.; Wang, Y.; Grater, R.; et al. Prospective Validation of Machine Learning Algorithms for Absorption, Distribution, Metabolism, and Excretion Prediction: An Industrial Perspective. J. Chem. Inf. Model. 2023, 63 (11), 3263–3274. The 3,521-compound public set is at github.com/molecularinformatics/Computational-ADME. 10.1021/acs.jcim.3c00160
4. Ash, J. R.; Wognum, C.; Rodríguez-Pérez, R.; et al. Practically Significant Method Comparison Protocols for Machine Learning in Small Molecule Drug Discovery. J. Chem. Inf. Model. 2025, 65 (18), 9398–9411. 10.1021/acs.jcim.5c01609
5. Banaszewski, B.; Fitzgibbon, A. W. Monroe: A Molecular Foundation Model for In-Context Probabilistic Inference. Preprint, 2026. A GRIT graph transformer pre-trained on 81 million PM6 molecules and 1,089 PCBA assays, adapted downstream by a frozen encoder and TabPFN rather than by fine-tuning. Code and weights at github.com/blazejba/monroe. arXiv:2608.18982
6. Rottach, F.; Schieferdecker, S.; Rudman, W.; et al. Mol-JEPA: A Multimodal Joint Embedding Predictive Architecture for Molecules. Preprint, 2026. Masks whole molecular modalities and predicts their latents rather than augmenting the structure. Code, checkpoint and the full pre-training table at github.com/Boehringer-Ingelheim/mol-jepa. arXiv:2608.22642
7. Qu, J.; Holzmüller, D.; Varoquaux, G.; Le Morvan, M. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. Preprint, 2025. arXiv:2502.05564
8. Hollmann, N.; Müller, S.; Purucker, L.; et al. Accurate Predictions on Small Data with a Tabular Foundation Model. Nature 2025, 637 (8045), 319–326. 10.1038/s41586-024-08328-6
9. Jin, T.; Jin, K.; Li, Y.; et al. MEGA-CL: A Molecular Foundation Model for Generalizable ADMET Prediction through Graph External Attention and Contrastive Learning. Preprint, 2026. arXiv:2607.24314