5×5 cross validation · two data sets

Which foundation model, and does it replicate?

Nine modelling approaches, fifteen ADME and physicochemical endpoints across two unrelated data sets, 25 replicate models each, every one scored on a held-out test set it never saw. Four of the nine are pre-trained molecular foundation models, and only one of those four wins anything at all. The method that wins most does no downstream training whatsoever — it freezes its encoder and predicts in context. It appears three times, under three different tabular models, which is what lets the page separate what the frozen representation is worth from what the predictor bolted onto it is worth. The answer is about ten to one.

How often is each method best?

Tukey HSD across the 25 folds, counted over every endpoint × metric combination. Tied means the method could not be distinguished from the best one at α = 0.05 — a distinction a bolded maximum would hide.

Of the 27 combinations, CheMeleon is alone at the top on 5 and shares it on 6. The three Monroe arms are never alone, because almost everywhere one of them is at the top the others are there with it: Monroe with TabPFN 3.5 shares the top on 21 combinations, and Monroe with TabPFN 3 and Monroe with TabICL on 18 each. Nothing else is ever at the top, alone or otherwise. The split between Monroe and CheMeleon is not noise and it does not follow data volume — it follows the assay. Monroe leads on LogD, both microsomal stability endpoints and both Caco-2 endpoints. CheMeleon leads on all three tissue-binding endpoints, plus LogS.

The one place a from-scratch model keeps up is LogD, the endpoint with the most measurements, where single-task ChemProp shares the top with Monroe on all three metrics. Given enough data, the model learns a representation as good as the one it would have been handed.

The other end of the table is just as clean. LightGBM, MEGA-CL and Mol-JEPA are best on nothing and tied with the best on nothing, across all 27 combinations. Two of those three are pre-trained foundation models. Pre-training is not by itself worth anything here.

R² by endpoint

Mean ± standard deviation over 25 folds.

Question 1 — does pretraining help?

CheMeleon² against the identical multitask D-MPNN¹ trained from scratch. Same molecules, same folds, same hyperparameters; the only difference is how the message-passing block was initialised.

Question 2 — does multi-task training help?

The same from-scratch architecture, trained one model per assay family versus one model per endpoint. This is the control that separates multi-task transfer from the architecture itself, and it produces the cleanest gradient in the study: sort the endpoints by how much data they have and the benefit falls away almost monotonically. LOG_MGMB, with 222 training molecules, gains 0.34 R² from being trained alongside its assay family. LogD, with 5,039, is actively hurt.

Question 3 — graph network or fingerprints?

LightGBM on Morgan count fingerprints against the single-task D-MPNN — both single-task, so this is the like-for-like comparison, and it is a split decision. The graph network wins on the two endpoints with the most data (LogD, LogS) and loses on five of the remaining seven. Stripped of pretraining and multi-task transfer, a D-MPNN is not a general improvement on fingerprints for this data set.

Question 4 — does every foundation model help?

MEGA-CL¹⁰ is a graph contrastive-learning model pre-trained on roughly 100 million molecules. It is single-target by construction, so the like-for-like comparison is against single-task ChemProp: same one-model-per-endpoint setup, same folds, pre-trained backbone against random initialisation. It loses that comparison on five endpoints, wins on three and ties on one. Against LightGBM it is a coin flip, three wins, three ties, three losses. Against CheMeleon it loses nine out of nine.

Where it does help is where the data runs out. On LOG_MGMB, the smallest endpoint at 222 training molecules, MEGA-CL beats LightGBM by 0.187 R². On the two Caco-2 endpoints it collapses, and not only on R²: its Spearman ρ of 0.225 on LOG_Caco_AB is half of what LightGBM manages. The lesson is not that pre-training fails, since the only two methods that win anything here are both pre-trained. It is that the benefit belongs to a particular pre-trained model, not to pre-training as an idea.

Question 5 — does a foundation model need fine-tuning at all?

Monroe⁵ is a 58.5 M-parameter graph transformer pre-trained on 81 million PM6 molecules and 1,089 PCBA assays. What separates it from everything else here is that nothing is trained downstream. The encoder is frozen, each molecule becomes a single 720-dimensional vector, and TabPFN⁸ predicts the endpoint in context: it is handed the training embeddings together with their labels and returns the test predictions in one forward pass. No weight updates, no epochs, no per-endpoint hyperparameters. Embedding all 7,608 molecules took 32 seconds on one GPU and the 225 fold predictions took 14 minutes. On the same card the CheMeleon arm took about six hours and MEGA-CL took twenty-one.

Which TabPFN matters, and the page carries two arms because of it. This section is the one the Monroe paper was written against, TabPFN 3. The next section is the same encoder, the same embeddings and the same folds with TabPFN 3.5, released while this study was already built.

Caco-2 is where the gap is widest. Monroe reaches R² 0.471 on permeability and 0.275 on efflux; nothing else clears 0.16 and four of the other five land below zero on both. Its Spearman ρ on permeability is 0.706 against CheMeleon's 0.564 and LightGBM's 0.469, and it wins all 25 folds on all three metrics. Microsomal stability goes the same way, if less dramatically.

Protein binding reverses it. On LOG_MPPB, CheMeleon wins all 25 folds on both R² and Spearman, 0.541 against 0.300, and it takes LOG_MBPB and LOG_MGMB as well. Whatever the frozen embedding captures about passive permeability and microsomal turnover, it does not capture what fraction of a compound stays unbound in plasma — and there, fine-tuning the representation on the assay still wins.

One more thing separates it. Monroe's fold-to-fold standard deviation is the smallest on the page on R² and on Spearman ρ, and within 0.0001 of Mol-JEPA's on MAE: 0.052 against CheMeleon's 0.088 on R², averaged over the endpoints. That follows from the design. With no training loop there is no initialisation, no early-stopping epoch and no optimiser trajectory to vary. The only thing that changes between folds is which molecules are in the support set.

Question 6 — what is a newer tabular model worth?

TabPFN 3.5⁹ was released on 15 September 2026, after everything above had been run. It reads the same frozen embeddings out of the same cache, on the same folds, through the same wrapper at the same ensemble settings. The checkpoint that does the in-context prediction is the only thing that differs between this arm and the last one, which makes it the cleanest question on the page: what does a year of tabular foundation model buy a molecular one?

Less than the release notes would suggest, and more than nothing. Across the 45 endpoint × metric combinations on both collections, the newer head never costs Monroe a place in the top group and gains it four: Spearman ρ on mouse brain binding and R² and MAE on solubility in the ExpansionRx set, and MAE on MDR1 efflux in the Biogen set, which is the one combination anywhere on this page where a Monroe arm stands alone at the top. Everywhere else the two heads are statistically indistinguishable.

The per-fold pairing is sharper than the Tukey correction, and it shows a real but small effect with a direction rather than a uniform lift. On ExpansionRx the newer head raises mean R² on seven of the nine endpoints and lowers it on two; four of those gains and both of the losses survive a paired test over the folds. The largest gains are on mouse plasma protein binding (+0.044) and human microsomal stability (+0.040), the largest loss on Caco-2 permeability (−0.026), and mouse microsomal stability is the one endpoint it loses on all three metrics. On Biogen it improves three of six — human microsomal stability, human plasma protein binding and MDR1 efflux — and the other three do not move enough to separate from zero. Median movement across both collections is about a hundredth of an R², which is detectable over 25 folds and invisible next to the gap between Monroe and anything that is not Monroe.

It is not free. The 225 ExpansionRx folds took 63 minutes against TabPFN 3's 14, and the 150 Biogen folds 23 minutes more, so the newer head costs about four and a half times the inference for a hundredth of an R². That is still the cheapest arm on the page by a wide margin — CheMeleon takes six hours on the same card and MEGA-CL twenty-one — but it is the one place in this study where a clear accuracy gain and a clear cost sit next to each other, and the gain is the smaller of the two.

The conclusion the rest of the page reaches is untouched by which head is used. Every statement below about Monroe against CheMeleon, against Mol-JEPA, against LightGBM or against MEGA-CL holds for both arms, with the same sign and very nearly the same margin.

Question 7 — does adding modalities beat adding scale?

Mol-JEPA⁶ is the same shape of arm as Monroe and a different bet about what a molecule is. Instead of augmenting a structure and asking for matching views, it collects fourteen modalities of the same molecule — graph, ECFP, MOE descriptors, xTB and DFT calculations, embeddings borrowed from five other pre-trained models, and experimental ChEMBL, PCBA and TDC label vectors — masks whole modalities out, and trains a transformer to predict the missing latents from the ones that remain. The idea is that a molecule is defined by the company it keeps, not by a perturbation of its own graph. Only structure is needed at inference. The 45.4 M parameter encoder is frozen here, every molecule becomes one 512-d CLS token, and TabICL⁷ predicts in context, which is what the authors recommend.

It is best on nothing and tied with the best on nothing. Monroe beats it on all nine endpoints and all three metrics, and on most of them it wins all 25 folds. The mean R² gap is 0.18.

The obvious objection is the head, since Monroe uses TabPFN and this arm uses TabICL. That objection can be answered from both sides, because both crossings have been run. Mol-JEPA's embeddings through TabPFN at Monroe's settings move it by 0.013 R² and 0.013 Spearman ρ — detectable, at p = 0.003 and 3 × 10⁻¹⁰, and about a fourteenth of the distance to Monroe. And Monroe's embeddings through TabICL are an arm of this comparison in their own right.

None of which makes it a bad model. On Caco-2 permeability it reaches R² 0.135 where LightGBM manages −0.014 and MEGA-CL −0.747, and on LOG_MGMB, the smallest endpoint in the study, it reaches 0.503 against CheMeleon's 0.642. It is a respectable model that happens to be in a study with two better ones.

Ranking is healthier than R²

Several endpoints post R² at or below zero while ranking test compounds perfectly respectably. The test set is genuinely shifted from the training set — for LOG_Caco_Efflux the test mean is 0.92 against 0.51 in training — so a model that hedges toward the training mean is punished by R² while its ordering survives. Spearman ρ tells the more useful story for triage.

Mean absolute error

MAE is the metric a chemist reads closest to directly: it is in log units of the measurement, and it does not depend on how the test set happens to be spread. That makes it the fairest of the three here, because it is unmoved by the label shift that drags R² below zero. The ordering barely changes. A Monroe arm has the lowest error on five of nine endpoints and CheMeleon on the other four, split the same way as before: metabolism and permeability against binding. LightGBM, MEGA-CL and Mol-JEPA are significantly worse than the best on all nine.

The gaps are worth reading in absolute terms rather than as ranks. On LOG_MGMB CheMeleon is at 0.175 against LightGBM's 0.302, so the fingerprint model is wrong by roughly 73% more per compound. On LOG_HLM the eight methods span 0.285 to 0.377, a spread of 0.093 log units, narrow enough that the assay noise probably matters more than the choice.

---

Does any of it replicate?

Everything above is one data set. A comparison run once is a hypothesis, so the same nine methods, the same protocol and the same statistics were run again on Biogen's public ADME set³: 3,521 commercially sourced compounds on six endpoints, unrelated to the first collection in chemistry, in provenance and in who measured it.

The two files are not the same kind of thing. ExpansionRx is a lead optimisation campaign — 7,608 molecules in 651 clusters, 170 of them singletons. The Biogen set is a diverse commercial selection: 3,521 molecules in 1,905 clusters, 1,123 singletons. It carries no train/test split, so one was built by holding out whole clusters to the same 30% the ExpansionRx file uses. That makes the Biogen holdout cluster-pure, which the ExpansionRx one is not: 59 of its 651 clusters straddle the boundary. Biogen is the harder test of the two, so absolute numbers do not transfer between the halves of this page. Rankings within each half do.

Eighteen combinations this time, six endpoints by three metrics. Monroe takes all eighteen. Nothing that is not Monroe is best on one, and nothing that is not Monroe is so much as tied with the best on one. Against CheMeleon a Monroe arm wins every fold of every endpoint on every metric — 450 out of 450.

Which Monroe matters more here than it did on ExpansionRx. The two TabPFN arms are on top together on 17 of the 18, the exception being MAE on MDR1 efflux, where TabPFN 3.5 separates from TabPFN 3 and stands alone. The TabICL arm reaches the top on only 9. Its R² is never far off — it trails TabPFN 3 by between 0.002 and 0.025 on five of the six endpoints — but the folds are tightly enough paired that a deficit that small still separates. This is the one place in the study where the choice of head is worth arguing about, and it is the half that the ExpansionRx result would not have predicted.

So the headline replicates and the interesting part does not. On ExpansionRx the wins split by assay, with CheMeleon taking the three tissue-binding endpoints. That pattern is gone. Biogen's two plasma protein binding endpoints are precisely where CheMeleon ought to have been strong, and they are where it loses by the widest margin on the page: R² 0.478 against 0.230 on human, 0.415 against 0.154 on rat.

What shows through instead is data volume, and the clearest sight of it is the one comparison where the two frozen-encoder models disagree. Mol-JEPA, best at nothing in the first half, beats CheMeleon here on the three endpoints with the least data — solubility, and the two protein binding assays with 128 and 109 training molecules — and loses to it on the three with the most. In-context prediction from a frozen encoder degrades gently as the labels run out. Fine-tuning falls off a cliff: on those two endpoints single-task ChemProp posts R² of −0.064 and −0.005, worse than predicting the training mean, and MEGA-CL manages −0.292 and −0.426.

Three things do carry across. Monroe is the most accurate method on both collections and also the steadiest, with the smallest fold-to-fold standard deviation on all three metrics in both, and its two heads are indistinguishable on that too. MEGA-CL is the worst on both. And a from-scratch D-MPNN is still not reliably better than a fingerprint baseline: on Biogen, single-task ChemProp beats LightGBM on the four larger endpoints and loses to it on the two smallest.

MAE says the same thing, which is worth checking rather than assuming: R² is sensitive to how the test set happens to be spread, and a verdict that held on only one metric would be a verdict about the split. Monroe has the lowest error on all six endpoints, on either head, by 0.029 log units on rat microsomal stability at the narrowest and 0.091 on rat protein binding at the widest. The runner-up changes with the amount of data, and splits the six endpoints the same way R² did: CheMeleon is second on the three with the most training molecules, Mol-JEPA on the three with the least.

The Mol-JEPA pre-training check was run again, and it matters more on this side: the Biogen set is public and reaches that pre-training table through Therapeutic Data Commons, which carries a column for each of the six endpoints. 59 of the 3,521 molecules do appear by exact InChIKey, 1.7% of them, through nabla-DFT, PubChem BioAssay and TDC, with 132 more sharing a connectivity block. Not one carries a Biogen measurement, and those six label columns are empty across all 4.66 million rows. Structures overlap slightly. Labels do not.

---

How this was run

The measurements are the OpenADMET–ExpansionRx blind challenge set, contributed by Expansion Therapeutics and released by OpenADMET under CC BY 4.0. Every endpoint was put on a log10(x+1) scale except LogD, which is already a log quantity. The +1 is there because six endpoints report zeros, which plain log10 cannot take. LogS and Caco-2 Papp carry a further −6, which moves them from µM and 10⁻⁶ cm/s into molar and cm/s. That shift is a constant, so it cancels in every metric here. The 651 chemical clusters come from BitBIRCH-Lean at its default settings.

The second collection is Biogen's public ADME set, 3,521 compounds on six endpoints, released under MIT with the paper of Fang et al.³ Its values arrive already log transformed, so nothing was done to them. It carries no train/test split, so 00b_prepare_biogen.py builds one: BitBIRCH-Lean clusters the 3,521 molecules into 1,905 groups, and whole clusters are held out in a seeded order until the test set reaches the same 30% the ExpansionRx file uses. Everything downstream of that point is shared between the two collections — the same fold construction, the same seven arms, the same statistics.

The ds column fixes the train/test split (5,326 / 2,282 molecules after dropping unparseable SMILES and rows with no measured endpoint). The 25 replicates come from five repeats of a five-fold GroupKFold over the training molecules, grouped by cluster so no chemotype straddles the train/validation boundary. Each fold fits on four fifths, early-stops on the held-out fifth where there is a training loop to stop, and is scored on the same untouched test set — so every method sees identical training molecules in every fold and a difference in the metrics is a difference in the method.

• LightGBM + Morgan — count fingerprints, radius 2, 2048 bits; library defaults; one model per endpoint.
• ChemProp ST / MT¹ — chemprop 2.3.0 D-MPNN, 50 epochs, batch 64, one network per fold. Multitask groups are the three assay families: physicochemical and tissue binding, microsomal stability, Caco-2 permeability.
• CheMeleon² — the same multitask model with message passing initialised from the CheMeleon foundation model, then fine-tuned.
• MEGA-CL¹⁰ — the authors' pre-trained checkpoint and their own fine-tuning recipe (100 epochs, batch 32), one model per endpoint. Only the split was overridden, so it reads the same folds as everything else.
• Monroe + TabPFN 3⁵ — the authors' pre-trained encoder, frozen. Every molecule becomes one 720-d embedding, computed once for the whole data set, and TabPFN⁸ predicts each endpoint in context from that fold's training embeddings. Nothing is fitted by gradient descent, so the held-out fifth goes unused, exactly as it does for LightGBM.
• Monroe + TabPFN 3.5 — the same encoder, the same cached embeddings, the same folds and the same wrapper at the same ensemble settings, with TabPFN 3.5 in place of TabPFN 3. The checkpoint is the only difference between the two arms. It is chosen by the installed library rather than by anything in the script, so each arm runs in its own environment and the runner checks which checkpoint it is about to load before the first fold.
• Monroe + TabICL⁷ — the same encoder and the same cached embeddings once more, read by TabICL instead of a TabPFN. It is the same call the Mol-JEPA arm makes, so the head is identical across the two representations and the four combinations of representation and head close into a square. Monroe's authors recommend TabPFN, so this configuration is nobody's published method; it is here to price the head against the representation, and it is named that way wherever it appears.
• Mol-JEPA + TabICL⁶ — the authors' pre-trained encoder, frozen. One 512-d CLS token per molecule, then TabICL⁷ in context, which is what their model card recommends. A TabPFN head was run over the same embeddings as a control.

Every script, the data set and all 3,192,750 predictions are at model-validation-central/studies/expansion-ml-comparison. The figures and tables on this page rebuild from the stored predictions in about a minute, with no retraining.

Statistics follow the approach in “Even More Thoughts on ML Method Comparisons”: distributions of fold-level metrics, Tukey HSD corrected for multiple comparisons, and paired tests using the folds as the pairing. Every individual prediction is retained in predictions_all.parquet.

References

1. Graff, D. E.; Morgan, N. K.; Burns, J. W.; et al. Chemprop v2: An Efficient, Modular Machine Learning Package for Chemical Property Prediction. J. Chem. Inf. Model. 2026, 66 (1), 28–33. 10.1021/acs.jcim.5c02332
2. Burns, J. W.; Zalte, A. S.; Abreu, C. R. A.; et al. Deep Learning Foundation Models for Low-Data Regimes from Classical Molecular Descriptors. J. Chem. Inf. Model. 2026, articles ASAP. Introduces CheMeleon, the descriptor-pretrained D-MPNN that chemprop ships as --from-foundation CHEMELEON; preprint at arXiv:2506.15792. 10.1021/acs.jcim.6c01546
3. Fang, C.; Wang, Y.; Grater, R.; et al. Prospective Validation of Machine Learning Algorithms for Absorption, Distribution, Metabolism, and Excretion Prediction: An Industrial Perspective. J. Chem. Inf. Model. 2023, 63 (11), 3263–3274. The 3,521-compound public set is at github.com/molecularinformatics/Computational-ADME. 10.1021/acs.jcim.3c00160
4. Ash, J. R.; Wognum, C.; Rodríguez-Pérez, R.; et al. Practically Significant Method Comparison Protocols for Machine Learning in Small Molecule Drug Discovery. J. Chem. Inf. Model. 2025, 65 (18), 9398–9411. 10.1021/acs.jcim.5c01609
5. Banaszewski, B.; Fitzgibbon, A. W. Monroe: A Molecular Foundation Model for In-Context Probabilistic Inference. Preprint, 2026. A GRIT graph transformer pre-trained on 81 million PM6 molecules and 1,089 PCBA assays, adapted downstream by a frozen encoder and TabPFN rather than by fine-tuning. Code and weights at github.com/blazejba/monroe. arXiv:2608.18982
6. Rottach, F.; Schieferdecker, S.; Rudman, W.; et al. Mol-JEPA: A Multimodal Joint Embedding Predictive Architecture for Molecules. Preprint, 2026. Masks whole molecular modalities and predicts their latents rather than augmenting the structure. Code, checkpoint and the full pre-training table at github.com/Boehringer-Ingelheim/mol-jepa. arXiv:2608.22642
7. Qu, J.; Holzmüller, D.; Varoquaux, G.; Le Morvan, M. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. Preprint, 2025. arXiv:2502.05564
8. Hollmann, N.; Müller, S.; Purucker, L.; et al. Accurate Predictions on Small Data with a Tabular Foundation Model. Nature 2025, 637 (8045), 319–326. 10.1038/s41586-024-08328-6
9. Prior Labs. TabPFN-3.5. Technical report, 15 September 2026. The default checkpoint from tabpfn 9.0.0, and the head of the second Monroe arm here. priorlabs.ai/technical-reports/tabpfn-3-5
10. Jin, T.; Jin, K.; Li, Y.; et al. MEGA-CL: A Molecular Foundation Model for Generalizable ADMET Prediction through Graph External Attention and Contrastive Learning. Preprint, 2026. arXiv:2607.24314