5×5 cross validation · two data sets

Is picking a model per endpoint worth it?

Trimole-Hybrid¹ argues that no single molecular representation suits every ADMET endpoint, so it builds a pool of candidate predictors, fits all of them, and keeps whichever wins on a validation split. Here it is put against four fixed architectures on fifteen endpoints across two unrelated data sets — 22,500 candidate fits to produce 1,425 fold models, every one scored on molecules it never saw. It loses to a single well-initialised graph network on both, for two reasons that only show up when you look at what it picked: nothing in its pool learns a representation, and on one of the two data sets the split it selects on does not resemble the split it is scored on.

What the method actually is

Every other method on this page is an architecture. You hand it molecules, it trains, and the model you get is the model the architecture implies. Trimole-Hybrid is a procedure instead. For each endpoint and each fold it assembles sixty candidate predictors from three ingredients, fits every one on the training molecules, scores them all on the held-out validation fifth, and keeps exactly one.

The protocol this project already uses fits the method without modification, because every fold here already carries a validation split. Each candidate trains on the same four fifths of the training molecules that every other method trains on, is scored on the held-out fifth, and the winner is applied to the fixed test set. Nothing is refit on train-plus-validation afterwards, which would have handed this arm more training data than the other four and quietly broken the comparison.

How often is each method best?

Tukey HSD⁹ across the 25 folds, counted over every endpoint × metric combination. Tied means the method could not be distinguished from the leader at α = 0.05.

The answer to the question in the headline, on this data set, is no. CheMeleon — one fixed architecture, chosen once and applied to every endpoint — is on top of 25 of the 27 combinations. Trimole-Hybrid, having fitted sixty candidates per fold and kept the best of them on evidence it was entitled to use, is on top of 11, and beaten outright on 16.

It is not last. It comfortably beats LightGBM, which is on top of nothing, and it beats both from-scratch ChemProp variants on the tally. The selection is buying something. It is just buying less than a well-initialised message-passing network costs.

R² by endpoint

MAE by endpoint

What did it choose?

This is the part no other method has. Because the selection happens before the test set is touched, what it picked is a result rather than a diagnostic — and it is the result that can falsify the method's premise. If the choice never varies between endpoints, the pool is decoration. If it never repeats within an endpoint, the choice is noise.

The choice varies by endpoint and holds within one, which is the pattern the method needs: the modal view accounts for 75% of an endpoint's folds. Both microsomal stability endpoints take the chemistry priors alone on 24 of 25 folds; LogD takes KPGT on 20 of 25. So the pool is doing real work rather than dressing up a fixed answer.

But look at what it declines. Across all 225 folds the classical chemistry priors win alone 140 times, KPGT 66, and the other two encoders together 11. Random forest, one of the four backends, wins twice. Most of the candidate pool exists to be rejected — which is the paper's own ablation finding, arrived at from the other direction.

One fold is worth singling out. LOG_MGMB r1 f0 has six molecules in its validation split, and the candidate that won them scored an R² of −0.090. On the smallest endpoint the selection signal runs out, and what is left is a coin toss between sixty models.

Against the best fixed architecture

Folds are paired: both methods saw the same molecules in the same split, so the comparison is within-fold rather than between averages.

CheMeleon wins seven of the nine endpoints outright, loses one — LOG_MLM — and draws one. The losses are not close: on LogD, LogS, LOG_MPPB, LOG_MBPB and LOG_MGMB the selection procedure fails to take a single fold out of 25.

The reason is structural, and it is visible in the R² table above. On LogD, the endpoint with the most training data, single-task ChemProp reaches 0.700 and Trimole-Hybrid reaches 0.399 — worse than plain LightGBM on Morgan fingerprints, at 0.510. Every one of the sixty candidates is a fixed representation: a frozen encoder or a block of classical descriptors, handed to a tree ensemble or a ridge. Nothing in the pool learns a representation from the endpoint. Where there is enough data to learn one, a network that does so wins, and no amount of choosing between fixed alternatives closes the gap.

Where Trimole-Hybrid does win — LOG_MLM, and on the draw at Caco-2 A→B — is where every method is struggling and no representation is much good.

Does it replicate?

Everything above is one data set. The Biogen public ADME set⁸ is 3,521 commercially sourced compounds on six endpoints, far more chemically diverse, and split by holding out whole clusters. Same protocol, same pool, nothing retuned.

The ranking replicates: CheMeleon is on top of 14 of the 18 combinations, Trimole-Hybrid 6, and the two from-scratch ChemProp variants 2 each. What does not replicate is what the method chose. On ExpansionRx the chemistry priors alone won 62% of folds and KPGT 29%; here KPGT wins most often and the priors drop to a third, while the modal choice within an endpoint falls from 75% of folds to 53%. The same procedure, given different chemistry, reaches for different evidence — and is less sure of itself.

Head to head it takes LOG_RPPB from CheMeleon on 19 of 25 paired folds and draws on LOG_HPPB, its only two results against CheMeleon here that are not defeats. These are the two smallest endpoints in either data set, 109 and 128 training molecules, and they are also precisely where CheMeleon is weakest. The selection procedure is not winning because it is good at small data; it is winning where the strongest fixed architecture happens to fail.

Why selecting on validation was the wrong bet

Putting the two data sets side by side explains the whole result. A procedure that picks its model on a validation split is making one assumption: that doing well on that split predicts doing well on the test set. That assumption is checkable, because every selection record stores the score the winning candidate earned on validation, and the fold metrics store what it then achieved.

ExpansionRx

Biogen

On ExpansionRx the winning candidate is flattered by between 0.16 and 0.47 R². It looks like a model that explains 82% of the variance in LogD and delivers 40%. On Biogen's four well-populated endpoints the same gap is 0.03 to 0.05 — the validation split is telling very nearly the truth. The two endpoints where Biogen's gap opens up, to 0.23 and 0.25, are the two with about two dozen validation molecules, which is small-sample noise rather than a split that misleads.

The difference is in how the test sets were made. Biogen's holdout was built here, by holding out whole BitBIRCH clusters — the same rule that builds the cross-validation folds, so validation and test are the same kind of problem and equally hard. The ExpansionRx split arrived with the challenge and was drawn some other way, so its held-out fifth and its test set are not the same kind of problem at all.

That is the risk a selection framework carries and a fixed architecture does not. CheMeleon does not care whether validation resembles the test set, because it never consults validation to decide what it is. Trimole-Hybrid stakes its entire output on that resemblance, sixty candidates deep. When it holds, the method is competitive. When it does not, the method optimises confidently for the wrong thing — and nothing in the procedure can tell it so.

Reported the way “Even More Thoughts on ML Method Comparisons” argues comparisons should be: distributions over folds, simultaneous confidence intervals, paired tests, and no bolded maxima.

References

1. Luo, Z.; Huang, D.; Shao, Y.; Yu, Q.; Li, Y. A Multimodal Representation Learning Platform for Accurate Molecular ADMET Prediction. Bioinformatics 2026, in review. The method this page reimplements. Their release is an audit package rather than a runnable pipeline, which is why this is a reimplementation. 10.1101/2026.08.24.746660
2. Graff, D. E.; Morgan, N. K.; Burns, J. W.; et al. Chemprop v2: An Efficient, Modular Machine Learning Package for Chemical Property Prediction. J. Chem. Inf. Model. 2026, 66 (1), 28–33. 10.1021/acs.jcim.5c02332
3. Burns, J. W.; Zalte, A. S.; Abreu, C. R. A.; et al. Deep Learning Foundation Models for Low-Data Regimes from Classical Molecular Descriptors. J. Chem. Inf. Model. 2026, articles ASAP. 10.1021/acs.jcim.6c01546
4. Li, H.; Zhao, D.; Zeng, J. A Knowledge-Guided Pre-training Framework for Improving Molecular Representation Learning. Nat. Commun. 2023, 14, 7568. The graph branch of the candidate pool, run from the authors' pre-trained checkpoint. 10.1038/s41467-023-43214-1
5. Zhou, G.; Gao, Z.; Ding, Q.; et al. Uni-Mol: A Universal 3D Molecular Representation Learning Framework. ICLR 2023. The 3D branch, via unimol_tools. ICLR 2023
6. Chithrananda, S.; Grand, G.; Ramsundar, B. ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction. arXiv 2020, 2010.09885. The sequence branch, checkpoint seyonec/ChemBERTa-zinc-base-v1. arXiv:2010.09885
7. Notwell, J. H.; Wood, M. W. ADMET Property Prediction through Combinations of Molecular Fingerprints. arXiv 2023, 2310.00174. The chemistry-prior feature set the candidate pool's sidecars are built from. arXiv:2310.00174
8. Fang, C.; Wang, Y.; Grater, R.; et al. Prospective Validation of Machine Learning Algorithms for Absorption, Distribution, Metabolism, and Excretion Prediction. J. Chem. Inf. Model. 2023, 63 (11), 3263–3274. The second data set. 10.1021/acs.jcim.3c00160
9. Tukey, J. W. Comparing Individual Means in the Analysis of Variance. Biometrics 1949, 5 (2), 99–114. 10.2307/3001913