Model Validation Central

What survives a hard split

Four modelling approaches on the ten ChEMBL potency data sets and six splitting schemes of Lee, Moldagulov and Grzybowski¹ — on that paper's own folds, against that paper's own naive baseline and its own six arms.

The paper this page is built on makes two claims. The first is that k-nearest neighbours performs comparably to modern machine learning under every splitting scheme it tries, which is the finding of Janela and Bajorath restated on ten targets and six schemes. The second is that as the schemes get more stringent, every method's accuracy deteriorates towards a naive baseline, so that reported accuracies obtained under lenient splits are inflated. Both claims are tested here on the paper's own folds, with two kinds of model it did not have: a pre-trained graph network that is fine-tuned, and a pre-trained encoder that is frozen and read in context, with nothing trained downstream at all.

The second claim survives and the first does not. Raw MAE rises for every arm as the splits harden — for Monroe + TabPFN 3.5, the strongest arm here, by +0.358 pIC₅₀ units from the random scheme to the time one. But the naive baseline rises by +0.061 over the same span, and most of what looks like a collapsing model is a moving target. What does not move much is the distance between the two: 0.439 pIC₅₀ of MAE taken off the naive predictor on the easiest scheme, 0.141 on the hardest.

On the first claim, k-NN is separable from Monroe + TabPFN 3.5 on all six schemes by a Holm-corrected paired test over fifty folds, including the time split, where the gap is 0.131 pIC₅₀ and Monroe + TabPFN 3.5 is ahead on 90% of the fifty folds. Across all 18 scheme × metric combinations Monroe + TabPFN 3.5 is best alone on 7 and tied for best on 11. The paper's own six arms, imported at their published per-fold numbers, do not change that: they are all variations on learning from one target's labels and a fingerprint, and the thing that separates here is the representation.

Whether that is good news depends on what is being asked. If the question is whether the field's reported accuracies are inflated by lenient splits, the paper is right and this page agrees with it. If the question is whether anything beats nearest-neighbour lookup out of distribution, the answer turns out to depend on which models are in the comparison.

---

The collapse, and what is left under it

The paper's Figure 2 is a set of boxplots of MAE, one panel per scheme, and the trend in it is unmistakable: error climbs as the splits harden until the models sit on top of a naive baseline. The figure below is that figure, rebuilt on this study's arms and this study's folds, and the trend is the same. Nothing here disputes it.

What the raw axis hides is that the baseline moves too. The training median scores 0.879 MAE on the random scheme and 0.940 on the time one, a drift of +0.061 that has nothing to do with any model. The schemes select structurally different training sets, so the potency distribution a fold trains on is different, and its median is a different guess. The paper says this in a parenthesis under its own figure. It is worth promoting to an axis.

Read as a distance from that baseline, the picture is less uniform. The best arm on each scheme takes between 0.121 and 0.439 pIC₅₀ off the naive predictor — thinnest on the umap cluster scheme, widest on the random one. On the time split, Monroe + TabPFN 3.5 still beats the naive predictor on 86% of the fifty folds. 2 arm × scheme combinations fall below zero — GNN (published) on umap cluster and time — which is to say those arms would have done better predicting the training median for every molecule.

The distinction matters for what the paper concludes from the trend. Accuracy quoted on a random split is inflated as a guide to prospective performance — that is established, here and there. Accuracy converging towards a naive baseline is a stronger claim, and on this evidence it holds for the arms the paper tested and not for Monroe + TabPFN 3.5.

Mean MAE taken off the naive predictor, in pIC₅₀ units, paired on the fold; the small figure is the share of the fifty folds on which the arm beats it. Positive is better than predicting the training median.

---

What “stringent” actually means

Up to this point the six schemes have been an ordering borrowed from the paper. They can be measured. For every one of the 300 folds, each test molecule's nearest training neighbour was found by ECFP4 Tanimoto, and the fold summarised by the mean of those similarities — the quantity the paper plots in its Figures 5 and 6.

The ordering mostly holds. Mean nearest-neighbour similarity falls from 0.751 on the random scheme to 0.393 on the umap cluster one, where 40% of test molecules have no training neighbour above the Butina threshold the paper itself clusters at. It is not quite monotone in the paper's presentation order — the time and UMAP splits swap, and the diverse split is closer to them than its position in the sequence suggests — which is worth knowing before reading a trend across the six as a straight line.

That turns the independent variable from a six-valued label into a continuous one, and with it each method's accuracy can be plotted against distance directly. The slope of that line is the claim the paper is really making: that every method decays at the same rate as the test molecules move away. It is fitted below on the gap to the naive baseline rather than on raw MAE, because raw MAE confounds the decay with the baseline drift of the previous section. The shallowest decay belongs to RF (published).

Extrapolated to a similarity of 0.30 — roughly what the hardest schemes deliver — LightGBM + Morgan, ChemProp + CheMeleon, Monroe + TabPFN 3.5, SVM (published) and RF (published) still take MAE off the naive predictor, while k-NN (Tanimoto), k-NN (published), XGBoost (published), MLP (published) and GNN (published) no longer do.

Averaged over the fifty folds of each scheme. “Below 0.35” is the share of test molecules whose nearest training neighbour falls under the Butina threshold the paper clusters at — molecules with no close analogue to learn from. Train and test are molecules per fold.

Before reading a trend across the six, it is worth asking whether the six vary one thing or two. Four of them train on 72.0% of a target and test on 20.0%, and they do it to within a rounding error: across one target's five folds the largest training set and the smallest differ by about one molecule on the random, scaffold and Butina schemes. The other two do not. The diverse (maxmin 25%) scheme trains on 18% of a target, 242 molecules against 968. The time scheme trains on 50% of a target, 672 molecules against 968.

That matters because training-set size and distance travel together across the whole collection: Spearman ρ = +0.38 between the two over all 300 folds. Within the four size-matched schemes it disappears, ρ = +0.09, p = 0.18. So random, bemis-murcko scaffold, butina cluster and umap cluster are a clean four-point experiment — nearest-neighbour similarity falls from 0.751 to 0.393 while the training set stays at about 968 molecules — and the other two confound moving the test set away with taking the training data away. A model doing worse on the diverse split than on the random one has been handed a quarter of the data as well as a harder test set, and nothing in a raw comparison separates those.

Nothing below is dropped on that account, because the paper's six schemes are what this study set out to run. But the decay fitted against distance pools all six, so it carries some of the confound, and the comparison that does not is the one across the four size-matched schemes. Where the two tell the same story it is said once; where they differ, the four-scheme version is the one to believe.

Percentages are of each target, because the ten data sets run from 1,000 to 2,273 molecules. “Fold range” is the largest minus the smallest training set among one target's five folds, averaged over targets, and “Test CV %” the same spread for the test sets — both are what a scheme controls, unlike the differences between targets. Green marks the four schemes that hold training-set size fixed.

A linear fit of the gap to the naive baseline against each fold's mean nearest-neighbour similarity, over all 300 folds and all six schemes pooled. The slope is how fast a method loses its advantage as the test molecules move away; the last two columns are what the fit says it is worth at a similarity typical of the scaffold split and of the UMAP split. A negative number is a method that would have done better predicting the training median.

---

Which arms can actually be separated

The tally below counts, over every scheme and every metric, how often each arm is best on its own, tied for best, and separably worse. A method that cannot be distinguished from the leader is counted as tied and the leader is not called a winner, which is the convention the rest of this repository uses and the reason there are no bold maxima anywhere on the page.

The verdicts come from Holm-corrected paired tests over each scheme's fifty folds, not from the pooled Tukey the paper's aggregation implies. The two figures below show why that choice is not cosmetic. In the pooled version every interval is wide enough to swallow the field, and the honest reading of it is that nothing can be told from anything — which is, in effect, the paper's conclusion. The between-target spread in this collection is around 0.2 pIC₅₀ and the differences between methods are a few hundredths, so fifty numbers treated as independent samples have almost no power. Paired on the fold, where both arms saw the identical training molecules, the same data separates 1 arm at the top of the random scheme from the rest.

The correction is not doing the work of hiding differences either. On the umap cluster scheme 4 arms share the top on MAE — ChemProp + CheMeleon, Monroe + TabPFN 3.5, SVM (published) and RF (published) — and that is a real tie rather than a failure of power, because the same test separates the field elsewhere. k-NN (Tanimoto), LightGBM + Morgan, Training median, k-NN (published) and XGBoost (published) reach the top of nothing, anywhere.

Two things could make the verdicts above look stronger than they are, and both are checked rather than argued about. The differences might not be normal, so every pair is also tested with a Wilcoxon signed-rank test: it agrees with the t-test on 894 of the 930 pairs on this page. More seriously, the fifty folds of a scheme are not fifty independent observations — the time split gives each fold its own year cutoff, so a target's five test sets overlap. Averaging each arm's five folds within a target first and pairing on the ten targets removes that entirely, at the cost of going from fifty paired observations to ten.

That target-level test agrees with the fold-level one on 696 of 930 pairs. Where they differ it is almost always the same direction — 231 calls are lost and 3 gained — which is what losing 80% of the sample should do, not evidence that the fold-level test was wrong. The question is whether the headline survives it, and it does: Monroe + TabPFN 3.5 separates from k-NN on five of the six schemes even on ten paired targets, the exception being diverse (maxmin 25%). Every p value in head_to_head.csv is stored under all three tests.

---

k-NN against the rest

The k-NN arm here is the paper's own, re-implemented from its released code: a distance-weighted KNeighborsRegressor on a precomputed Tanimoto distance matrix over ECFP4 bit vectors, with k chosen from {1, 3, 5} by validation MAE. It is the one arm in this study whose numbers can be checked against the source, and 500 of 500 (5 schemes × 50 folds × MAE and R²) reproduce the published per-fold values to within 6e-09. The fifty folds that do not are all on the one scheme whose validation split upstream did not record; the protocol section says what was done about that. So when the table below disagrees with the paper, it is not disagreeing about what k-NN did.

It disagrees about what that means. Against Monroe + TabPFN 3.5, k-NN is separably worse on all six schemes (random, bemis-murcko scaffold, butina cluster, diverse (maxmin 25%), umap cluster and time). There is no scheme on which it holds its own. A Wilcoxon signed-rank test in place of the t-test agrees on 6 of those 6.

Against the fingerprint baseline the paper's conclusion fares better: LightGBM on Morgan counts separates from k-NN on three of the six schemes (bemis-murcko scaffold, umap cluster and time). Two similarity models on the same fingerprint do behave alike, which is the part of the Janela and Bajorath result this study reproduces rather than contradicts. What breaks the pattern is not a better learner on the same features. It is different features.

---

Ranking, not just error

MAE is the paper's metric and it is the right one for a headline, because it is in the units of the endpoint and it can be read against a naive baseline. It is also the metric least kind to a model that has the ordering right and the scale wrong, which is the failure mode of a model extrapolating off its training distribution. Spearman ρ asks the question a screening campaign actually asks: can the compounds be put in the right order.

The answer degrades the same way, and further. The best ρ on the random scheme is 0.809; on the time scheme it is 0.378. Unlike MAE, there is no moving baseline to subtract here — a rank correlation of zero is zero on every scheme — so this is the cleanest statement of how much predictive signal is left when the test molecules are genuinely elsewhere in chemical space, and it is not much.

The naive baseline is absent from this table and from the figure under it. A constant prediction has no ranking to score, and Spearman ρ against it is undefined rather than zero; it is recorded as missing rather than filled in, so it does not appear as a bar at the origin that a reader could mistake for a measurement.

---

How this was run

Ten ChEMBL 30 potency data sets, 1,000 to 2,273 molecules each and 13,444 in total, with pIC₅₀ from 5 to 11. They are the collections assembled by Janela and Bajorath and redistributed in the paper's release, and they arrive here unchanged.

Nothing was split here. The release ships, for each target and each of five folds, six splitting schemes as explicit train / validation / test index lists, and those lists are the protocol. Every arm fits on a fold's training indices and is scored on its test indices, so every arm sees identical training molecules in every fold and a difference in the metrics is a difference in the method. The validation indices are used only where an arm needs them — k for k-NN, early stopping for ChemProp. LightGBM and Monroe have nothing to tune per fold and no training loop to stop, so they leave those molecules unused rather than training on molecules the other arms never see.

Which copy of the splits. The release ships them twice and the two copies disagree: data/splits/indices/*.pkl and data/splits/*.json agree on the random, scaffold and time schemes and differ on butina, umap and diverse — for umap the two test sets do not share a single molecule. The .pkl files are the ones the paper's numbers came from: re-running its released k-NN on them reproduces 500 of 500 published per-fold MAE and R² values to 6e-09 on the five schemes recorded in full, where the .json indices reproduce butina and umap not at all. So the .pkl indices are the protocol, and that reproduction is kept as a check that runs on every collection pass rather than as a note.

The sixth scheme needed a repair. diverse_25 has an empty validation set in every one of the fifty .pkl files — upstream carved one out of the training pool inside the run and did not record it. The .json copy does record one, and on all fifty combinations it is a subset of the .pkl training pool, the two test sets are identical, and .pkl train equals .json train plus validation. Taking the validation molecules from the .json copy therefore contradicts nothing, and it is what gives ChemProp something to stop early on. All three relations are asserted before it is done. The fifty k-NN folds that do not reproduce are all on this scheme, and the published arms on it trained on a different 237 molecules of the same 263-molecule pool against an identical test set — comparable, but not identical.

• k-NN — ECFP4 bit vectors, radius 2, 2048 bits; Tanimoto distance; distance weighting; k from {1, 3, 5} on the validation split. The paper's own arm, re-implemented from its code.
• LightGBM + Morgan — count fingerprints, radius 2, 2048 bits; library defaults; one model per target.
• ChemProp + CheMeleon — chemprop 2.3.0 D-MPNN with message passing initialised from the CheMeleon foundation model, then fine-tuned. 50 epochs, batch 64, one network per fold, no ensembling.
• Monroe + TabPFN 3.5 — the authors' pre-trained encoder, frozen. Every molecule becomes one 720-d embedding, computed once for all 13,257 distinct structures, and TabPFN 3.5 predicts each target in context from that fold's training embeddings. Nothing is fitted by gradient descent.
• Training median — the paper's naive baseline, drawn as a line rather than a bar because it is not a competitor.
• The paper's six arms — k-NN, SVM, RF, XGBoost, MLP and a GNN, entered at their published per-fold metrics rather than re-run, because predictions were not released. They are marked with an asterisk everywhere. Each was given 50 Optuna trials per fold while the four arms above run at their libraries' defaults — an asymmetry that favours the published arms and is left uncorrected.

The statistics are the part that differs most from the paper. Each scheme gives fifty folds per arm, ten targets by five folds, and the between-target spread is an order of magnitude larger than the differences between methods. So the primary test is paired on the fold, where the target effect cancels, with Holm's correction applied within each scheme and metric across every pair on the page. Three tests are computed and stored for every pair: the paired t-test over the fifty folds, a Wilcoxon signed-rank test that drops the normality assumption, and a target-level t-test that drops the independence assumption instead, by averaging each method's five folds within a target and pairing on the ten targets. The last one is there because the time split gives every fold its own year cutoff, so a target's five test sets overlap and fifty differences computed on overlapping molecules are not fifty independent observations. It costs power, so it is a check rather than the headline. The Tukey figure is drawn on target-centred values, the standard randomized-block adjustment, which leaves every arm's mean exactly where it was and only narrows the intervals. The raw pooled Tukey is drawn beside it because the difference between the two is part of the argument.

Every script, both copies of the splits, and all predictions are at model-validation-central/studies/split-fragility. The figures and tables on this page rebuild from the stored predictions in under a minute, with no retraining. test_split_stats.py checks the statistics against synthetic data with a known answer.

Statistics follow the approach in “Even More Thoughts on ML Method Comparisons” and the protocol paper it points at: distributions of fold-level metrics, corrections for multiple comparisons, and paired tests using the folds as the pairing.

References

1. Lee, K.; Moldagulov, G.; Grzybowski, B. A. The Fragility of Bioactivity Prediction: Rigorous Dataset Splits Expose the Illusion of ML Accuracy. Chem. Eur. J. 2026, e71208. doi:10.1002/chem.71208
2. Janela, T.; Bajorath, J. Simple nearest-neighbour analysis meets the accuracy of compound potency predictions using complex machine learning models. Nat. Mach. Intell. 2022, 4, 1246-1255. doi:10.1038/s42256-022-00581-6
3. Burns, J. W.; Zalte, A. S.; Abreu, C. R. A.; et al. Deep Learning Foundation Models for Low-Data Regimes from Classical Molecular Descriptors. J. Chem. Inf. Model. 2026. doi:10.1021/acs.jcim.6c01546
4. Banaszewski, B.; Fitzgibbon, A. W. Monroe: A Molecular Foundation Model for In-Context Probabilistic Inference. Preprint, 2026. arXiv:2608.18982
5. Heid, E.; Greenman, K. P.; Chung, Y.; et al. Chemprop: A Machine Learning Package for Chemical Property Prediction. J. Chem. Inf. Model. 2024, 64, 9-17. doi:10.1021/acs.jcim.3c01250
6. Hollmann, N.; Müller, S.; Purucker, L.; et al. Accurate predictions on small data with a tabular foundation model. Nature 2025, 637, 319-326. doi:10.1038/s41586-024-08328-6
7. Walters, P. Even More Thoughts on ML Method Comparisons. Practical Cheminformatics, 2025. practicalcheminformatics.blogspot.com
8. Ash, J. R.; Wognum, C.; Rodríguez-Pérez, R.; et al. Practically significant method comparison protocols for machine learning in small molecule drug discovery. Preprint, 2025. doi:10.26434/chemrxiv-2025-l0h2b