5×5 cross validation · two data sets

A graph network taught to predict fingerprints, against the fingerprints

PT-GIN¹ is a graph network pre-trained to reproduce a molecule’s own Morgan fingerprint, then frozen and handed to LightGBM. Its authors report it beating hashed ECFP on five of six Biogen ADME tasks. Run here against those same fingerprints through that same predictor — fifteen endpoints, two data sets, 25 folds each — it wins six endpoints and loses six, and on Biogen, its own benchmark, it wins one and loses three. Which six it wins is not random, and it is not about how much data an endpoint has. PT-GIN improves on the fingerprints precisely where the fingerprints were already doing badly, and degrades them where they were doing well.

What PT-GIN is

Every atom in a molecule is given one token per circular substructure radius, drawn from a vocabulary that Sort & Slice² builds by ranking substructures on how many pre-training molecules they appear in and keeping the most common. A Graph Isomorphism Network³ reads those tokens and is trained, on 462,189 QMugs molecules⁴, to answer 2,048 yes-or-no questions about the molecule it is looking at: which bits of that molecule's own hashed ECFP4⁵ are set.

That last detail is what makes this method unusually easy to judge. PT-GIN and the LightGBM baseline are the same pipeline with different inputs. One is handed 2,048 Morgan counts, the other 2,048 to 3,072 numbers from a frozen network, and both hand them to LightGBM at library defaults. Whatever separates them is the representation, and nothing else.

Which checkpoint?

The paper does not have one PT-GIN. It pre-trains a grid of maximum substructure radius by vocabulary size, and for each task it picks whichever pre-trained model does best in downstream tuning. Ten of those checkpoints are released, so the choice is part of the method and gets reproduced — on the validation fifth of each fold, which is the only place this protocol allows a choice to be made. Every checkpoint is fit on the four fifths of all 25 folds and scored on the held-out fifth; the best mean validation R² wins the endpoint. The test set plays no part in it. That is 3,750 LightGBM fits to choose 15 models.

The first thing the sweep says is that the grid barely exists. Across all nine ExpansionRx endpoints the ten checkpoints span a median of 0.027 R², and the winner beats the runner-up by a median of 0.002. Set that against the fold-to-fold standard deviation of the winner itself — 0.037 to 0.287 — and the comparison is not close: not one of the fifteen endpoints across both data sets has a margin as large as a single fold’s worth of noise.

Five different checkpoints get chosen here and four on Biogen, which looks like the per-task variation the method is built around. It is not. It is what picking the maximum of ten near-identical numbers looks like. A radius-2 network with a 1,024-token vocabulary takes five of the nine ExpansionRx endpoints; a radius-0 network — one token per atom type, no circular environment at all — takes LogD here and three of the six Biogen endpoints. That a tokenisation carrying no substructure information beyond the atom finishes first as often as it does is the clearest evidence that the choice is not carrying signal.

The one endpoint with a real spread, LOG_MGMB at 0.184, is the smallest in either data set at 431 measurements, and its fold standard deviation is 0.287. That is not a grid with structure in it. That is a small endpoint being noisy in ten directions at once.

How often is each method best?

Tukey HSD¹⁰ across the 25 folds, counted over every endpoint × metric combination. Tied means the method could not be distinguished from the leader at α = 0.05.

PT-GIN is on top of six of the 27 combinations and best alone on none of them. CheMeleon is on top of 24, and LightGBM on none. But the tally flattens the only part of this that is interesting, because PT-GIN’s six are not scattered at random across the endpoints.

R² by endpoint

Spearman ρ by endpoint

Against the fingerprints it was trained to predict

This is the paper's central claim under this protocol, and the cleanest test on the page: same predictor, same molecules, same folds, one representation swapped for another. Folds are paired.

Not a sweep in either direction. PT-GIN takes five endpoints, loses three, and cannot be separated on one. The paper’s claim, that pre-training on fingerprints beats the fingerprints, half survives.

Which half is the result, and the obvious explanation is the wrong one. It is not about how much data an endpoint carries: across all fifteen endpoints on both data sets, the PT-GIN advantage has a rank correlation of −0.01 with the number of measurements (p = 0.98). LOG_MLM has 5,692 measurements and PT-GIN wins 23 folds of 25; LOG_MBPB has 1,426 and it loses.

What predicts it is how well the fingerprints were already doing. Sort the fifteen endpoints by the LightGBM baseline’s R² and the advantage runs downhill: Spearman ρ = −0.65 (p = 0.009) over both data sets, and −0.73 (p = 0.025) on ExpansionRx alone. The crossover sits near a baseline R² of 0.3. Caco-2 efflux ratio, where LightGBM scores −0.135, gains 0.136 and PT-GIN wins all 25 folds. LogD, where LightGBM reaches 0.510, loses 0.091 and PT-GIN wins none of 25. LogS, the next strongest, also none of 25.

That is what a smoothed fingerprint looks like. The embedding was trained to reconstruct ECFP4 and cannot contain substructure information ECFP4 does not already have; what it has instead is 2,048 to 3,072 dense, correlated coordinates in place of 2,048 sparse counts. Where the signal is weak enough that variance dominates, that is the better-behaved input and it wins. Where the counts are already carrying a learnable signal, the smoothing is spending resolution the model needed. The one clear exception is LOG_MGMB, which gains 0.162 from a baseline of 0.260 — and which is the smallest endpoint in either data set, at 431 measurements, with a fold standard deviation of 0.29.

Against a graph network with no pre-training

The other half of the claim. ChemProp single-task⁷ is a message-passing network trained from scratch on each endpoint — a different architecture from a GIN, but the same bet that a learned molecular representation beats a fixed one, without the pre-training.

Here the pre-training does earn its keep, on six endpoints against two, and by margins that are not small: +0.405 R² on Caco-2 A→B, where a from-scratch D-MPNN scores −0.307 and does worse than predicting the training mean.

The same rule fits. The two endpoints PT-GIN loses are LogD and LogS, 0 folds out of 25 apiece — and those are exactly the two endpoints where a from-scratch D-MPNN is strongest, at R² 0.700 and 0.447. A different architecture, a different comparator, and the boundary lands in the same place: PT-GIN pulls a weak model up and drags a strong one down.

Does it replicate?

Everything above is ExpansionRx. The Biogen public ADME set⁹ is the paper's own primary benchmark, and the one it reports winning five of six tasks on. It is not the same experiment: the authors use 200 repeats of 5-fold Butina-clustered cross validation with no fixed holdout, where this project holds out whole BitBIRCH clusters once and cross-validates within what remains. Absolute numbers do not carry across the two designs. Rankings within each do.

It also matters that these released checkpoints were filtered against this data set. Any QMugs molecule within Tanimoto 0.5 of a Biogen compound was excluded from pre-training. ExpansionRx was not one of the paper's benchmarks and got no such filter, so if anything the ExpansionRx result above is the one with more opportunity to be flattered.

What it chose here

Against the fingerprints, again

It does not replicate, and this is the data set the paper reports winning five of six tasks on. Against LightGBM, PT-GIN wins one endpoint, loses three, and cannot be separated on two. Against CheMeleon it loses all six endpoints on R² and all six on Spearman, and five of six on MAE. The single combination of 18 where it is on top, it is on top of by being indistinguishable rather than better.

The rule from ExpansionRx transfers, though less sharply, and the clearest instance is against a from-scratch D-MPNN rather than against the fingerprints. The two plasma protein binding endpoints have 128 and 109 training molecules, and single-task ChemProp scores R² of −0.064 and −0.005 on them — worse than predicting the training mean. They are the only two of the six where PT-GIN beats it, by +0.224 and +0.072, winning 25 folds of 25 on human PPB. On the four endpoints where that method actually works, PT-GIN loses to it every time.

One explanation can be ruled out. These checkpoints were pre-trained on QMugs filtered at Tanimoto 0.5 against Biogen, so a Biogen-like molecule was excluded by construction — but ExpansionRx was never one of the paper’s benchmarks and got no such filter, which would let the ExpansionRx result be the flattered one. It is not. Neither test set has a single exact or connectivity-block InChIKey match in the 462,189-molecule corpus, and the median nearest-neighbour Tanimoto is 0.372 on ExpansionRx against 0.431 on Biogen. The unfiltered data set is the less overlapping of the two.

What this does and does not say

Two things are worth carrying away, and only one of them is about PT-GIN.

The first is that self-supervision on a fingerprint gives you a fingerprint. Every result above is consistent with the embedding being a smoothed, denser ECFP4 and nothing more: its advantage over the sparse counts is predicted by how badly those counts were doing and by nothing else, it rescues a graph network wherever that network is starved, it drags one down wherever it is not, and it never once beats a graph network initialised from a foundation model that was pre-trained on something other than a fingerprint. The ceiling of the pre-training target is the ceiling of the representation, which is a general point about choosing a self-supervised objective rather than a criticism of this one.

The second is a warning about validation splits that this arm gives for free. On eight of the nine ExpansionRx endpoints PT-GIN’s validation R² runs 0.11 to 0.44 above the test R² it then achieves — 0.720 against 0.420 on LogD. On Biogen’s four well-populated endpoints the same gap is −0.01 to +0.03. Nothing here is selecting on validation in a way that could inflate it: the number is one fixed model’s honest score on a held-out fifth. The gap is a property of the ExpansionRx split, which shipped with the challenge and is not cluster-pure, against the Biogen split, which was built by holding out whole BitBIRCH clusters exactly as the folds are. Any method that tunes anything on the first of those is being told a comfortable lie.

Reported the way “Even More Thoughts on ML Method Comparisons” argues comparisons should be: distributions over folds, simultaneous confidence intervals, paired tests, and no bolded maxima. Every method saw identical training molecules in every fold.

References

1. Money-Kyrle, S.; Dablander, M.; Hanser, T.; Werner, S.; Deane, C. M.; Morris, G. M. On Improving Graph Neural Networks for QSAR by Pre-training on Extended-Connectivity Fingerprints. arXiv 2026, 2605.10722. The method benchmarked here, run from the authors' own code and released checkpoints. arXiv:2605.10722
2. Dablander, M.; Hanser, T.; Lambiotte, R.; Morris, G. M. Sort & Slice: A Simple and Superior Alternative to Hash-Based Folding for Extended-Connectivity Fingerprints. J. Cheminform. 2024, 16, 135. How PT-GIN builds its atom-token vocabulary. 10.1186/s13321-024-00932-y
3. Xu, K.; Hu, W.; Leskovec, J.; Jegelka, S. How Powerful are Graph Neural Networks? ICLR 2019. The architecture. ICLR 2019
4. Isert, C.; Atz, K.; Jiménez-Luna, J.; Schneider, G. QMugs, Quantum Mechanical Properties of Drug-like Molecules. Sci. Data 2022, 9, 273. The pre-training corpus, 665k molecules before the similarity filter and 462,189 after it. 10.1038/s41597-022-01390-7
5. Rogers, D.; Hahn, M. Extended-Connectivity Fingerprints. J. Chem. Inf. Model. 2010, 50 (5), 742–754. The pre-training target, and the baseline it is put against. 10.1021/ci100050t
6. Ke, G.; Meng, Q.; Finley, T.; et al. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. NeurIPS 2017. The downstream predictor for both PT-GIN and the fingerprint baseline. NeurIPS 2017
7. Graff, D. E.; Morgan, N. K.; Burns, J. W.; et al. Chemprop v2: An Efficient, Modular Machine Learning Package for Chemical Property Prediction. J. Chem. Inf. Model. 2026, 66 (1), 28–33. 10.1021/acs.jcim.5c02332
8. Burns, J. W.; Zalte, A. S.; Abreu, C. R. A.; et al. Deep Learning Foundation Models for Low-Data Regimes from Classical Molecular Descriptors. J. Chem. Inf. Model. 2026, articles ASAP. 10.1021/acs.jcim.6c01546
9. Fang, C.; Wang, Y.; Grater, R.; et al. Prospective Validation of Machine Learning Algorithms for Absorption, Distribution, Metabolism, and Excretion Prediction. J. Chem. Inf. Model. 2023, 63 (11), 3263–3274. The second data set, and the paper's own primary benchmark. 10.1021/acs.jcim.3c00160
10. Tukey, J. W. Comparing Individual Means in the Analysis of Variance. Biometrics 1949, 5 (2), 99–114. 10.2307/3001913