3D quantum-property pre-training · 5×5 cross validation · two data sets

Pre-training a 3D equivariant transformer on quantum mechanics, against the ADMET field

QuADMET-Former¹ pre-trains a 1.3-million-parameter, six-layer TorchMD-ET equivariant transformer² on PubChemQC quantum mechanical properties — total energy, HOMO–LUMO gap, dipole moment, 3D E3FP fingerprints, and per-atom Löwdin charges — across ten released single- and three-conformer checkpoints, then adapts it to ADMET endpoints by two-stage linear probing and fine-tuning (LP-FT). Its paper reports that quantum pre-training consistently beats training the same 3D architecture from scratch on TDC and Biogen ADME benchmarks. Run here on the same fifteen ExpansionRx and Biogen endpoints, 3D conformers, 5×5 cross-validation folds and held-out test sets as the rest of this site, that internal claim holds: pre-trained LP-FT beats from-scratch training on 14 of 15 endpoints (7 significant wins to 0 losses) by a median +0.083 R², beats its own frozen TabPFN 3 arm on 31 of 45 endpoint × metric combinations (losing on 12), and beats LightGBM + Morgan on 38 of 45 combinations (losing on three). Against the modern pre-trained field, fine-tuned QuADMET-Former reaches the top Tukey tier on 5 of 27 combinations on ExpansionRx (leading or tying on LogS, LOG_MPPB, and LOG_MBPB) and nearly splits ExpansionRx with fine-tuned CheMeleon⁶ (9 wins, 8 ties, 10 losses; 15–9–21 across both data sets), while trailing the two overall leaders: fine-tuned Atom-JEPA³ (2 wins, 8 no call, 35 losses) and Monroe + TabPFN 3⁴ (10 wins, 2 no call, 33 losses).

The scoreboard

Tukey HSD per endpoint and metric, counted the same way across every study on this site: a method is on top if Tukey's simultaneous correction cannot separate it from the best mean at α = 0.05, and it is only best alone if no other arm shares the top group. The primary tables show MAE in the log10(x + 1) units of each endpoint, followed by R².

ExpansionRx⁹

Biogen ADME¹⁰

Across the two data sets, the leaderboard is led by the two heavyweights from the Atom-JEPA study: on ExpansionRx fine-tuned Atom-JEPA is on top of 19 of 27 endpoint × metric combinations and Monroe + TabPFN 3 of 9, while fine-tuned QuADMET-Former reaches the top Tukey tier on 5 of 27 combinations — posting the highest mean Spearman ρ on LogS (0.616, tied with Atom-JEPA FT), the best mean R² (0.656) and MAE (0.182) on LOG_MPPB (tied with Atom-JEPA FT), and tying for best Spearman ρ on LOG_MPPB (0.819) and LOG_MBPB (0.879). On Biogen ADME, Monroe is on top of 18 of 18 combinations and Atom-JEPA FT of 5.

Head-to-head across all 45 endpoint × metric calls

Paired by fold across all fifteen endpoints and three metrics (45 combinations), two-stage multitask LP-FT (quadmet_ft) substantially outperforms reading the frozen 128-D QM embedding in context with TabPFN 3 (quadmet_tabpfn), winning 31 of 45 combinations, tying 2, and losing 12 (17–0–10 on ExpansionRx and 14–2–2 on Biogen). Against the classical LightGBM + Morgan floor, fine-tuned QuADMET-Former wins 38 of 45 combinations (4 no call, 3 losses, going 17–1–0 on Biogen and 21–3–3 on ExpansionRx), while frozen QuADMET-Former + TabPFN 3 wins 29 (3 no call, 13 losses). Against fine-tuned CheMeleon, fine-tuned QuADMET-Former wins 15, ties 9, and loses 21 — nearly splitting ExpansionRx (9 wins, 8 ties, 10 losses) and winning all six calls on Biogen LOG_HPPB and LOG_RPPB. Against Monroe + TabPFN 3, fine-tuned QuADMET-Former wins 10 combinations on ExpansionRx (LogS, LOG_MPPB, LOG_MBPB, and LOG_MGMB) while losing 33 across both sets; against the 30× larger fine-tuned Atom-JEPA, it wins 2 (Spearman ρ on ExpansionRx LogS and Biogen LOG_RPPB), ties 8 (including all three metrics on LOG_MPPB and Spearman ρ on LOG_HPPB), and loses 35.

Choosing among the ten released QM checkpoints

QuADMET-Former releases ten checkpoints spanning five subsets of quantum mechanical pre-training targets — Löwdin atomic charges alone ([Lw]), 3D E3FP fingerprints alone ([3D]), the three molecular energy/dipole properties ([DM,HL,TE]), those three plus Löwdin charges ([Lw,DM,HL,TE]), and all five together ([All]) — each trained with either a single conformer or three conformers per molecule. Scoring all ten on each fold's held-out validation fifth shows two consistent patterns.

First, the spread from the best checkpoint to the worst is large — a median of 0.137 R² on ExpansionRx and 0.177 R² on Biogen — because checkpoints pre-trained on a single target ([Lw] or [3D] alone) generally lag the multi-property checkpoints ([Lw,DM,HL,TE] and [All]). The one striking exception is ExpansionRx LogD, where the single-conformer Löwdin atomic-charge checkpoint (Single [Lw]) ranks first on the validation fifth (0.551 R²), consistent with lipophilicity being governed by local atomic partial charges and polarity. Second, among the top multi-property checkpoints the runner-up is almost always within fold noise: across ExpansionRx (5 distinct winners) the median margin over the runner-up is 0.026 R² (0 of 9 exceeding one fold SD), and across Biogen (4 distinct winners) the median margin is 0.023 R² (1 of 6 exceeding one fold SD, on LOG_RLM).

ExpansionRx checkpoint selection

Biogen ADME checkpoint selection

Decomposing representation and fine-tuning

Just as in the Atom-JEPA study, the frozen TabPFN 3 arm splits the difference between Monroe + TabPFN 3 and fine-tuned QuADMET-Former into two steps: first swapping Monroe's embedding for QuADMET-Former's 128-D graph embedding under the exact same TabPFN 3 call, and second replacing TabPFN 3 with two-stage LP-FT.

Swapping Monroe's representation for QuADMET-Former's frozen embedding under TabPFN 3 reduces R² on 13 of 15 endpoints, by a median −0.130 R² on ExpansionRx (where frozen QuADMET-Former + TabPFN 3 already edges out Monroe on LOG_MPPB and LOG_MBPB) and −0.195 R² on Biogen. Replacing TabPFN 3 with two-stage multitask LP-FT across all six equivariant layers recovers substantial ground, lifting median R² by +0.105 on ExpansionRx and +0.024 on Biogen — with the largest fine-tuning gains on the binding and physicochemical endpoints (LOG_MPPB +0.276 R², LOG_MGMB +0.242 R², LOG_RPPB +0.219 R², LogD +0.218 R², LOG_HPPB +0.215 R², and LogS +0.204 R²).

Across the fifteen endpoints the Spearman rank correlation between training-set size and fine-tuned QuADMET-Former's R² margin over Monroe is −0.13.

Testing the paper's internal claims — and auditing the released code

Why do the paper's internal ablations favour QuADMET-Former while the external leaderboard still favours Atom-JEPA and Monroe? Running four controlled internal ablations alongside a line-by-line audit of github.com/arunraja-hub/quadmetformer answers both halves of that question.

1. Quantum pre-training and LP-FT genuinely beat training TorchMD-ET from scratch. The paper's central hypothesis is internal: that pre-training TorchMD-ET on QM properties and adapting via two-stage LP-FT improves over training the same 3D architecture from random initialisation (scratch) or freezing it with a linear probe (lp). Our ablation confirms both claims directly: across the 15 endpoints, pre-trained LP-FT beats from-scratch training of the exact same multitask 6-layer architecture on 14 of 15 endpoints (7 significant wins to 0 losses) by a median +0.083 R², and Stage-2 fine-tuning improves over Stage-1 linear probing on all 15 of 15 endpoints by a median +0.348 R². The paper did not benchmark against 2D pre-trained MPNNs (CheMeleon), in-context foundation models (Monroe + TabPFN 3), or 30× larger 3D self-supervised encoders (Atom-JEPA); its baseline was its own 3D architecture trained from scratch.

2. Intermediate versus final layers under an MLP probe, and why full-backbone multitask LP-FT matters. All ten released checkpoints contain 6 equivariant attention layers (1,312,268 parameters), matching Section 3.3 of the paper, yet the repository's finetuning/downstream_train.py (line 289) restricts Optuna to num_layers ∈ {1, 2, 3}. Comparing a frozen 6-layer single-task MLP probe against a truncated 3-layer MLP probe shows why: under a frozen single-task MLP head, intermediate layer 3 edges out the final QM-specialised layer 6 on 12 of 15 endpoints (median Δ 6L−3L of −0.039 R²). Moreover, single-task unstandardized L1 fine-tuning or freezing the lower layers suffers severe test-set miscalibration on sparse/shifted endpoints (the same gap seen between single-task chemprop_st and multitask chemprop), whereas unfreezing all 6 layers (n_frozen = 0) in multitask LP-FT lifts every endpoint well above both the 3-layer and 6-layer linear probes.

What it costs

Because TorchMD-ET has 1.3 million parameters (one thirtieth the size of Atom-JEPA's 39M-parameter EquiformerV3) and static 5.0 Å radius graphs can be pre-computed once per molecule on GPU, evaluating all ten checkpoints across all 11,129 molecules takes under two minutes, and two-stage LP-FT in bfloat16 takes only seconds per fold across the two GPUs (apollo, RTX 5070 Ti; spark, DGX Spark GB10).

Reported the way “Even More Thoughts on ML Method Comparisons” and the protocol paper it points at¹¹ argue comparisons should be: distributions over folds, simultaneous confidence intervals, paired tests on the folds, and no bolded maxima. Where two methods cannot be separated, both are reported as tied.

Code, per-fold metrics and every figure: studies/quadmet-former.

References

1. Raja, A.; et al. QuADMET-Former: Pre-training on Quantum Mechanical Properties Improves ADMET Prediction. ChemRxiv 2026, doi:10.26434/chemrxiv.15002429/v3. The method benchmarked here, run with the authors' TorchMD-ET implementation (github.com/arunraja-hub/quadmetformer) and their ten released PubChemQC checkpoints. 10.26434/chemrxiv.15002429/v3
2. Thölke, P.; De Fabritiis, G. TorchMD-NET: Equivariant Transformers for Neural Network Based Molecular Potentials. ICLR 2022. The SE(3)-equivariant transformer backbone underlying QuADMET-Former. arXiv:2202.02541
3. Petersen, K. H.; Tirsgaard, R. H.; Cornet, F. R. J.; Jordahn, M.; Schmidt, M. N. Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems. Preprint, 2026. The 39M-parameter 3D self-supervised comparator fine-tuned on the same ten-conformer caches. arXiv:2610.08400
4. Banaszewski, B.; Fitzgibbon, A. W. Monroe: A Molecular Foundation Model for In-Context Probabilistic Inference. Preprint, 2026. The frozen encoder in the strongest in-context arm. arXiv:2608.18982
5. Grinsztajn, L.; Flöge, K.; Key, O.; et al. TabPFN-3: Technical Report. Preprint, 2026. The head of the Monroe arm and of the frozen QuADMET-Former arm. arXiv:2605.13986
6. Burns, J. W.; Zalte, A. S.; Abreu, C. R. A.; et al. Deep Learning Foundation Models for Low-Data Regimes from Classical Molecular Descriptors. J. Chem. Inf. Model. 2026, articles ASAP. The pre-trained 2D D-MPNN comparator. 10.1021/acs.jcim.6c01546
7. Graff, D. E.; Morgan, N. K.; Burns, J. W.; et al. Chemprop v2: An Efficient, Modular Machine Learning Package for Chemical Property Prediction. J. Chem. Inf. Model. 2026, 66 (1), 28–33. 10.1021/acs.jcim.5c02332
8. Rogers, D.; Hahn, M. Extended-Connectivity Fingerprints. J. Chem. Inf. Model. 2010, 50 (5), 742–754. The LightGBM baseline's representation. 10.1021/ci100050t
9. OpenADMET; Expansion Therapeutics. OpenADMET–ExpansionRx Blind Challenge Data. Hugging Face, 2026, CC BY 4.0. 10.57967/hf/9687
10. Fang, C.; Wang, Y.; Grater, R.; et al. Prospective Validation of Machine Learning Algorithms for Absorption, Distribution, Metabolism, and Excretion Prediction: An Industrial Perspective. J. Chem. Inf. Model. 2023, 63 (11), 3263–3274. 10.1021/acs.jcim.3c00160
11. Ash, J. R.; Wognum, C.; Rodríguez-Pérez, R.; et al. Practically Significant Method Comparison Protocols for Machine Learning in Small Molecule Drug Discovery. J. Chem. Inf. Model. 2025, 65 (18), 9398–9411. 10.1021/acs.jcim.5c01609