Total posts: 87

Generated on 2026-04-11.

The Hype Cycle and the Reality of Virtual Screening

Working on machine learning in drug discovery can be profoundly frustrating. Every week, the purveyors of social media hype herald a new “revolution” in AI for the field. As someone deeply invested in this space, I often follow the trail to these revolutionary papers, only to find researchers repeating the same fundamental mistakes that have plagued the discipline for more than a decade.

Much of the recent hype has centered on virtual screening. Some recent papers have touted approaches that purportedly enable screening of the entire proteome in mere hours. These results appear impressive—until you realize they are validated against datasets that were effectively discredited seven years ago. A primary offender, and one I will focus on here, is the DUD-E dataset. Originally developed to validate docking methods, it has been frequently (and incorrectly) used to validate machine learning models.

What is the DUD-E Dataset?

In the early days of virtual screening (the late 1980s and early 1990s), researchers validated their methods by “seeding” known active compounds among inactive decoy structures. The goal was to determine whether a virtual screening method—typically docking—could successfully separate the two. Although the logic appeared sound, many early test cases were unduly easy because of inherent physical differences between the actives and the decoys.

Consider the first-generation HIV protease inhibitors. These were large molecules; Saquinavir has a molecular weight of 671, and Amprenavir is 506. If these structures are “hidden” among a set of decoys with molecular weights in the 350–450 range, any algorithm—especially docking, which is biased to reward larger molecules with higher scores—can “cheat” by simply picking out the biggest structures.

To address this bias, John Irwin, Brian Shoichet, and their colleagues developed the Directory of Useful Decoys (DUD). They collected active compounds and designed decoy sets whose physical properties matched those of the actives. By aligning these distributions, the authors hoped to eliminate size and property bias from the validation process. In addition to molecular weight, they matched lipophilicity (LogP), charge, and the number of hydrogen-bond donors and acceptors. This collection evolved into DUD-E, an enhanced collection of activities and decoys that incorporated topological similarity and property distributions when constructing validation sets.

For example, consider the AKT1 active and decoy subset of the DUD-E dataset. The figure below shows the distributions of five key physical properties for active and decoy compounds. The sets have nearly identical molecular weight and polar surface area distributions. The differences in the LogP distributions are likely due to my use of a different LogP implementation than the one used to generate the DUD-E set.

The Fatal Flaw: Chemical Dissimilarity

If the physical properties are matched, what’s the problem? While DUD-E balances bulk properties like molecular weight and LogP among the actives and decoys, the decoys were intentionally designed to be structurally distinct from the actives. This was done to ensure that the decoys were truly inactive. While this makes DUD-E an excellent benchmark for docking, it makes it a poor choice for machine learning.

In fact, the DUD-E authors point this out explicitly in their paper: “Fundamentally, DUD and DUD-E are designed to measure value-added screening performance of 3-D methods over simple 1-D molecular properties. Decoys that might bind are removed using 2-D ligand similarity, so DUD-E is inappropriate to test 2-D methods.”

To visualize why the DUD-E set is inappropriate for ML, I used t-SNE (t-distributed Stochastic Neighbor Embedding) with RDKit Morgan Fingerprints to map the chemical diversity across six DUD-E datasets. In this projection, each point represents a molecule; similar structures cluster together, while dissimilar ones are pushed apart.

The result is striking. In the maps above, the active compounds (red) form tight, isolated clusters, while the decoys (light blue) are scattered elsewhere. This reveals the “analog bias” inherent in the dataset. Because the actives are often structurally related to one another but completely distinct from the decoys, a machine learning model doesn’t actually have to learn anything about protein-ligand interactions. It only has to learn how to recognize the “red” clusters.

It doesn’t take a sophisticated deep-learning architecture to “solve” DUD-E. In the next section, I’ll show how easily a simple, baseline ML model can identify the actives with near-perfect accuracy—further proving that we aren’t measuring model performance but rather a flaw in the data.

This Is Not News

What I’m pointing out here is not new information. Many of the issues with the DUD-E dataset and ML models were reported in a 2019 paper by Tom Kurtzman, David Koes, and colleagues. In this paper, the authors evaluated the ability of Convolutional Neural Networks (CNNs) to identify active compounds in the DUD-E datasets.

Kurtzman and coworkers found that their models were equally predictive when they “turned off” the protein representation. In fact, there was a near-perfect correlation between the affinities calculated with and without the protein. Several additional experiments showed that the CNN wasn’t learning from the protein and was instead simply learning to distinguish active molecules from decoys. They demonstrated that one could train a model on complexes from a different protein and still distinguish actives from decoys.

A Simple Demonstration

To illustrate the flawed nature of the DUD-E dataset for ML model validation, we will train a model on each of the 28 kinase datasets in the DUD-E set and examine the ability of that model to separate the actives from the decoys in the other kinase datasets.

Before we begin, we should address the elephant in the room. Many will point out that since these targets are all kinases, this is an easy task. When building models to predict protein-ligand binding affinities, most groups use a 90% overall sequence similarity cutoff to separate their training and test sets. I’m not arguing this is the right way to do it; I’m simply pointing out what people do. As shown in the box plot below, according to conventional criteria, none of these targets would be considered “similar.” In fact, the only pair approaching the 90% cutoff is AKT-1 and AKT-2.

With the obvious out of the way, let’s build some models. I employed a simple strategy that has become my initial “go to” for ML model building: I generated 2048-bit Morgan fingerprints with radius 2 for the molecules in the 23 DUD-E kinase datasets. I then used these fingerprints to train a LightGBM classification model using each target as a training set and predicting on all of the other datasets.

The results are shown in the figure below. We have the 28 DUD-E kinase datasets on the horizontal axis and the AUC (Area Under the Receiver Operating Characteristic) on the vertical axis. We observe that, in most cases, a model trained on data for a single kinase can readily distinguish active molecules from decoys across all other kinases.

To push this silly test even further, we can examine whether a model trained on kinase data can separate active and decoy molecules from the DUD-E nuclear receptor targets. As shown in the plot below, the models trained on kinase data showed weak predictive ability when tested on nuclear receptor actives and decoys, but did not perform as well as when tested on kinase targets. Given the lack of similarity between kinase inhibitors and molecules that bind to nuclear receptors, this isn’t unexpected.

Some Closing Thoughts

While reading recent papers, I was surprised to find that they used the DUD-E dataset as a validation set. As I noted above, many authors have pointed out that this dataset is inappropriate for ML. If we have a model trained on thousands of datapoints from ChEMBL and other databases, tasks such as classifying molecules from the DUD-E dataset become trivial.

How did we get here? How do we have papers published in high-profile journals that make bold claims based on incredibly weak evidence? I don’t believe that people are being intentionally dishonest; they are simply failing to familiarize themselves with prior work done in the field. This is a prime example of those who do not study history being doomed to repeat it. As new groups come into drug discovery from computer science and related fields, it’s important for them to read the literature and familiarize themselves with work that came before.

Unfortunately, our field is littered with poorly constructed benchmarks that later become “standards” for comparison. I believe that journals need to assume greater responsibility for what they publish by defining clear standards and criteria and by using editors and reviewers with deep expertise in the field.

One final note to editors and reviewers who might be reading this: I commend journals for establishing guidelines that require including source code with a paper. However, when I look at the code repository for a recently published paper and see that the last commit was more than two years ago, the chances that I’ll be able to run that code are essentially zero. In the fast-moving world of machine learning, two years is an eon. It’s always good to make an effort to ensure the code is up to date and runnable. The same goes for authors. You’re releasing your code so that people can use it and build upon it. Make that easy to do.

If anyone is interested in reproducing or extending this work, all code is available on GitHub..

Over the last few months, I’ve been using marimo to create summary and detail views that help me quickly explore datasets and understand SAR. I’ve also created a GitHub repository with a few examples of how marimo can be used to create interactive cheminformatics tools. For those who haven’t read my previous post, marimo can be thought of as a “better Jupyter notebook”. It has all the features of Jupyter, plus several additions that make it easy to integrate interactivity into the notebook. There are three key features that distinguish marimo from Jupyter.

1. Reactive Execution Model

In Jupyter, cells are executed sequentially and manually; if you change a variable in “Cell A,” you must re-run “Cell B” to see the updated result. This often leads to “hidden state” bugs.

Marimo uses a directed acyclic graph (DAG) to track dependencies between cells.

• Automatic Updates: If you modify a variable in one cell, marimo automatically re-runs all other cells that reference that variable.
• Consistency: This ensures that the code you see on the screen always matches the output displayed, eliminating the “out-of-order execution” problem common in Jupyter.

2. Pure Python File Format (.py vs .ipynb)

Jupyter notebooks are stored as JSON files (.ipynb), which include code, metadata, and raw output (like large image strings). This makes them notoriously difficult to manage with version control.

Marimo notebooks are stored as pure Python files (.py).

• Git-Friendly: Because they are plain text, you can use standard git diff to see exactly what code changed without wading through messy JSON.
• Execution as Scripts: You can run a marimo notebook directly from your terminal using python notebook.py. It treats the notebook as a legitimate Python script, making it easier to integrate into production pipelines.

3. Native UI Interactivity (No Callbacks)

While Jupyter requires external libraries (like ipywidgets) and complex callback functions to create interactive sliders or buttons, marimo builds this directly into the Python-variable relationship.

• Variables as Widgets: In marimo, UI elements like sliders or dropdowns are bound directly to Python variables.
• Instant Apps: Moving a slider automatically triggers the reactive engine to update any dependent code. This allows you to turn a notebook into a functional web app or dashboard (using marimo run) without writing any specialized “app” code or using frameworks like Streamlit.

For more information on marimo, I recommend checking out these resources.

· The marimo website has a ton of great information and tutorials.
· The marimo YouTube channel has dozens of great videos where Vincent does everything from providing an overview to deep dives on specific features.

In this post, I’ll provide a quick overview of three simple but effective marimo notebooks I’ve recently created. Hopefully, this will give a sense of what you can do with marimo. In addition, I’ll highlight a few useful features in marimo-chem-utils, a pip-installable, open-source package I built to add cheminformatics features to marimo. The GitHub repository for marimo-chem-utils also includes links to demo versions of the notebooks that run on molab, marimo’s answer to Google Colab. You can run the notebooks on molab without installing any software on your computer. However, to run the notebooks in interactive mode, you must be logged in to molab. This is easy and just requires setting up a free account.

Viewing Clustering Results

Clustering provides an efficient way to organize sets of chemical structures. Although several useful clustering algorithms, such as Butina and BitBirch, are available as open-source software, I haven’t found many tools for visualizing clustering results. Fortunately, it’s easy to integrate this capability into a marimo notebook. In the examples directory of the marimo-chem-utils repository, there is a file called cluster.py that demonstrates how marimo can provide an overview/detail view of clustering results. The figure below shows the core of the interface. On the left is an overview table that shows one representative from each cluster. Clicking the checkbox to the left of a table row displays a grid of molecules from the selected cluster in the panel on the right. The great thing about marimo is that it lets you generate this view with just a few lines of code. The pandas dataframe containing the cluster representatives is passed to a marimo table object, which enables selection. Note that the marimo table view also automatically provides pagination. The structures on the right are shown using a grid class from marimo-chem-utils. This grid is a simple, lightweight wrapper around some code from the RDKit.

Functional Group Filtering

I’ve written a few posts in the past describing how sets of SMARTS patterns, such as the REOS filters, can be used to filter sets of chemical structures and identify compounds containing functionality that might be reactive, toxic, or problematic in assays. When running these filters, it’s often useful to review the results to make sure you’re not removing compounds you might want to keep. The script reos.py in the marimo-chem-utils examples directory uses a view similar to the one described above to enable a quick review of filtering results. In this case, we have a summary view on the left that shows the functional group filters, and a detail view on the right that shows the structures that triggered the filter, with the offending functionality highlighted. The detail view uses the same grid view we used above to show the structures. In this case, we make use of an additional argument specifying a SMARTS pattern to highlight in the structures.

Scatterplots with Chemical Structures

When evaluating data or examining model results, we often create a scatter plot to visualize the relationship between two properties. In other cases, we may project a molecular representation, such as chemical fingerprints, onto a two-dimensional plot using techniques like TSNE, UMAP, or GTM to provide an overview of chemical space. In either case, it’s useful to quickly select a point or a set of points and view the associated chemical structures. The third example notebook, scatterplot.py, provides two ways for marimo to associate chemical structures with a scatterplot.

· Tooltips – in the example, a tooltip showing a chemical structure is displayed when the user hovers the mouse pointer over a point in the plot.
· Selection – When a set of points is selected in the plot, the corresponding chemical structures are shown in the grid on the right. This grid view uses the same function from marimo-chem-utils as above.

Conclusions

The three notebooks described here provide a brief overview of a few ways marimo notebooks can be integrated into cheminformatics workflows. There’s a lot more you can do, and the marimo team is constantly adding new capabilities. Over the next few months, I’ll add more capabilities and examples to marimo-chem-utils. Stay tuned. It’s going to be fun.

Acknowledgments

This work has been a collaboration with Shivam Patel at PsiThera. We would like to thank the marimo team for an excellent tool and rapid support. We can’t wait to see what’s next for marimo.

> get the smiles,chembl_id, target_name, publication year, article doi,
and IC50 for all kinase inhibitors published after 2022
and write this into a file called kinase_inhibitors_after_2022.csv

This sounds compelling, but does it really work? To put this to the test, I gave the prompt above to Gemini and compared the results with a query I generated by hand. Determining which result was better was tricky and required some detective work. As usual, things weren’t as simple as they initially appeared. Before jumping to the punch line, I should explain a bit about what I did.

Setup

As long as the ChEMBL database file is located in the current directory or below it, Gemini will find it, construct queries, and extract the necessary information. In my case, I was using the SQLite version of ChEMBL, which is great because it doesn’t require me to set up a server, authentication, etc.

To run queries, I only needed two things.

1. The sqlite3 executable, which can be easily installed on any system.
2. The SQLite version of the ChEBML, which can be downloaded from here. Once uncompressed, the file is 28GB, so make sure you have plenty of disk space available.
My directory setup looks like this.

.  
└── chembl_36  
    └── chembl_36_sqlite  
        └── chembl_36.db

Gemini had no trouble finding the ChEMBL database and constructing and running queries.

I found that this works much better with Gemini 3.0. To do this, type “/settings” and set “Preview Features (e.g., models)” to “true”. One caveat is that I found Gemini doesn’t always give the same answer. I ran the same query several times across several systems. In most cases, it gave the “correct” answer (more on this below), but it also made a few mistakes.

• In one case, Gemini limited the size of the result set. I guess it was trying to save computing resources.
• In another case, it returned the ChEMBL DOI rather than the article DOI.
• In a third case, it used a more restrictive version of “kinase” and returned only a few results.

In all three cases above, Gemini returned the correct result after an additional prompt. We’re not there yet, but we’re getting closer.

Does Gemini Return the Correct Answer?

Gemini can construct and run the query, but does it return the correct results? To test, I built my own workflow and ran it as a comparison. I’m starting to feel like a contestant on the old game show “Are You Smarter Than a 5th Grader?,” except in this case it’s “Am I Smarter Than an LLM?” (probably not). My approach to this was pretty simple.

1. Use the bioservices API to query the UniProt database for UniProt IDs corresponding to all human protein kinases.
2. Convert the UniProt IDs to ChEMBL IDs
3. Use an SQL query to extract compound structures and IC50s for all the targets specified in #2.

For those interested in the gory details, a Jupyter notebook with code I used to extract and analyze the data is available on GitHub. My search returned 44,992 records, while the Gemini search returned 38,571. Why the difference?

The approach Gemini took was similar to mine. The primary difference was how we defined which proteins were kinases. I took the easy route and simply asked UniProt for a list of human kinases. Gemini took a different, and I must admit, clever approach and used ChEMBL’s protein classification hierarchy. The ChEMBL database contains a table called “protein classification,” which encodes a hierarchy of protein classes. At the top level are fundamental protein classes like “Enzyme,” “Adhesion,” and “Secreted protein.” Below “Enzyme” are classes like “Kinase”, “Transferase”, “Oxoreductase”, etc. A small subset of the protein classification hierarchy is shown in the figure below. If we start with a target at the bottom of the hierarchy and work our way up from a kinase target, we should eventually reach the category “Kinase”. If we do, our target is a kinase. This sounds like a reasonable approach, so what went wrong?

It’s Not You, It’s Me

As I mentioned above, my query returned more results than Gemini’s. Let’s look at what happened. First, let’s look at the records Gemini found that my query didn’t. There were 35 records across 4 targets, shown in the table below. Technically, none of these are kinases. The first three are protein kinase regulatory subunits, which lack catalytic activity. The final protein is a pseudokinase. However, all four are classified as kinases in the ChEBML protein classification hierarchy. The query constructed by Gemini did what it was supposed to do. The issue was with how the data was curated.

Next, let’s look at the records my query found that the Gemini query missed. In this case, the difference was much larger. My query found an additional 6,456 records covering 43 proteins. The top-level protein classifications for these proteins are shown in the table below. We can see that 30 are classified as Transferase, which is at the same level in the hierarchy as “Kinase”. An additional 11 proteins were classified as “Enzyme”, not as “Kinase”. One protein, Q96QT4 (TRPM7), is classified as “Transient receptor potential channel” but is also a kinase. More specifically, it is a bifunctional protein that uniquely combines an ion channel with an intrinsic serine/threonine protein kinase domain.

This harkens back to a theme I’ve been harping on for several years. It’s all about the data. Gemini constructed a valid query, but that query is only as good as the data it searches. Currently, the quality of that data depends on the people curating it. If the curators are inconsistent, an LLM will return flawed results.

• Marimo Documentation: https://marimo.readthedocs.io/en/latest/
• Marimo YouTube Channel: https://www.youtube.com/c/MarimoDataScience

There are a few aspects of marimo that might confuse some long-time Jupyter users (including me):

1. The output of a code cell in marimo appears above the cell in the notebook instead of below it, as in Jupyter.
2. A marimo notebook is reactive, meaning that when you change the value of a variable, any code cells that depend on that variable will automatically update. This differs from Jupyter, where you need to manually re-run code cells to see the updated output. It also means that a variable can only be defined once in a marimo notebook.
3. When you run a marimo notebook, it first checks if you have the necessary libraries installed. If not, marimo will ask if you’d like to install them and will install them for you. This makes it easy to share marimo notebooks with others without worrying about dependencies. This notebook has the dependencies inlined; if you run it using the --sandbox flag, marimo will create a sandboxed environment and automatically install the dependencies.
4. Once the notebook loads you can run the cells by clicking on the run button associated with each shell or hitting shift-return (just like Jupyter). You can also click on the yellow run button in the bottom right to run the whole notebook.

Using the marimo notebook is easy, just follow these simple steps.

1. Download the notebook from GitHub. Note that marimo notebooks are simply Python files with a .py extension. You can download the file from the command line with this command.

wget https://raw.githubusercontent.com/PatWalters/practical_cheminformatics_posts/refs/heads/main/expansion_data_exploration/openadmet_expansion_exploration.py

2. Install marimo and uv using the following command:

pip install uv marimo

3. Use the marimo command to run the notebook. This command installs all the dependencies and launches the marimo notebook in a sandboxed environment.

marimo edit openadmet_expansion_exploration.py --sandbox

4. Enjoy!

You can also try out the notebook on molab, marimo’s answer to Google Colab.

1. Go to https://molab.marimo.io and sign up for a free account.
2. To run the notebook on molab click here

Where’s the Code?

The code and notebook for this post can be found in this GitHub repo https://github.com/PatWalters/practical_cheminformatics_posts/tree/main/expansion_data_exploration

Acknowledgements

Thanks to Hugo MacDermott-Opeskin for testing the notebook. I wouldn’t have tried marimo if it weren’t for blogs by Eric Ma and Srijit Seal. Those guys are a constant source of inspiration. Thanks to Ramon Miranda-Quintana, Ignacio Pickering, and Kenneth Lopez Perez for their help with the BitBIRCH and BBLean clustering methods I used in the notebook. Special thanks to the marimo team. They’ve created an amazing tool, their support is fantastic, and Vincent’s videos are the best!

Last week’s NVIDIA GPU Technology Conference (GTC) featured two announcements that highlighted both the potential and ongoing challenges of protein-ligand co-folding. The recently renamed Genesis Molecular AI announced PEARL, a proprietary co-folding method. Additionally, the OpenFold consortium released the code and model weights for a preview of OpenFold3 (OF3p). Both groups also provided technical reports with initial benchmarks. Besides co-folding, the OpenFold team also shared structure prediction results for protein monomers and complexes, as well as antibody-antigen complexes and RNA monomers.

While OF3p aims to accurately reproduce AlphaFold3, PEARL seeks to push the current state of the art by incorporating “physics-based synthetic data” into its training set. Unfortunately, the source and features of this synthetic data have not yet been disclosed. Hopefully, Genesis will reveal details in future publications. Another major advancement with PEARL is its ability to condition on project-specific data. The technical report showed significant performance improvements when this approach was applied using publicly available PDB structures to condition.

Welcome to the Jungle

I was pleased to see that both groups provided benchmark data using the Runs N’ Poses dataset developed by Torsten Schwede’s team at the Swiss Institute for Bioinformatics. As I mentioned in a previous post, the protein-ligand complexes that make up the Runs N’ Poses benchmark are classified based on their similarity to the structures used to train the co-folding model. This similarity is assessed through protein sequence, binding pocket similarity, and ligand similarity. As the Schwede group demonstrated in their paper, the performance of co-folding methods declines significantly as the similarity to the training set decreases.

While PEARL and OF3p demonstrated impressive results on several benchmarks, the outcomes on Runs N’ Poses highlight ongoing challenges that co-folding approaches often encounter when adapting to systems beyond the training data. The plots below, from the PEARL and OF3p technical reports, show that all methods still struggle to accurately predict ligand poses for protein-ligand complexes that differ from those in the training set. In these plots, the x-axis groups test set compounds by their similarity to the training set, while the y-axis displays the fraction where the RMSD of the ligand structure is within 2 Å of the known experimental structure.

The figure below, from the PEARL technical report, shows that although the method enhances the current state of the art, performance still drops significantly for pockets and ligands that differ from the training set. Note that the results displayed below are the best RMSD from the top five solutions generated by each co-folding method.

The results below, from the OF3p technical report, show a similar trend. In the bar plots below, “Oracle” represents the best of the top five solutions (as shown above), while “Ranked” indicates the RMSD of the best-scoring solution. Once again, we see a significant decline in performance as the similarity to the training set decreases.

A Missed Opportunity

One aspect I wish had been included in both technical reports was the performance on allosteric binding sites. As reported in a recent paper by Eva Nittinger and coworkers at AstraZeneca (AZ), Boltz-1, RoseTTA-Fold, and NeuralPlexer struggled to accurately reproduce the poses of allosteric ligands. I repeated the AZ team’s work with Boltz-2 and obtained similar results. Most allosteric ligands were placed in the orthosteric binding site, which typically lies 10-20 Å away from the experimentally observed binding pose. While I don’t believe anyone has a clear explanation for why allosteric inhibitors are poorly predicted, many think it is due to the limited amount of evidence. The Protein Data Bank (PDB) contains hundreds of structures of ligands bound to orthosteric sites and only a few structures of ligands bound to allosteric sites. Perhaps these algorithms get overwhelmed by the comparatively large number of orthosteric structures. Either way, this is an important issue for the field and should be part of standard benchmarks. I don’t have access to PEARL, so I can’t evaluate Nittinger’s allosteric benchmark. Hopefully, the team at Genesis will do this and publish the results. However, now that the code and weights for OF3p are available, I can run that benchmark myself. Please stay tuned for the results.

What Next?

While the field still faces challenges, it’s exciting to see the rapid pace of recent advancements. I’m especially encouraged by efforts like OpenFold3 and Boltz-2, which are releasing top-tier models as open-source software. It will be interesting to see where the field goes next. As I see it, two clear trends are emerging. The first involves using synthetic data generated through physics-based methods to supplement existing data from the Protein Data Bank (PDB). Although the Genesis team didn’t specify how their synthetic data was created, this will likely motivate others to try similar approaches. One thing has become clear in today’s AI age: even if you don’t share exactly how you did something, simply doing it will inspire others to figure it out.

Another way to enhance co-folding performance is by increasing access to experimental data. Although pharmaceutical companies possess thousands of crystal structures of protein-ligand complexes, concerns over intellectual property prevent these structures from being publicly shared. The AI Structural Biology (AISB) Network, led by Apheris, addresses this by creating a federated learning infrastructure that allows co-folding methods to improve through secure integration of proprietary data into models. I see this as going deep rather than broad. The PDB is broad; it contains structures of thousands of different proteins. While there are some exceptions, usually only a few structures of ligands bound to each unique protein exist. In contrast, structural data from pharmaceutical companies is deep. They typically study a few specific proteins each year, but drug discovery projects for these targets can generate hundreds of structures. Additionally, these structures often come from only a few related series. It will be interesting to see how large collections of highly related structures can enhance co-folding methods.

Another initiative with the potential to greatly increase the number of publicly accessible structures is the OpenBind project led by Oxford University and the Diamond Light Source. This project, aiming to produce 500,000 x-ray structures in the next five years, could supply crucial data to help develop the next generation of co-folding algorithms.

Although we’re still a long way from predicting the structure of any ligand bound to any protein, the field continues to advance rapidly. I’m eager to see where we’ll be in a few months.

Unfortunately, the answer to all of these questions is often no. An example of this concerning trend is provided in a recent paper from JCIM, which compares several approaches to modeling aqueous solubility. In it, the authors collected data from four different studies, removed duplicates, harmonized the solubility values, and created a dataset with 17,937 chemical structures (actually 17,935, as two structures had errors) with corresponding aqueous solubility values expressed as the log of the molar solubility (LogS). This data was used to train various machine learning models, which were then tested on a solubility dataset originally published by Huuskonen.

To satisfy my curiosity, I downloaded their data and built an ML model using 2D descriptors calculated with the RDKi and XGBoost. The performance of my model, with an R² of 0.92 and MAE of 0.41, was comparable to the best R² of 0.92 and MAE of 0.40 reported in the paper.

On the surface, this appears to be a great model. Unfortunately, the performance on the Huuskonen dataset doesn’t reflect what can be expected with “real-world” datasets. There are two factors that make the Huuskonen dataset too easy:
1. The dynamic range is unrealistically large. As I’ve written about before, the dynamic range of a test set should reflect real-world use cases. In drug discovery, aqueous solubility datasets typically span 2 to 3 logs. The Huuskonen dataset spans a whopping 12 logs. When the dynamic range is this large, regression becomes much easier, and R² values are inflated.
2. While the authors removed structures that were duplicated between the training and test sets, most of the Huuskonen test set compounds have similar neighbors in the training set.

Additionally, the Huuskonen dataset, along with the datasets used to train the model, was assembled from dozens of papers. As I’ve mentioned before, using data like this to build an ML model is a poor idea. Since the experiments were not performed under consistent conditions, it doesn’t make sense to use the data for training or testing an ML model. In a related example, a 2024 paper by Landrum and Riniker showed a significant lack of consistency in IC50 values for the same compounds tested in the “same” assay across different papers. If anything, the situation with solubility values is probably even worse.

Comparing Training and Test Sets

To provide a more realistic view of model performance, I selected two solubility datasets from the Polaris collection. Unlike the datasets used to train the model, the experiments that generated these values were carried out in a consistent manner. The datasets I chose are quite different in nature; the Biogen dataset includes a diverse set of compounds from a screening collection, while the Antiviral dataset was obtained from a lead optimization campaign that was part of the Covid Moonshot effort.

Before building an ML model, it’s always a good idea to perform some exploratory data analysis. We begin by comparing the range of values in the training set with those in the test sets. One effective way to do this is by creating a box plot. The plot below shows the distribution of LogS for the training and test sets. The boxplots reveal two main points. First, the distributions of LogS for the training set and the Huuskonen test set are similar. Second, the dynamic range for the Antiviral and Biogen sets is considerably smaller than that of the other two sets.

This becomes even clearer when looking at the interquartile range (IQR) for the three test sets. The IQR represents the height of the box in the box plots. The IQR for the train and Huuskonen sets exceeds 2.5 logs, while the other two sets have an IQR below one log. This smaller dynamic range makes it more difficult to achieve a good R² value from a regression model.

Another factor to consider in exploratory data analysis is how similar the test set(s) are to the training set. We can measure this by calculating the Morgan fingerprint Tanimoto similarity between the training and test sets and recording the highest similarity for each test set compound relative to the training set. By analyzing the Tanimoto similarity distribution, we can assess the overall difficulty of evaluating the model. In the figure below, we display the distributions of maximum Tanimoto similarity for each of the three test sets. For the Huuskonen dataset, at least half of the compounds have a Tanimoto similarity greater than 0.5 to the training set. If we set a heuristic cutoff for Tanimoto similarity at 0.35, then most compounds in the Huuskonen data can be considered similar to the training set. We can also see that the Antiviral and Biogen sets are much less similar to the training set.

Performance on Realistic Test Sets

As mentioned earlier, a relatively simple ML model using RDKit descriptors and XGBoost achieved what appeared to be impressive performance. Now, let’s see what happens when this same model is applied to predict the aqueous solubility of datasets that differ from the training set and cover a narrower, more realistic, dynamic range. To evaluate the model’s performance, I generated the same RDKit descriptors for the Antiviral and Biogen datasets. The XGBoost model, trained on the same training set as before, was then used to predict LogS. The plot below compares the experimental and predicted values for each of the three test sets. As shown, there is no apparent correlation between the experimental and predicted values for the Antiviral and Biogen test sets. While an MAE around 10-fold may seem somewhat useful, a simple null model that predicts each value as the dataset mean has an MAE of approximately 0.53.

How Can We Improve the Situation?

Papers like the one in question aren’t helping to advance the field. In fact, they’re doing the opposite. They lead readers to believe that solubility prediction is a solved problem, when it isn’t. Journals should establish guidelines that specify the quality of datasets. It’s 2025, and there’s no reason for anyone to rely on low-quality datasets like those in MoleculeNet or the Therapeutic Data Commons. Unfortunately, we’ve fallen into a cycle where people assume that because a dataset was used in a paper published in a high-impact journal, it must be a “gold standard.”

I believe there are a few things we can do to improve the situation. As authors, we can make sure our papers use high-quality datasets and our benchmarks include proper statistical comparisons. As reviewers, we have the chance to ensure that the papers we review follow best practices. I’ve reached a point where at least half of my reviews contain similar language.

Oops, I guess my reviews are no longer anonymous.

Finally, journal editors and editorial advisory boards can establish guidelines that incorporate best practices. Journals should endorse datasets like those provided by the Polaris initiative, which involves groups of experts from industry and academia certifying the quality and relevance of benchmark datasets. I hope to reach a point where reviewers of ML papers have a checklist to ensure that high-quality datasets are used and that appropriate statistical tests are applied when comparing methods. In 2011, I was part of a team that developed guidelines for computational papers in the Journal of Medicinal Chemistry. These guidelines significantly improved the quality of papers in the journal. I am convinced we can achieve the same with papers focused on machine learning in drug discovery.

Code and Data
The code and data used to generate this analysis can be found on GitHub

I have spent the past thirty years working on a wide range of drug discovery projects. During that time, I have noticed that potency optimization is rarely the primary cause of delays in drug discovery. Instead, teams usually focus most of their efforts on improving pharmacokinetics and reducing off-target interactions that could cause adverse side effects. They encounter challenges managing interactions with cytochrome P450 enzymes, which can lead to drug-drug interactions, hERG inhibition—potentially affecting normal heart function—and other unintended effects. Unlike potency optimization, where tools like free energy perturbation (FEP) guide decisions, ADMET optimization often depends on heuristics based on hard-earned experience.

Mark Murcko and James Fraser introduced the term “avoidome” to describe targets that our drug candidates should avoid. For more background on the avoidome and the reasons behind OpenADMET, I recommend watching Mark’s recent talk at Drug Hunter. Essentially, OpenADMET tackles the avoidome by combining three components: targeted data generation, structural insights from x-ray crystallography and cryoEM, and machine learning. This combined approach allows us to better understand the factors that influence interactions with avoidome targets. By gaining this knowledge, we can develop reusable strategies to steer clear of these targets and break the frustrating cycle of “whack-a-mole”—where progress is often undone by unexpected setbacks—that frequently occurs in drug discovery projects.

I strongly believe that close interaction between computation and experimentation is crucial for advancing drug discovery. This is one of the main reasons I am enthusiastic about OpenADMET. We can not only generate data and build models but also develop structural insights to help interpret that data. For example, consider hERG inhibition. As mentioned earlier, hERG inhibition can cause potentially fatal disruptions in normal cardiac rhythm. The literature contains examples where medicinal chemistry teams have combined intuition and brute-force synthesis to reduce hERG liability. Instead of just collecting data to build a model, we can collaborate with our colleagues at UCSF to determine experimental protein-ligand structures and understand the relationship between chemical structure and hERG binding. Additionally, with team members at OctantBio, we can synthesize and test additional analogs to further explore what drives activity, an informative exercise usually reserved for on-target activity. This collaboration among assays, ML models, and structures will help us better understand outliers and cases where our models fall short. We are fortunate to have initial funding that we believe will enable us to test these ideas and accelerate the development of better ML models.

Machine Learning to the Rescue?
Over the past decade, machine learning (ML) models have become vital in modern drug discovery. However, despite their guidance, their success heavily depends on large, high-quality datasets. An ML model relies on three key elements: high-quality training data, the representation—which converts a chemical structure into a vector understandable by the model—and the algorithm that finds the relationship between data and representation. These elements are vital in that order: data is the most important, followed by representation, with algorithms providing smaller, incremental improvements. Sadly, the field has often focused on the wrong aspects. Much attention has been on algorithms like neural networks, only to find that their gains over simpler methods, such as decision tree ensembles, are limited given the dataset sizes and quality. Until recently, there were few initiatives dedicated to generating experimental data specifically to improve ML model development. High-quality experimental data, like that from OpenADMET, can be the foundation for better molecular representation and ML algorithms, strengthening the entire process.

Most of the literature datasets currently used to train and validate ML models were curated, sometimes inaccurately, from dozens of publications, each of which conducts experiments differently. A recent paper by Greg Landrum and Sereina Riniker compared cases where the same compounds were tested in the “same” assay by different groups across papers. When comparing IC50 values, they found almost no correlation between the reported values from different papers. Although Landrum and Riniker didn’t compare ADMET assays, the same lack of correlation exists there as well. Instead of relying on low-quality literature data, we need consistently generated data from relevant assays with compounds similar to those synthesized in drug discovery projects. With these datasets in hand, the field can pursue new advances in molecular representation and algorithms.

The Proof is in the Pudding
While high-quality datasets provide a solid foundation for ML models, questions still remain about how to practically split these datasets for testing model performance. Ultimately, our models should be evaluated prospectively on compounds the model has not previously seen. One of the most effective methods for such prospective testing is through blind challenges, where teams receive a dataset and are asked to submit predictions, which are then compared to ground truth data. One can argue that the recent Nobel Prize-winning success of protein structure prediction methods, such as AlphaFold and RoseTTAFold, would not have been possible without the semiannual Critical Assessment of Protein Structure Prediction (CASP) challenges.

With OpenADMET, we will generate data and use it to regularly host blind challenges. Our aim is not just to publish results but also to share tutorials and hold seminars using challenge datasets to showcase best practices. In collaboration with the teams involved in the ASAP Initiative and Polaris, the OpenADMET team has already organized a blind challenge focused on activity, structure prediction, and ADMET endpoints. More challenges will be announced soon.

Democratizing ADMET Models
So far, I’ve emphasized the data generation capabilities of OpenADMET, but this effort isn’t only for ML enthusiasts. There is also an internal initiative to develop high-quality models and share them with the community. We will create and update models based on data generated in our labs and from other sources. These models will be packaged and made freely accessible. We will also explore the best ways to combine data from multiple sources, update models with new data, and incorporate experimental data to improve model performance. By working openly and incorporating community feedback on the design and performance of the models, we can enhance their scope, accuracy, and usability in real-world applications.

Answering Some Fundamental Questions
Besides supplying data for model training and future assessments, OpenADMET will serve as a platform to tackle some unresolved core issues facing ML in drug discovery.

• Molecular representation - Many of the most successful ML approaches use molecular representations, such as chemical fingerprints, which have been utilized for decades. Although many other methods have been proposed, it has been hard to determine if they genuinely improve performance. The datasets created by OpenADMET and related efforts, like OpenBind, will enable robust prospective and retrospective comparisons of molecular representations. Hopefully, these datasets will also encourage progress that pushes the field beyond simple molecular graphs toward a more generalizable description of chemical structure.
• Defining a model’s applicability domain - The OpenADMET datasets will help us systematically analyze the relationship between training data and a set of compounds whose properties need to be predicted. These datasets can support the community in proposing and assessing methods for identifying where models are likely to succeed and where they might fail.
• Global vs. local models - Although there is ongoing debate about whether global models outperform series-specific, local models, few systematic comparisons between the two methods have been made. OpenADMET will gather diverse datasets, along with data on specific chemical series. These datasets will allow us to evaluate and compare different model-building approaches.
• Multi-task models - In multitask ML, a model is trained to predict multiple outcomes simultaneously. The literature includes examples where multi-task learning has been beneficial, as well as cases where it has been less successful. OpenADMET is gathering data on various ADMET-related properties, and these datasets should enable the community to test different multi-task strategies.
• Foundation models and fine-tuning - Over the past few years, the field has made significant progress developing foundation models trained on large datasets and fine-tuned for property prediction and related tasks. Unfortunately, most subsequent validation studies were conducted on low-quality datasets and lacked proper statistical validation. The datasets created by OpenADMET will allow the community to compare a wide range of approaches. When combined with recently published guidelines for comparing ML methods, this data should support more robust comparisons that help us identify genuine advances.
• Quantifying Uncertainty - Using data that a machine learning model was trained on, we should be able to estimate our confidence in a prediction. Although many publications have addressed uncertainty estimation, testing these estimates prospectively has been difficult. The regular data releases from OpenADMET should serve as an excellent testbed for new methods of uncertainty quantification.

In addition to these computational questions, we will also address critical experimental issues such as assay drift and reproducibility. All experimental data will be made publicly available for the community to evaluate and learn from.

We Should Talk
OpenADMET won’t succeed in isolation. Our efforts must be part of a dialogue with the larger community. Over the upcoming weeks and months, I’ll be reaching out to numerous people in industry and academia. If you have thoughts on this very important topic, please share them with me. I’d love to have a conversation.

Acknowledgements
I’d like to thank the members of the OpenADMET team for helpful comments on earlier versions of this post.

While these methods show great promise, they also have limitations. In this post, I’d like to highlight three papers where the authors conducted careful, systematic studies to examine where co-folding methods succeed and where they fall short. I will conclude by discussing where co-folding methods are effective and what steps are necessary to improve them.

Don’t Stray Too Far From Home

The first paper I want to highlight is “Have protein-ligand co-folding methods moved beyond memorization?” by Peter Škrinjar and colleagues from Torsten Schwede’s group at the Swiss Institute of Bioinformatics. In this paper, the authors examined the relationship between co-folding performance and similarity to the training set. They used metrics developed for the PLINDER benchmarking effort to evaluate the similarity between a predicted protein-ligand complex and the complexes used to train the co-folding model. Similarity was measured in three ways: overall sequence similarity, binding site similarity, and ligand similarity.

The main takeaway from this paper is summarized in their Figure 1 (shown below), which shows similarity to the training set on the x-axis and co-folding success on the y-axis. As mentioned earlier, similarity to the training set is assessed based on protein sequence, ligand similarity, and binding site interactions. The “success” of the co-folding methods is measured by the root mean squared deviation (RMSD) of the experimental and predicted ligand coordinates, as well as the predicted Local Distance Difference Test (pLDDT), which evaluates how well the predicted structure captures the intermolecular interactions in the experimental structures.

As shown in the figure above, there is an almost linear decline in performance as similarity to the training set decreases. Below 80% similarity to the training set, this performance noticeably diverges from the 80-90% success rate reported in most co-folding studies. Notably, many co-folding papers use a time split in the PDB to separate training and test sets. Models are typically trained on structures deposited before 2021 and tested on structures deposited in 2021 and after. This approach is problematic because crystallographers often deposit structures that are similar or even identical to previously deposited ones. The figure below, which I generated, shows eight structures from the widely used Posebusters dataset, which are nearly identical to structures deposited before 2021. To their credit, the authors of the Posebusters paper pointed out that their dataset has a range of similarities to the training set, a fact that may have been overlooked by those reporting Posebusters as a “gold standard benchmark”. For a higher resolution version of the figure below, please click here.

There are different views on Figure 1 from the Škrinjar paper. The optimistic perspective, mainly supported by those working on co-folding methods, is that most work in drug discovery takes place on the right side of the plot, where current techniques perform well. While research on well-understood protein classes like kinases continues, many in drug discovery believe that most efforts are aimed at new targets, which would be on the left side of the plot. The truth likely lies somewhere in between, with a 50% success rate still falling short of ideal.

Overwhelming Evidence Can Be Misleading

The second paper I want to examine is a study titled “Co-folding, the future of docking – prediction of allosteric and orthosteric ligands” by Eva Nittinger and colleagues at AstraZeneca in Sweden. In this paper, the authors compare the performance of co-folding methods on orthosteric versus allosteric binding sites. They selected 16 cases from the PDB where structures had been solved for both orthosteric and allosteric ligands targeting the same protein. For example, consider checkpoint kinase 1 (CHK-1), a serine/threonine protein kinase that has been a target for therapies aimed at repairing DNA damage. The PDB includes structures such as 2e9n with a ligand bound in the orthosteric ATP binding site. The PDB also includes the structure 3f9n, which contains a ligand bound in an adjacent allosteric pocket. The figure below shows an overlay of these two structures, highlighting the distance between the two binding pockets.

In their paper, Nittinger and colleagues compared the effectiveness of co-folding methods in generating structures of orthosteric and allosteric ligands. The authors used NeuralPlexer, RoseTTAFold All-Atom, and Boltz-1 to generate structures of the orthosteric and allosteric ligands for each of the 16 protein targets. They then calculated the RMSD of the predicted structures against the corresponding experimental structures from the PDB. To provide a simplified overview of the data, I replicated their study using only Boltz-1 and created the figure below, which resembles Figure 1 in the Nittinger study. I chose to focus on Boltz-1 to make the figure simpler than the one in the Nittinger paper. Additionally, I had been using Boltz-1, so I was familiar with the software. While the plot below shows only Boltz-1, the results reported by Nittinger demonstrate a notable agreement between the different co-folding methods. Stay tuned, I’ll repeat this plot with Boltz-2 next week.

The x-axis in the plot shows the RMSD between predicted and experimental ligand structures. The RMSD is calculated by first aligning the predicted protein structure with the experimental structure from the PDB, then computing the RMSD between the corresponding ligand atoms. Each row in the figure represents a pair of protein structures, with blue points indicating RMSD for five orthosteric poses and orange points for five allosteric poses. The labels on the y-axis correspond to the genes associated with the proteins in the PDB. As in the Nittinger paper, a vertical line is drawn at 2.5 Å to mark the limit for what is considered a “successful” pose. As shown in the figure, Boltz-1 was able to successfully generate poses for most of the orthosteric ligands. Two cases where Boltz-1 failed to reproduce orthosteric ligand poses —CYP3A4 and RORC—are flexible proteins that are notoriously difficult to model. Conversely, performance on allosteric ligands was poor, with RMSD values exceeding 10 Å in most cases. As Nittinger points out, this is because the co-folding methods often place the allosteric ligand in the orthosteric pocket.

In their paper, Nittinger and coworkers achieved a modest improvement by co-folding two copies of the allosteric ligand, hoping that one would block the orthosteric pocket and force the other into the allosteric pocket. When two copies of the ligand were used, the co-folding methods managed to place half of the ligands within 2.5 Å of the center of mass of the allosteric ligands. Although this strategy appears somewhat effective, it is unsatisfying and may be difficult to implement prospectively.

As I mentioned in a LinkedIn post, all the orthosteric and allosteric structures used in the Nittinger benchmark were deposited in the PDB before 2021 and were likely included in the training sets for the co-folding methods. Given their presence in the training data, I was surprised that the co-folding methods performed so poorly on the allosteric systems. As Nittinger points out, most structures in the PDB for these targets are orthosteric. It’s possible that this abundance of evidence overrides the signal from the allosteric ligands and causes most predictions to focus on the orthosteric site. This highlights one of the challenges in the current generation of protein structure prediction and co-folding methods. While we can indirectly examine these issues, it’s not easy to look inside and understand the decisions made by the co-folding models.

What Are We Learning?

The third paper I’d like to cover is “Do Deep Learning Models for Co-Folding Learn the Physics of Protein-Ligand Interactions?” by Matthew Masters and colleagues from Marcus Lill’s group at the University of Basel in Switzerland. In this paper, the authors examined the impact of binding site mutations on the poses predicted by AlphaFold3. As part of this study, they performed co-folding using the same ligand SMILES with four variations of the original wild-type protein sequence. The resulting modified protein sequences were co-folded with the original ligand SMILES, and the RMSD to the corresponding X-ray structure was compared. The four variations were:

• The wild-type sequence (No Mutation), which remains unaltered from the original PDB.
• A set of mutations that removed the binding site (GLY Mutation) by replacing any residue within 3.5 Å of the ligand with GLY.
• A set of mutations that crowded the binding site (PHE Mutation) by replacing any residue within 3.5 Å of the ligand with PHE.
• A set of mutations that used scores originally developed by Miyata to replace any residue within 3.5 Å of the ligand with a dissimilar residue. In this way, TYR was replaced with GLY, LEU was replaced with ASP, and so on. A complete table of the replacements is in Appendix A of the Masters paper.

The table below displays the RMSD in Å of the predicted ligand coordinates compared to those in the original x-ray structure. The table rows list three different proteins:

• 8x61 - a cryoEM structure of ATP-bound FtsE
• 7y97 - an x-ray crystal structure of CYP109B4 with a heme as the ligand
• 1mzm - an x-ray crystal structure of a lipid transfer protein with a flexible palmitate ligand

The second column in the table shows the RMSD for the wild-type protein, while the following columns present the RMSD for three mutation strategies. Notably, mutations at the binding site had little effect on co-folding performance. As observed, the co-folding methods still managed to position the ligands in the binding site despite removing interacting residues or the steric hindrance caused by larger residues. This is not ideal and suggests a lack of generalizability. The authors propose that the co-folding methods may be learning longer-range patterns rather than specific protein-ligand interactions.

It’s important to note that the preprint by Masters was a preliminary study. The authors examined only three protein-ligand systems, and all co-folding was performed using an early server-based version of AlphaFold3. Given the emergence of several other co-folding approaches, it will be essential to conduct similar tests on a wider variety of protein-ligand systems.

What Is It Good For? (with apologies to Edwin Starr)

This raises questions about where co-folding methods should be applied in drug discovery. Since they are slower than docking and their binding poses differ, co-folding approaches are likely not suitable for large-scale virtual screening. Although some claim that binding affinity predictions from co-folding are “as accurate as FEP and 1,000 times as fast,” the current evidence does not support this. The only reliable way to assess how well co-folding methods can predict binding affinity is probably through prospective challenges and real drug discovery projects. To this end, I commend the Boltz team for seeking collaborators who can experimentally validate their methods. I am also excited about initiatives like OpenBind and OpenADMET, which might offer the first opportunities to continually evaluate the prospective performance of machine learning models.

I have been using co-folding methods in a similar way to how I use molecule generation techniques— as tools to develop hypotheses that can be tested experimentally. We have many methods that tell us if a molecule binds, such as biochemical assays, size exclusion chromatography-MS, examining binding in a DEL screen, or performing various other experiments. However, apart from X-ray crystallography or cryoEM, we have very few experiments that can definitively identify where a compound binds. In most cases, we need a hypothesis to guide experiments, such as mutagenesis or installing a photoaffinity label. Even more definitive experiments, like NMR or HDX-MS, are much more useful when a binding hypothesis is available. Co-folding provides an easy way to generate these hypotheses; then, it’s up to our scientific knowledge, intuition, and creativity to decide how to proceed.

Although it may not seem like it, I am very excited about co-folding and other uses of machine learning in drug discovery. We have made some progress, but we still have a long way to go. It’s crucial that we keep testing these methods, understanding their limitations, and working to improve the core algorithms. These advancements will only come through close collaboration among computational method developers, drug designers, and experimentalists.