The chemical space of possible drug-like molecules is often estimated to be on the order of 10⁶⁰ possible drug-like molecules. No experimental program can search that space exhaustively. Machine learning earns its place in drug discovery not by replacing experiments, but by computationally narrowing candidate pools so that scarce wet-lab resources are spent where they are most likely to yield useful results. The engineering challenge is building computational pipelines that researchers can trust: systems that quantify their own uncertainty, acknowledge their limitations, and treat every prediction as a hypothesis that requires experimental confirmation.

The engineering problem

Drug discovery begins with a target, typically a protein implicated in a disease. The objective is to identify small molecules that interact with the target in a therapeutically useful way. Traditional high-throughput screening physically tests large libraries of compounds, but even the largest screening campaigns evaluate only a small fraction of the available chemical space. Computational methods extend the practical reach of screening by predicting which molecules are worth synthesizing and testing.

The engineering problem is one of reliable prioritization under uncertainty. A computational pipeline ingests molecular structures, applies learned representations to predict relevant properties, and ranks candidates for experimental follow-up. The value of this pipeline depends entirely on the reliability of its predictions and, critically, on its ability to communicate when those predictions should not be trusted.

Molecular representation and graph neural networks

Before a model can predict molecular properties, it needs a way to represent molecular structure computationally. Traditional approaches use fixed-length fingerprints or descriptor vectors that encode structural features into numerical representations. These methods are well understood but limited: they impose a predetermined view of which structural features matter, which may not align with the property being predicted.

Graph neural networks (GNNs) treat molecules as graphs, with atoms as nodes and bonds as edges. This allows the model to learn which structural features are informative for a given prediction task rather than relying on pre-specified descriptors. During training, message-passing layers propagate information between neighboring atoms, building representations that capture local chemical environments and, with sufficient depth, broader structural patterns.

GNNs have demonstrated strong performance on molecular property prediction benchmarks, and several architectures (MPNN, SchNet, DimeNet, and related variants) are now widely used in computational chemistry. However, their limitations are important to state clearly. GNNs learn from the data they are trained on. Their performance degrades when applied to molecules that differ substantially from the training set in terms of scaffold, functional group composition, or physicochemical properties. This is not a flaw unique to GNNs; it is a general property of learned models. But in drug discovery, where the goal is often to explore novel chemical matter, this limitation is particularly consequential.

Molecular property prediction and ADMET

Property prediction is the most direct application of ML in early-stage drug discovery. Given a molecular structure, the model estimates properties relevant to drug development: binding affinity to a target, aqueous solubility, membrane permeability, metabolic stability, and potential toxicity.

ADMET prediction (absorption, distribution, metabolism, excretion, and toxicity) is a particularly important application because poor ADMET properties are among the most common reasons compounds fail in later development stages. Computational ADMET models can flag likely problems before synthesis, allowing medicinal chemists to focus on compounds with a better chance of progressing.

The engineering considerations for property prediction are straightforward but often inadequately addressed in practice. First, the training data must be representative of the chemical space being explored. Models trained predominantly on published bioactivity data will reflect the biases of that data, including the over-representation of certain scaffold families and the systematic under-reporting of negative results. Second, prediction uncertainty must be estimated and communicated. A point prediction without a confidence interval is incomplete. Third, the distinction between interpolation (predicting within the training domain) and extrapolation (predicting outside it) must be tracked explicitly, because extrapolation errors can be large and systematic.

Limitations of current ADMET models

ADMET properties are influenced by complex biological processes that are difficult to capture from molecular structure alone. Metabolism, for example, depends on enzyme specificity, tissue distribution, and individual genetic variation. Current models approximate these processes through structure-activity relationships learned from historical data. This works reasonably well for compounds similar to those in the training set, but prediction quality for novel chemotypes is inherently uncertain. Responsible use of ADMET models requires treating their outputs as filters for prioritization, not as substitutes for the in vitro and in vivo assays that remain necessary for reliable characterization.

Virtual screening and docking

Virtual screening ranks a library of molecules by their predicted fitness for a target. Structure-based approaches use docking, which computationally estimates how a small molecule binds to a protein target and assigns a score reflecting the predicted binding affinity. Ligand-based approaches use similarity to known active compounds or learned property models to rank candidates.

Docking is useful as a coarse filter. It reduces large libraries to computationally tractable subsets that can be prioritized for experimental testing. However, docking scoring functions involve significant approximations. They typically model binding as a rigid or semi-rigid geometric fit and use simplified energy functions that do not fully capture solvation effects, protein flexibility, or entropic contributions. As a result, docking scores are better understood as rough rank-orderers than as quantitative binding affinity predictions.

ML-based scoring functions have been developed to improve docking accuracy by learning from experimentally measured binding data. These can outperform physics-based scoring functions within their training domain, but they inherit the same domain-applicability concerns as other learned models. A scoring function trained on kinase-ligand complexes may not generalize to GPCR targets, for example. The responsible engineering approach is to validate scoring functions against experimental data for the specific target class being studied and to use ensemble methods or uncertainty estimates to flag predictions where the model is operating outside its reliable range.

Generative models for molecular design

Rather than screening existing libraries, generative models propose new molecular structures with desired properties. Variational autoencoders (VAEs), generative adversarial networks (GANs), and autoregressive models (including transformer-based architectures) have all been applied to molecular generation. More recently, diffusion models have shown promise for generating three-dimensional molecular conformations.

The appeal of generative models is that they can, in principle, explore regions of chemical space that are not represented in existing compound libraries. A generative model trained on known bioactive molecules can interpolate between known scaffolds or combine structural motifs in novel ways, potentially identifying starting points that would not emerge from conventional screening.

The practical limitations are substantial. Generating a valid molecular graph is only the first step. The generated molecule must be synthetically accessible, meaning a practical synthesis route exists. It must satisfy the property constraints relevant to the program (potency, selectivity, ADMET profile). And the generative model's predictions about these properties are subject to the same uncertainty and domain-applicability concerns as any other learned model.

Synthetic accessibility scoring attempts to address one of these limitations by estimating how difficult a proposed molecule would be to synthesize. However, current synthetic accessibility models are approximate, and the ultimate test of synthesizability is whether a competent chemist can actually make the compound. Close collaboration between computational and synthetic chemistry teams is essential for generative approaches to produce actionable results.

Bayesian optimization and active learning

Drug discovery is an iterative process: experimental results from one round inform the selection of candidates for the next. Bayesian optimization and active learning provide principled frameworks for managing this iteration.

Bayesian optimization maintains a probabilistic model (typically a Gaussian process or Bayesian neural network) that estimates both the predicted property value and the uncertainty of that prediction for each candidate. It then selects the next candidates to evaluate by balancing exploitation (choosing candidates with high predicted values) against exploration (choosing candidates where the model is uncertain and additional data would be most informative). This balance is formalized through acquisition functions such as expected improvement or upper confidence bound.

Active learning follows a similar logic but focuses specifically on selecting the most informative data points to label (experimentally characterize) in order to improve the model most efficiently. In drug discovery, where each experimental measurement has a significant cost in time and resources, this is a direct engineering benefit: the pipeline focuses experimental effort where it will reduce uncertainty most.

The assumptions underlying Bayesian optimization deserve explicit attention. Gaussian process models scale poorly to very large candidate pools and may not capture complex structure-activity relationships. Bayesian neural networks offer more flexible modeling but introduce their own approximation challenges, including the choice of prior and the reliability of posterior uncertainty estimates. In practice, well-calibrated uncertainty is more important than model complexity, because the optimization loop depends on uncertainty estimates to make acquisition decisions. Miscalibrated uncertainty can lead the optimization to either exploit prematurely or explore inefficiently.

Foundation models and transfer learning

Foundation models pre-trained on large molecular datasets (such as ChemBERTa, MolBERT, and related transformer architectures trained on SMILES strings, or GNN-based models pre-trained on millions of molecular property records) are increasingly used as starting points for drug discovery tasks. The premise is that pre-training captures generalizable molecular representations that can be fine-tuned for specific prediction tasks with limited task-specific data.

This approach has genuine utility when task-specific training data is scarce, which is common in drug discovery where labeled data for a specific target may be limited to dozens or hundreds of compounds. Pre-trained representations can provide a stronger starting point than training from scratch.

The limitations are also real. Pre-training data composition determines what the model learns. If the pre-training corpus is dominated by certain compound families, the resulting representations may not transfer well to underrepresented chemical classes. Fine-tuning on small datasets risks overfitting, particularly when the fine-tuning data is not representative of the deployment domain. And the black-box nature of large transformer models makes it difficult to understand why a particular prediction was made, which complicates scientific interpretation and debugging.

Uncertainty quantification: the core engineering requirement

Across all of these methods, the single most important engineering requirement is reliable uncertainty quantification. A drug discovery pipeline that returns only point predictions without confidence estimates is incomplete in a way that directly affects research decisions. A medicinal chemist who receives a predicted affinity along with a calibrated prediction interval will make a different decision than one who is given only a point estimate with no indication of how much that number should be trusted.

A computational pipeline should not only estimate molecular properties, but also recognize when it is operating beyond the chemical domain in which those estimates remain reliable. This is fundamentally a robustness problem: the system must behave predictably not just on typical inputs, but also on inputs that probe the boundaries of its training distribution.

Equally important, uncertainty estimates are useful only if they are calibrated. A system that reports high confidence when it is frequently wrong creates false trust rather than informed decision-making. Calibration should be validated empirically on held-out data that reflects realistic deployment conditions, not only on curated benchmarks.

Reliable uncertainty estimates also support robustness by enabling the pipeline to recognize when it is operating outside its validated domain and should defer further decisions to experimental evaluation.

Several approaches to uncertainty quantification are available, each with distinct trade-offs.

The choice among these approaches depends on the specific application, the available computational budget, and the tolerance for prediction error. In practice, combining multiple uncertainty estimation methods (for example, using ensembles with applicability domain checks) provides more robust uncertainty characterization than any single method.

Closing the experimental loop

Computational predictions become valuable only when they are connected to experimental validation in a structured feedback loop. The engineering design of this loop matters as much as the quality of any individual model.

A well-designed pipeline selects candidates based on model predictions and uncertainty estimates, synthesizes and tests the top-ranked candidates, feeds experimental results back into the model as new training data, and uses the updated model to select the next round of candidates. This iterative cycle is the practical context in which Bayesian optimization and active learning operate.

Several engineering requirements govern the reliability of this loop. Experimental data must be captured in a structured format with appropriate metadata (assay conditions, batch information, measurement uncertainty). Model retraining or updating must be validated to ensure that new data improves predictions rather than degrading them through data quality issues. And the criteria for advancing a candidate must be defined in advance, not adjusted post hoc to accommodate model predictions.

Where these methods are weakest

Intellectual honesty requires stating clearly where current computational methods fall short. The most significant limitations include:

• Activity cliffs. Small structural changes can produce large changes in biological activity. ML models, which learn smooth mappings, struggle to capture these discontinuities, particularly in regions of chemical space with sparse training data.
• Selectivity prediction. Predicting that a molecule binds to the intended target is substantially easier than predicting that it does not bind to related off-targets. Selectivity is critical for drug safety but remains difficult to predict computationally.
• Multi-parameter optimization. A drug candidate must simultaneously satisfy constraints on potency, selectivity, solubility, permeability, metabolic stability, and toxicity. Optimizing multiple correlated properties introduces trade-offs that are difficult to navigate computationally, particularly when the correlations are not well characterized in the training data.
• Generalization to novel targets. Models trained on one target class may not transfer to another. A scoring function validated for kinases may perform poorly on protein-protein interaction targets, which have fundamentally different binding site characteristics.
• Data quality. Published bioactivity data contains measurement errors, inconsistencies across assay formats, and systematic biases from publication practices. Models trained on this data inherit these limitations, and the resulting predictions may be less reliable than the reported accuracy on curated benchmarks would suggest.

Building pipelines that earn trust

A computational drug discovery pipeline earns trust through engineering discipline, not through marketing claims. The following principles apply:

• Document the training domain. Specify the chemical space, target classes, and property ranges that the model was trained on. State explicitly where the model is operating within its domain and where it is extrapolating.
• Report performance honestly. Report prediction accuracy on held-out data that is representative of the deployment context, not on curated benchmarks that may overstate generalization. Include performance breakdowns by chemical series, property range, and domain distance.
• Attach uncertainty to every prediction. Use calibrated uncertainty estimates and validate their calibration empirically. A prediction without an uncertainty estimate is incomplete.
• Route uncertain predictions to human experts. When a model is operating outside its reliable domain, the pipeline should flag the prediction and defer to medicinal chemists rather than silently passing through unreliable scores.
• Close the loop. Feed experimental results back into the model systematically. Track whether the pipeline's predictions improve over successive rounds. If they do not, investigate why.
• Maintain reproducibility. Version control models, data, and code. Ensure that any prediction can be reproduced from the recorded inputs and model state. This is a basic engineering requirement that is frequently neglected in computational chemistry workflows.