From Information-Theoretic Secrecy to Molecular Discovery: A Unified Perspective on Learning Under Uncertainty

1. Introduction

The trajectory of modern machine learning research reveals a surprising convergence: problems that appear domain-specific—securing communication channels against eavesdroppers, or identifying drug-like molecules from billion-scale chemical libraries—share deep structural similarities when viewed through the lens of learning under uncertainty. This paper synthesizes insights from two seemingly disparate research programs to argue that information-theoretic principles provide a powerful, unifying framework for understanding and improving deep learning systems across domains.

The first research program, rooted in Shannon theory, addresses the fundamental limits of secure communication over broadcast channels [1]. The core question—how to simultaneously guarantee message delivery to intended receivers while maintaining confidentiality from unintended ones—requires precise characterization of rate-equivocation regions. This work established inner and outer bounds for discrete memoryless broadcast channels with confidential messages, generalizing classical results by Csiszár-Körner, Liu et al., and Marton.

The second program applies machine learning to drug discovery [2], specifically using DNA-Encoded Chemical Library (DEL) selection data to train models capable of virtual screening across billion-molecule chemical spaces. The discovery of a first-in-class small-molecule ligand for WDR91 demonstrated that ML models trained on noisy, combinatorial selection data can generalize to predict binding affinity in structurally diverse chemical libraries.

We argue that these two lines of work are connected by three fundamental principles:

1. Capacity under constraints: Both problems involve optimizing information throughput subject to structural constraints (secrecy requirements or binding specificity).
2. Generalization from finite samples: Channel coding theorems and ML generalization theory both address how to reliably infer structure from limited observations.
3. Robustness to noise: Equivocation in secure communication and noise in DEL selection data both require methods that degrade gracefully.

2. Background and Related Work

2.1 Secure Communication and Rate-Equivocation Regions

The broadcast channel with confidential messages (BCC) model considers a sender transmitting to two receivers, where certain messages must be kept secret from unintended receivers. Secrecy is measured by equivocation—the conditional entropy of the confidential message given the eavesdropper's observation. Xu, Cao, and Chen [1] established both inner and outer bounds for the full rate-equivocation region $\mathcal{R}$ of the discrete memoryless BCC:

$$\mathcal{R} \subseteq {(R_0, R_1, R_2, R_{e1}, R_{e2}) : R_0 + R_1 + R_2 \leq C_{outer}}$$

where $R_0$ is the common message rate, $R_1, R_2$ are private message rates, and $R_{e1}, R_{e2}$ are equivocation rates measuring secrecy. The inner bound generalizes several known results:

• Csiszár and Körner's region for single confidential messages
• Liu et al.'s region for perfect secrecy constraints
• Marton's and Gel'fand-Pinsker's regions for general broadcast channels

Key insight: the gap between inner and outer bounds characterizes our uncertainty about optimal encoding strategies—a form of model uncertainty that mirrors the generalization gap in machine learning.

2.2 Machine Learning for Molecular Discovery

DNA-Encoded Chemical Libraries (DELs) enable massively parallel screening of billions of compounds against protein targets. However, DEL selection data is inherently noisy: enrichment signals are confounded by synthesis bias, non-specific binding, and amplification artifacts. Ahmad, Xu et al. [2] demonstrated that ML models (including deep neural networks) trained on fingerprinted DEL data can:

1. Learn meaningful structure-activity relationships despite noise
2. Generalize from a 3-billion-molecule training library to predict active compounds in a 37-billion-molecule virtual library (Enamine REAL Space)
3. Discover novel chemotypes—the hit compound 1 binds WDR91 with $K_D = 6 \pm 2\ \mu\text{M}$, confirmed by co-crystal structure

The ML pipeline involved: molecular fingerprinting → supervised learning on DEL enrichment → virtual screening → experimental validation → ML-assisted SAR exploration.

3. The Information-Theoretic Bridge

3.1 Channel Capacity as Learning Capacity

We draw a formal analogy between channel coding and supervised learning:

The rate-equivocation region in [1] characterizes the fundamental trade-off between communication rate and secrecy. In ML terms, this maps to the trade-off between model utility (predictive accuracy) and model privacy (resistance to data extraction attacks). This connection has been formalized in differential privacy, but the broadcast channel model provides richer structure: it handles multiple receivers with heterogeneous secrecy requirements simultaneously.

3.2 Noise Resilience: From Equivocation to DEL Denoising

The equivocation-based secrecy metric in [1] quantifies how much information an eavesdropper can extract. Mathematically:

$$R_e = H(W | Z^n) / n$$

where $W$ is the confidential message and $Z^n$ is the eavesdropper's received sequence. High equivocation means the eavesdropper gains little information—the message is effectively "noisy" to them.

In DEL-based drug discovery [2], the analogous challenge is that the true binding signal is obscured by experimental noise. The ML model must extract the "message" (true binding affinity) from "noisy observations" (DEL enrichment counts). The success of [2] demonstrates that modern deep learning architectures have sufficient capacity to decode this noisy channel.

We formalize this connection: DEL selection can be modeled as communication through a noisy channel where:

• Input $X$: molecular structure (fingerprint)
• Channel $P(Y|X)$: the biochemical selection process (noisy, non-linear)
• Output $Y$: enrichment count

The ML model acts as a decoder, and its generalization performance is bounded by the mutual information $I(X; Y)$—the capacity of the "biochemical channel."

3.3 Generalization Across Distributions

Perhaps the most striking parallel: in [1], the coding scheme must work for any channel realization within a class (less noisy, deterministic, semi-deterministic). In [2], the ML model trained on one chemical library (HitGen OpenDEL, 3B molecules) must generalize to a structurally different library (Enamine REAL, 37B molecules).

This cross-distribution generalization is the ML analog of universal coding—designing codes that achieve near-capacity performance across a class of channels without knowing the exact channel. The success of [2] suggests that molecular fingerprints provide a representation that is approximately "sufficient" in the information-theoretic sense, capturing the relevant features for binding prediction while being invariant to irrelevant structural details.

4. Implications for Modern Deep Learning

4.1 Information-Theoretic Regularization

The rate-equivocation framework suggests new regularization strategies for deep learning. Rather than simply minimizing prediction error, we can explicitly optimize the information bottleneck:

$$\min_\theta \ \mathcal{L}(\theta) + \lambda \cdot I(Z; X | Y)$$

where $Z$ is the learned representation, $X$ is the input, and $Y$ is the target. The term $I(Z; X | Y)$ penalizes representations that retain information about the input beyond what is needed for prediction—directly analogous to maximizing equivocation of the "eavesdropper" (overfitting signal) while maintaining "communication rate" (predictive accuracy).

4.2 Multi-Objective Learning as Broadcast Coding

Modern ML systems increasingly serve multiple objectives simultaneously: a drug discovery model must predict binding affinity, selectivity, ADMET properties, and synthesizability. The broadcast channel framework [1] provides a natural model: the "sender" (training data) communicates to multiple "receivers" (objective functions), each requiring different information. The rate-equivocation region characterizes the achievable trade-offs.

4.3 Scaling Laws and Channel Capacity

The empirical scaling laws of deep learning—performance improving predictably with model size, data size, and compute—find a natural interpretation in information theory. Channel capacity sets the fundamental limit; model size determines the codebook complexity; training data determines how well we can estimate the channel. The outer bounds in [1] correspond to fundamental limits on what any model can achieve, while inner bounds correspond to what specific architectures (coding schemes) can attain.

5. Case Study: Revisiting WDR91 Discovery Through an Information Lens

We revisit the WDR91 drug discovery pipeline [2] through the information-theoretic lens developed above.

Stage 1: DEL Selection as Channel Estimation. The DEL experiment against WDR91 produced enrichment data for ~3 billion compounds. Information-theoretically, this is a single "channel use" revealing $I(X; Y)$ bits about the binding landscape. The success of ML training on this data implies $I(X; Y)$ is substantial—the biochemical channel has non-trivial capacity.

Stage 2: ML Training as Codebook Construction. Training the ML model on DEL data constructs a "codebook" mapping molecular fingerprints to predicted binding affinity. The model's test performance measures how close this codebook comes to achieving channel capacity. The use of molecular fingerprints (rather than raw SMILES or 3D coordinates) is a form of source coding that compresses the input to its relevant features.

Stage 3: Virtual Screening as Decoding. Applying the trained model to the 37B-molecule Enamine REAL library is decoding: using the learned codebook to identify messages (active molecules) from a vast space of possibilities. The hit rate (fraction of predicted actives that are experimentally confirmed) measures the block error rate of this "code."

Stage 4: SAR Exploration as Adaptive Coding. The ML-assisted structure-activity relationship exploration—where confirmed hits inform iterative model refinement—is adaptive coding with feedback, known in information theory to potentially achieve capacity more efficiently.

6. Discussion and Future Directions

6.1 Towards Capacity-Achieving Drug Discovery

If we can estimate the "capacity" of the DEL screening channel—the maximum amount of binding information extractable from a single DEL experiment—we can assess whether current ML approaches are near-optimal or whether substantial gains remain. This would guide investment in better ML models vs. better experimental protocols.

6.2 Secure ML for Drug Discovery

The secrecy framework of [1] has direct implications for protecting proprietary drug discovery data. Pharmaceutical companies could share DEL screening results through channels that provably limit information leakage about their proprietary libraries, using coding schemes inspired by the broadcast channel with confidential messages.

6.3 Foundation Models as Universal Codes

Large pre-trained molecular foundation models (e.g., those trained on billions of molecules) can be viewed as universal codes—encoding schemes that achieve near-capacity performance across diverse "channels" (protein targets). The information-theoretic framework predicts that such universal codes exist when the channel class has bounded complexity, suggesting that molecular foundation models should work well when the diversity of protein binding sites is "finite" in an information-theoretic sense.

7. Conclusion

By connecting two high-impact research programs—information-theoretic security [1] and ML-driven drug discovery [2]—we have demonstrated that deep learning systems operate under fundamental constraints that are precisely captured by Shannon theory. The rate-equivocation region provides a unifying language for discussing trade-offs between utility, privacy, and robustness in ML systems. The success of DEL-based drug discovery can be understood as near-capacity communication through a noisy biochemical channel.

This unified perspective suggests concrete research directions: information-theoretic regularization for deep learning, capacity estimation for experimental screening protocols, and secure multi-party drug discovery using broadcast coding principles. As deep learning continues to transform both communication systems and molecular science, the mathematical bridges between these fields will become increasingly valuable.

References

[1] J. Xu, Y. Cao, and B. Chen, "Capacity Bounds for Broadcast Channels With Confidential Messages," IEEE Transactions on Information Theory, vol. 55, no. 10, pp. 4529–4542, Oct. 2009. DOI: 10.1109/TIT.2009.2027500

[2] S. Ahmad, J. Xu, J. A. Feng, et al., "Discovery of a First-in-Class Small-Molecule Ligand for WDR91 Using DNA-Encoded Chemical Library Selection Followed by Machine Learning," Journal of Medicinal Chemistry, vol. 66, no. 23, pp. 16051–16065, 2023. DOI: 10.1021/acs.jmedchem.3c01471

[3] I. Csiszár and J. Körner, "Broadcast channels with confidential messages," IEEE Transactions on Information Theory, vol. 24, no. 3, pp. 339–348, 1978.

[4] R. Liu, I. Maric, P. Spasojević, and R. D. Yates, "Discrete memoryless interference and broadcast channels with confidential messages: Secrecy rate regions," IEEE Trans. Inf. Theory, vol. 54, no. 6, pp. 2493–2507, 2008.

[5] K. Marton, "A coding theorem for the discrete memoryless broadcast channel," IEEE Trans. Inf. Theory, vol. 25, no. 3, pp. 306–311, 1979.

[6] N. Tishby, F. C. Pereira, and W. Bialek, "The information bottleneck method," Proc. 37th Allerton Conference, 1999.

[7] A. Shrestha, S. Mahmood, and Y. Li, "Machine learning on DNA-encoded libraries: A new paradigm for hit finding," Journal of Medicinal Chemistry, vol. 64, no. 14, pp. 10230–10244, 2021.

[8] J. Kaplan, S. McCandlish, T. Henighan, et al., "Scaling laws for neural language models," arXiv:2001.08361, 2020.