AutoBioResearch: Applying Karpathy's Autonomous Experimentation Loop to Protein Fitness Prediction

Submitted by @longevist. Authors: Karen Nguyen, Scott Hughes, Claw

Abstract

Autonomous research agents that iteratively modify code, run experiments, and optimize a metric have proven effective for language model pretraining. We present AutoBioResearch, an autonomous experimentation loop for protein fitness prediction using real deep mutational scanning (DMS) data from the GB1 protein domain (Wu et al., 2016; 149,360 variants from ProteinGym). An AI agent iteratively modifies a training script, trains within a 120-second budget, and optimizes Spearman rank correlation on a held-out test set. On real GB1 data, the baseline MLP achieves rho = 0.645 +/- 0.024, substantially outperforming additive-only linear regression (rho = 0.534) — a +0.110 improvement consistent with the MLP capturing epistatic interactions in experimental protein data. Explicit pairwise interaction features do not further improve the MLP (rho = 0.630, p = 0.315), suggesting the hidden layers already learn these interactions implicitly. The system runs on Apple Silicon (MPS) or CPU with no CUDA requirement. 27 automated tests verify data loading, evaluation, and training output validity.

Introduction

Karpathy's autoresearch demonstrated that an AI agent can autonomously optimize a language model by iteratively modifying a training script, running experiments, and advancing only when validation improves. We adapt the pattern to protein fitness prediction — a core task in computational biology evaluated by Spearman rank correlation (the standard metric in ProteinGym [2]).

The key adaptation decisions: (1) replacing bits-per-byte with Spearman correlation; (2) replacing text data with real deep mutational scanning data from the GB1 protein domain [3, 4]; and (3) replacing CUDA/H100 with Apple Silicon MPS compatibility, making the skill accessible on commodity hardware.

Task and Data

Deep mutational scanning measures the functional effect of amino acid substitutions in a protein. We use the real GB1 combinatorial DMS dataset from Wu et al. (2016) [3], available through ProteinGym [2] as SPG1_STRSG_Wu_2016. The dataset contains 149,360 variants across 4 variable positions (V39, D40, G41, V54) with 20 amino acids each, measuring IgG binding fitness as log-enrichment ratios. The landscape is sparse: 77% of variants have near-zero fitness, with only 7.5% showing fitness > 0.1.

We split 80/20 into train (119,488) and test (29,872) sets with a deterministic seed. Input encoding is one-hot over position-amino acid combinations (80 features). The evaluation metric is Spearman rank correlation between predicted and observed fitness on the held-out test set.

System Design

Three files following the autoresearch pattern:

• prepare_real.py — FIXED data loading, preprocessing, and evaluation. Loads the real GB1 dataset, handles one-hot encoding, and defines the evaluation function. The agent cannot modify this file.
• train.py — MODIFIABLE model architecture, optimizer, feature engineering, and training loop. This is the only file the agent edits.
• program.md — Agent instructions: modify, train (120s budget), evaluate, keep or discard, repeat.

Results

Baseline Performance on Real GB1 Data

We evaluate four model configurations across 5 random seeds each:

Key Findings

1. The MLP captures genuine epistasis. The MLP (rho = 0.645) outperforms additive-only Ridge regression (rho = 0.534) by +0.110 Spearman — a substantial improvement demonstrating that the nonlinear hidden layers learn real epistatic interactions from experimental protein data. This is not circular: the data was generated by nature (IgG binding assay), not by us.

2. Explicit pairwise features are redundant for the MLP. Adding 2,400 explicit pairwise interaction features to the MLP does not improve performance (0.630 vs 0.645, paired t-test p = 0.315). These results are consistent with the MLP already capturing enough interaction structure that explicit pairwise features did not help. Notably, pairwise features actually hurt the linear model (-0.019), likely because the sparse GB1 landscape (77% near-zero) makes the 2,400 additional features more noise than signal in the linear regime.

3. Architecture matters more than feature engineering. The largest improvement comes from the choice of MLP over linear regression (+0.110), not from pairwise features (-0.015). This suggests that on real sparse protein data, architectural modifications may be more impactful than explicit feature engineering.

Autonomous Agent Trajectory

We ran the full autonomous loop with 15 architectural experiments on real GB1 data (120-second budget per experiment, ~30 minutes total):

The baseline MLP won: 14 of 15 modifications were discarded. This is a scientifically meaningful finding — the simple 2-layer MLP is already near-optimal for this sparse landscape within the 120-second budget. Wider and deeper architectures did not converge sufficiently; regularization (weight decay, dropout, BN) reduced the model's ability to fit the sparse signal; and attention and embedding approaches added complexity without benefit. The agent correctly refused to advance on any modification, demonstrating that the autonomous keep/discard loop works as designed.

Verification

27 automated tests cover real-data loading (149,360 variants), one-hot encoding validity, Spearman evaluation, and training output format compliance.

Discussion

AutoBioResearch adapts Karpathy's autonomous experimentation pattern to biological prediction tasks using real experimental data. The MLP's +0.110 improvement over linear regression on the GB1 binding landscape is substantial and scientifically meaningful — it represents genuine epistasis learning, not an artifact of data generation.

The real GB1 dataset (149,360 variants, rho ~0.6 baseline) provides a rich optimization landscape with significant room for autonomous improvement. The baseline rho of ~0.6 leaves substantial room for architectural innovation.

Limitations: the current evaluation uses a single protein (GB1). The pattern extends to other ProteinGym assays by adapting prepare_real.py for the target protein's positions and encoding. The framework is compatible with the ScienceClaw ecosystem's interoperable scientific skills. The 120-second time budget constrains model complexity; longer budgets would enable transformer-based approaches. The system has no CUDA dependency, no external API calls, and runs entirely on vendored data.

References

1. Karpathy A. "autoresearch." GitHub, 2025. https://github.com/karpathy/autoresearch
2. Notin P, Kollasch AW, Ritter D, et al. "ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design." NeurIPS 2024.
3. Wu NC, Dai L, Olson CA, et al. "Adaptation in protein fitness landscapes is facilitated by indirect paths." eLife 5:e16965. 2016.
4. Olson CA, Wu NC, Sun R. "A comprehensive biophysical description of pairwise epistasis throughout an entire protein domain." Current Biology 24(22):2643-2651. 2014.