Introduction

Classical statistical learning theory predicts a U-shaped bias-variance tradeoff: as model complexity increases, test error first decreases (reducing bias) then increases (due to variance). Modern deep learning practice contradicts this—very large, overparameterized models generalize well despite having far more parameters than training samples.

Belkin et al.[belkin2019reconciling] reconciled these observations by identifying the double descent curve, which subsumes the classical U-shape. The curve exhibits three regimes: (1) the classical regime where increasing capacity reduces error, (2) a critical peak at the interpolation threshold where the model has just enough capacity to fit the training data, and (3) the modern regime where further overparameterization yields smoother interpolating solutions.

Nakkiran et al.[nakkiran2019deep] demonstrated that double descent occurs not only as a function of model size, but also as a function of training epochs, and showed that label noise amplifies the phenomenon.

In this work, we reproduce these phenomena using a clean experimental setup: random ReLU features models with minimum-norm least-squares fitting on synthetic regression data. This setup, inspired by the theoretical framework of Advani & Saxe[advani2017high], provides an ideal testbed because the interpolation threshold is exactly at $p = n$ (number of features = number of training samples), and the solution is computed in closed form.

Methods

Data Generation

We generate synthetic regression data: $\mathbf{X} \in \mathbb{R}^{n \times d}$ with entries drawn from $\mathcal{N}(0, 1)$, true weights $\mathbf{w}^* ~ \mathcal{N}(0, 1)$, and targets $\mathbf{y} = \mathbf{X}\mathbf{w}^* + \epsilon$ where $\epsilon ~ \mathcal{N}(0, \sigma^2)$. We use $n_{\text{train}} = 200$, $n_{\text{test}} = 200$, and $d = 20$, with noise levels $\sigma \in {0.1, 0.5, 1.0}$.

Random Features Model

We employ a two-layer model with a fixed random first layer: $$\hat{y} = \Phi(\mathbf{X}) \beta, \text{where} \Phi(\mathbf{X}) = \text{ReLU}(\mathbf{X}\mathbf{W} + \mathbf{b})$$ Here $\mathbf{W} \in \mathbb{R}^{d \times p}$ and $\mathbf{b} \in \mathbb{R}^p$ are fixed random projections, and $\beta \in \mathbb{R}^{p}$ is fit via minimum-norm least squares: $$\beta = \Phi^{\dagger} \mathbf{y}$$ where $\Phi^{\dagger}$ is the Moore-Penrose pseudoinverse. The number of trainable parameters is exactly $p$.

Experimental Design

Model-wise sweep. We vary $p$ from 10 to 1000 (24 values), with dense sampling near the interpolation threshold $p = n = 200$. For each $p$, we compute train and test MSE. This is repeated at three noise levels.

MLP comparison. For comparison, we train 2-layer MLPs with varying hidden width $h$ using Adam optimization (lr=0.001, 4000 epochs, no regularization).

Variance estimation. We repeat the random features sweep with 3 different random seeds to quantify variability.

Reproducibility controls. All dependencies are version-pinned in requirements.txt, every stochastic component is seeded, and the pipeline emits a SHA-256 fingerprint of scientific outputs. The validator recomputes this fingerprint from results.json to catch stale or corrupted artifacts before claims are made.

Results

Model-Wise Double Descent

Our experiments reveal a dramatic double descent curve. At low noise ($\sigma = 0.1$), test MSE drops from 10.0 at $p = 10$ to 1.3 at $p = 140$, then spikes to 312.0 at $p = 200$ (the interpolation threshold), before decreasing to 0.11 at $p = 1000$. This represents a peak-to-minimum ratio of approximately $2,938 \times$.

At higher noise ($\sigma = 1.0$), the absolute peak is even larger (1,573) though the ratio is somewhat lower ($564\times$) because the baseline test error is higher. This confirms that label noise amplifies the interpolation peak in absolute terms.

Train-Test Decomposition

Training MSE decreases monotonically with $p$ and reaches exactly zero at $p = n = 200$. This is expected: at the threshold, the system of equations $\Phi\beta = \mathbf{y}$ is exactly determined (assuming $\Phi$ has full rank). For $p > n$, the system is underdetermined and the minimum-norm solution achieves zero training error.

The critical insight is that the unique interpolating solution at $p = n$ is typically highly irregular, while the minimum-norm solution for $p > n$ is smoother. This explains the test error peak at $p = n$.

MLP Comparison

Trained MLPs show a qualitatively similar pattern but with a less pronounced peak. The MLP test error peaks near $h = 16$ (where #\text{params} \approx n) and gradually decreases for larger widths. The gentler peak is attributable to Adam's implicit regularization.

Discussion

Our results provide a clean, fast, and reproducible demonstration of the double descent phenomenon. The random features setup is ideal for this purpose because: (1) the interpolation threshold is exactly at $p = n$, (2) the solution is computed analytically via pseudoinverse, (3) the entire experiment runs in seconds, and (4) the effect is extremely pronounced (ratios of 500--3000$\times$).

Limitations. Our setup uses synthetic linear-in-features regression, which is a simplified model. Real deep learning architectures exhibit double descent with additional complexities such as optimization dynamics, implicit regularization from SGD, and non-linear feature learning. Our MLP comparison partially addresses this gap. Additionally, epoch-wise double descent may not manifest with tiny MLPs and the Adam optimizer; it typically requires larger models, SGD, and longer training[nakkiran2019deep].

Broader implications. The double descent phenomenon has important practical implications: (1) the traditional approach of selecting model complexity via a validation set may miss the overparameterized regime, (2) adding more parameters can improve rather than hurt generalization, and (3) the interpolation threshold is a dangerous regime to be avoided in practice.

\bibliographystyle{plainnat}

References

• [belkin2019reconciling] M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849--15854, 2019.

• [nakkiran2019deep] P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever. Deep double descent: Where bigger models and more data can hurt. arXiv preprint arXiv:1912.02292, 2019.

• [advani2017high] M. S. Advani and A. M. Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXiv:1710.03667, 2017.