Introduction

Shortcut learning occurs when models exploit spurious correlations in training data that do not generalize to deployment[geirhos2020shortcut]. Classic examples include texture bias in image classifiers[geirhos2018imagenet] and annotation artifacts in NLP[gururangan2018annotation]. Understanding when and why models prefer shortcuts over genuine features is critical for building reliable AI systems.

We construct a minimal, fully controlled setting that isolates the shortcut learning phenomenon: synthetic Gaussian classification with an appended binary shortcut feature. This allows precise measurement of shortcut reliance through feature ablation—comparing model performance with and without the shortcut at test time.

Our contributions:

• A reproducible experimental framework for shortcut detection with synthetic data.
• Quantification of shortcut reliance across model capacities and regularization strengths.
• Evidence that only sufficiently strong L2 regularization reduces shortcut dependence, while over-regularization can suppress learning altogether.

Method

Data Generation

We generate binary classification data with $d = 10$ genuine features and 1 shortcut feature ($d_{\text{total}} = 11$). For class $k \in {0, 1}$, genuine features are drawn from $\mathcal{N}(\mu_k, \mathbf{I})$, where $\mu_0$ and $\mu_1$ are randomly generated with moderate separation.

The shortcut feature $x_{11}$ is constructed as:

• Training: $x_{11} = y$ (perfect correlation with label).
• Test (with shortcut): $x_{11} = y$ (still correlated).
• Test (without shortcut): $x_{11} ~ \text{Bernoulli}(0.5)$ (randomized).

We use $n_{\text{train}} = 2000$ and $n_{\text{test}} = 1000$.

Model and Training

We use a 2-layer MLP: $\text{Linear}(11, h) \to \text{ReLU} \to \text{Linear}(h, h) \to \text{ReLU} \to \text{Linear}(h, 2)$, where $h \in {32, 64, 128}$.

Training uses Adam with learning rate $0.01$, batch size 128, and 100 epochs. We sweep weight decay $\lambda \in {0, 0.001, 0.01, 0.1, 1.0}$.

Shortcut Reliance Metric

We define shortcut reliance as: [ R = \text{Acc}{\text{test, with shortcut}} - \text{Acc}{\text{test, without shortcut}} ] A large $R > 0$ indicates the model depends on the spurious shortcut. Values near $R \approx 0$ are only meaningful when accuracy remains above chance; otherwise they may simply indicate that the model failed to learn anything useful.

Experimental Design

Full factorial sweep: 3 hidden widths $\times$ 5 weight decays $\times$ 3 random seeds = 45 runs. We report mean $\pm$ standard deviation across seeds.

Results

All 45 configurations were trained on CPU in under 3 minutes.

Shortcut reliance without regularization. With weight decay $= 0$, models across all widths show substantial shortcut reliance, confirming that neural networks preferentially learn the spurious feature when it is a simpler predictor of the label.

Effect of weight decay. Increasing weight decay does not help uniformly. In our runs, weight decay values of 0.001 and 0.01 leave shortcut reliance essentially unchanged, weight decay 0.1 materially reduces it, and weight decay 1.0 drives reliance to zero only because the model remains near chance accuracy. L2 regularization can therefore mitigate shortcut use, but only in a narrow regime between under- and over-regularization.

Model width. The effect of model width on shortcut reliance is secondary to regularization. All three widths (32, 64, 128) exhibit qualitatively similar patterns.

Generalization accuracy. On the test set without the shortcut (the "honest" evaluation), the clearest gains come from weight decay 0.1, which improves average accuracy relative to the unregularized baseline. Weight decay 0.01 offers only marginal improvement, underscoring how narrow the helpful regularization regime is in this setup.

Discussion

Our results align with prior work showing that neural networks are biased toward simple, high-correlation features[geirhos2020shortcut, shah2020pitfalls]. The synthetic setting offers several advantages: (1) ground truth is known (we control which feature is spurious), (2) experiments are fast and fully reproducible, and (3) the framework is easily extended to test other mitigation strategies.

Limitations. Our synthetic data is low-dimensional and the shortcut is a single binary feature. Real-world shortcuts (e.g., background textures, demographic artifacts) are often more subtle and distributed across many features. Additionally, we only test L2 regularization; other methods such as group DRO[sagawa2020distributionally], Just Train Twice[liu2021just], and invariant risk minimization[arjovsky2019invariant] may be more effective.

Extensions. The framework can be extended to: (a) multiple simultaneous shortcuts, (b) partial (non-perfect) correlations, (c) deeper architectures, and (d) real-world spurious correlation benchmarks such as Waterbirds and CelebA.

Conclusion

We present a controlled experimental framework for detecting shortcut learning in neural networks. Through feature ablation on synthetic data, we confirm that models preferentially exploit spurious shortcuts, and that only sufficiently strong L2 regularization reduces this dependence without collapsing learning. The full experiment is packaged as an executable AI-agent skill for reproducibility.

\bibliographystyle{plainnat}

References

• [geirhos2020shortcut] R. Geirhos, J. H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665--673, 2020.

• [geirhos2018imagenet] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. {ImageNet}-trained {CNN}s are biased towards textures; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231, 2018.

• [gururangan2018annotation] S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. Bowman, and N. A. Smith. Annotation artifacts in natural language inference data. In Proc. NAACL, pages 107--112, 2018.

• [shah2020pitfalls] H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and P. Netrapalli. The pitfalls of simplicity bias in neural networks. In Advances in Neural Information Processing Systems, 2020.

• [sagawa2020distributionally] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.

• [liu2021just] E. Z. Liu, B. Haghgoo, A. S. Chen, A. Raghunathan, P. W. Koh, S. Sagawa, P. Liang, and C. Finn. Just train twice: Improving group robustness without training group information. In ICML, 2021.

• [arjovsky2019invariant] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.