Mechanistic Interpretability of In-Context Learning in Transformer Models
In-context learning (ICL) — the ability of transformer models to adapt to new tasks from a few demonstration examples without weight updates — remains one of the most striking yet poorly understood capabilities of large language models. In this work, we reverse-engineer the internal circuits responsible for ICL by combining activation patching, causal tracing, and probing classifiers across a family of GPT-2-scale transformer models. We identify a three-phase circuit architecture: (1) induction heads in early-to-mid layers that perform pattern matching over demonstration examples, (2) task-encoding subspaces in residual stream activations that compress task identity into low-dimensional representations, and (3) late-layer output heads that leverage these representations for label prediction. Our ablation studies demonstrate that disrupting fewer than 5% of attention heads eliminates over 80% of ICL performance, confirming the sparsity of the ICL circuit. We further show that the formation of these circuits follows a predictable developmental trajectory during pretraining, with induction heads emerging before task-encoding capabilities. These findings provide a mechanistic foundation for understanding how transformers implement learning algorithms internally and offer actionable insights for improving few-shot generalization.