Introduction

[wei2022] documented 137 tasks where large language model (LLM) performance appeared to exhibit "emergent abilities"—capabilities absent in smaller models that appear sharply at a critical scale. This observation suggested that scaling alone could produce qualitatively new capabilities, with profound implications for AI safety and development.

[schaeffer2023] challenged this interpretation, arguing that the appearance of emergence is primarily an artifact of evaluation metrics. They demonstrated that over 92% of claimed emergent abilities are measured by just two metrics—Exact String Match and Multiple Choice Grade—both of which are discontinuous. When continuous metrics such as Token Edit Distance are applied to the same model outputs, performance improves smoothly and predictably with scale.

We provide an independent re-analysis of this claim using hardcoded published benchmark data, requiring no model inference or API access. Our analysis introduces the Metric Sensitivity Index (MSI), a quantitative measure of how much apparent nonlinearity is attributable to metric choice versus genuine capability transitions.

Methods

Data

We hardcode published performance data from:

- **BIG-Bench**: 8 tasks (2-digit multiplication, 4-digit addition, IPA transliteration, word unscrambling, Persian QA, sports understanding, modified arithmetic, word sorting) across GPT-3, InstructGPT, LaMDA, and PaLM model families [srivastava2023, wei2022].
- **MMLU**: 13 models from 5 families (GPT-3, PaLM, LLaMA, Chinchilla, Gopher) [hendrycks2021, touvron2023, chowdhery2022, hoffmann2022].

All accuracy values are in $[0, 1]$ and parameter counts in billions (B).

Metric Transformation

Following [schaeffer2023], we model the relationship between per-token accuracy $p$ and exact-match accuracy as: $$\text{ExactMatch} = p^n$$ where $n$ is the number of tokens in the answer. This assumes token-level independence. The inverse yields the inferred per-token accuracy: $$p = \text{ExactMatch}^{1/n}$$

We define two continuous metrics:

- **Partial Credit**: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>PC</mtext><mo>=</mo><mi>p</mi></mrow><annotation encoding="application/x-tex">\text{PC} = p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord text"><span class="mord">PC</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">p</span></span></span></span> (fraction of tokens correct)
- **Token Edit Distance**: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>TED</mtext><mo>=</mo><mi>n</mi><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\text{TED} = n(1-p)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord text"><span class="mord">TED</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">n</span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">p</span><span class="mclose">)</span></span></span></span> (expected errors)

Nonlinearity Detection

For each task, we fit both linear and logistic sigmoid models to performance vs.\ $\log_{10}(\text{parameters})$ under both discontinuous and continuous metrics. We compare fits using $R^2$ and define the Metric Sensitivity Index: $$\text{MSI} = \frac{(R^2_{\text{sig}} - R^2_{\text{lin}})$${\text{disc}}}{(R^2{\text{sig}} - R^2_{\text{lin}})_{\text{cont}}}

High MSI ($> 2$) indicates the nonlinearity is primarily a metric artifact. MSI $\leq 2$ suggests potentially genuine nonlinear scaling.

To quantify uncertainty, we run 120 deterministic bootstrap resamples per task and report (i) a 95% confidence interval for MSI and (ii) $P(\text{MSI}>2)$. We classify a task as likely artifact only when both conditions hold: MSI $>2$ and $P(\text{MSI}>2)\geq 0.8$.

Synthetic Demonstration

We generate synthetic data where per-token accuracy improves linearly with $\log_{10}(\text{parameters})$ from $p=0.3$ to $p=0.95$ across 20 simulated model sizes. We then apply both exact-match ($p^5$) and partial-credit scoring to demonstrate how the nonlinear mapping creates apparent emergence.

Results

Metric Sensitivity Index

Of 8 BIG-Bench tasks analyzed, point estimates exceed MSI $>2$ for 7 tasks. However, after bootstrap uncertainty quantification, only 4 tasks satisfy our likely-artifact criterion (MSI $>2$ and $P(\text{MSI}>2)\geq 0.8$), while 3 tasks remain uncertainty-limited and 1 task is definitional due to single-token outputs (Table).

Metric Sensitivity Index for BIG-Bench tasks with deterministic bootstrap uncertainty (120 resamples).

Synthetic Demonstration

The synthetic demonstration confirms the core mechanism. With 5-token answers and linearly improving per-token accuracy ($p = 0.30 \to 0.95$):

- Partial credit improves smoothly from 0.30 to 0.95
- Exact match (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>p</mi><mn>5</mn></msup></mrow><annotation encoding="application/x-tex">p^5</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0085em;vertical-align:-0.1944em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">5</span></span></span></span></span></span></span></span></span></span></span>) shows near-zero performance below <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>≈</mo><mn>0.7</mn></mrow><annotation encoding="application/x-tex">p \approx 0.7</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6776em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">p</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.7</span></span></span></span>, then rises sharply
- The same underlying improvement appears as smooth scaling or dramatic emergence depending solely on the metric

MMLU Scaling

MMLU accuracy (5-shot, multiple-choice) scales relatively smoothly across all model families. Linear $R^2$ values are high for within-family scaling (GPT-3: $R^2 > 0.9$; LLaMA: $R^2 > 0.99$), consistent with the absence of phase transitions when using a more continuous evaluation metric.

Discussion

Our re-analysis provides evidence consistent with [schaeffer2023]: several apparent emergent abilities are metric artifacts caused by the nonlinear mapping $p \to p^n$ inherent in exact-match scoring. Under uncertainty-aware criteria, 4 tasks are likely artifacts, 3 remain uncertainty-limited, and 1 is definitional. The Metric Sensitivity Index, combined with bootstrap support, provides a more conservative framework for distinguishing genuine capability transitions from metric artifacts.

However, we note several important caveats:

- **Sparse data**: With only 3--14 model sizes per task, curve-fitting comparisons have limited statistical power and wide MSI bootstrap intervals.
- **Token independence**: Our per-token accuracy inference assumes independence, which may not hold for reasoning-intensive tasks.
- **Aggregated scores**: We use published accuracy values, not raw model outputs, preventing direct verification of the per-token distribution.
- **Hardcoded data**: All data is transcribed from published figures and tables, introducing potential transcription error.

Sports understanding should not be treated as counterevidence to the metric-artifact hypothesis: with a single-token output, exact match and partial credit are the same metric. This means our current MSI analysis cannot adjudicate whether that task contains genuine nonlinearity, and future work should use task-level metrics that remain informative when $n=1$.

Conclusion

We partially confirm the central finding of [schaeffer2023] using an independent re-analysis: MSI point estimates are high in 7 of 8 BIG-Bench tasks, and 4 tasks remain likely artifacts under bootstrap support thresholds while 3 are uncertainty-limited. The Metric Sensitivity Index with uncertainty reporting provides a useful quantitative tool for future studies of scaling behavior, and our fully reproducible analysis requires no model inference.

\bibliographystyle{plainnat}

References

• [chowdhery2022] Chowdhery, A., et al. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311.

• [hendrycks2021] Hendrycks, D., et al. (2021). Measuring Massive Multitask Language Understanding. ICLR 2021. arXiv:2009.03300.

• [hoffmann2022] Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.

• [schaeffer2023] Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? NeurIPS 2023. arXiv:2304.15004.

• [srivastava2023] Srivastava, A., et al. (2023). Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. arXiv:2206.04615.

• [touvron2023] Touvron, H., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.

• [wei2022] Wei, J., et al. (2022). Emergent Abilities of Large Language Models. arXiv:2206.07682.