Introduction

The evaluation of large language models (LLMs) has converged on a small set of standard benchmarks. The Open LLM Leaderboard v1 [open-llm-leaderboard] evaluated over 7,000 models on six benchmarks: ARC-Challenge [clark2018arc], HellaSwag [zellers2019hellaswag], MMLU [hendrycks2021mmlu], WinoGrande [sakaguchi2020winogrande], TruthfulQA [lin2022truthfulqa], and GSM8K [cobbe2021gsm8k].

A natural question arises: Are these benchmarks measuring distinct capabilities, or are they largely redundant? If a model scores well on ARC, can we predict its MMLU score? How many independent dimensions of "LLM ability" do these benchmarks actually capture?

We address these questions using published benchmark scores for 40 models across 11 families, employing correlation analysis, PCA, hierarchical clustering, and greedy subset selection. Our analysis requires no model inference—all data is hardcoded from published sources.

Data

We collected benchmark scores for 40 base (pre-trained) models from the Open LLM Leaderboard v1, cross-referenced with original papers where available. Model families include Llama-1/2 [touvron2023llama, touvron2023llama2], Mistral [jiang2023mistral], Falcon [falcon], Pythia [biderman2023pythia], OPT [zhang2022opt], GPT-NeoX [black2022gptneox], GPT-Neo, Cerebras-GPT [dey2023cerebrasgpt], MPT, and StableLM. Model sizes range from 70M to 70B parameters.

All six benchmarks use the same evaluation harness (EleutherAI lm-evaluation-harness) with standardized shot settings: ARC-Challenge (25-shot), HellaSwag (10-shot), MMLU (5-shot), WinoGrande (5-shot), TruthfulQA (0-shot), GSM8K (5-shot).

Methods

Correlation Analysis. We compute both Pearson (linear) and Spearman (rank) correlation matrices between all 15 benchmark pairs.

PCA. We standardize scores (zero mean, unit variance) and compute principal components to determine the effective dimensionality of the benchmark space.

Hierarchical Clustering. We cluster benchmarks using average linkage on a correlation-distance matrix ($d_{ij} = 1 - |r_{ij}|$).

Greedy Forward Selection. We iteratively select the benchmark that maximizes total variance explained (via OLS regression) to identify the minimal informative subset.

Model Family Analysis. We project models into PC space and compute silhouette scores, intra/inter-family distances, and PC1-vs-$\log(\text{params})$ correlation.

Bootstrap Robustness. We perform 400 model-level bootstrap resamples (sampling models with replacement) and recompute key statistics: pairwise Pearson correlations, $n_{90}$ (components needed for 90% variance), PC1-vs-$\log(\text{params})$ correlation, and greedy top-2 benchmark subsets. We report 95% percentile confidence intervals and subset-selection frequencies.

Results

Correlation Structure

Pearson correlation matrix. ARC-C = ARC-Challenge. Bold: |r| > 0.9.

Key pairs with $r > 0.95$: ARC-Challenge--WinoGrande ($r = 0.985$), HellaSwag--WinoGrande ($r = 0.971$), and MMLU--GSM8K ($r = 0.967$). TruthfulQA is the only benchmark with negative correlations to most others, reflecting its measurement of a fundamentally different property (resistance to common misconceptions).

Principal Component Analysis

PC1 captures 74.0% of variance and loads approximately equally on ARC-C, HellaSwag, MMLU, WinoGrande, and GSM8K—it represents "general LLM ability," which scales with model size ($r = 0.86$ with $\log N$). PC2 captures 23.4% and loads primarily on TruthfulQA ($+0.80$), reflecting its orthogonal measurement axis. Together, 2 components explain 97.4% of variance; 3 components reach 99.1%.

Clustering

Hierarchical clustering at $k=2$ isolates TruthfulQA from all other benchmarks. At $k=3$, benchmarks split into: (1) knowledge/math: MMLU, GSM8K; (2) reasoning/commonsense: ARC-C, HellaSwag, WinoGrande; (3) truthfulness: TruthfulQA.

Minimal Benchmark Subsets

Greedy selection yields: (1) ARC-Challenge alone explains 72.9% of variance; (2) adding TruthfulQA reaches 95.4%; (3) adding GSM8K reaches 98.2%. This means 3 of 6 benchmarks capture 98% of the information, and even 2 benchmarks suffice for 95%.

Robustness Under Resampling

Bootstrap analysis confirms that the main conclusions are not artifacts of the specific 40-model sample. ARC-Challenge--WinoGrande remains strongly correlated with 95% CI [0.974, 0.991], and the PC1-vs-$\log(\text{params})$ correlation remains high with 95% CI [0.783, 0.921]. In all 400 bootstrap runs, exactly 2 components were sufficient to exceed 90% variance explained. ARC-Challenge + TruthfulQA was the most frequent top-2 subset (94.8% of runs), indicating stable benchmark-pruning recommendations.

Model Scale Dominance

The negative silhouette score ($-0.29$) for model families in PC space confirms that model size matters more than architecture. A Pythia-12B is closer to an OPT-13B than to a Pythia-70M, because PC1 (scale) dominates over family-specific effects.

Discussion

Our finding that 2 PCs explain 97% of variance aligns with prior work suggesting benchmark saturation [open-llm-leaderboard-blog]. The practical implication is clear: researchers can evaluate on just ARC-Challenge and TruthfulQA to capture 95% of the information from all six benchmarks, reducing evaluation cost by 67%. The bootstrap stability results strengthen this recommendation by showing narrow confidence intervals for core correlations and high repeatability of the top-2 subset.

The two independent dimensions we identify correspond to (1) general capability scaling with model size, and (2) truthfulness/calibration, which does not scale reliably with size. This suggests that future benchmark suites should prioritize orthogonal measurements rather than adding more correlated reasoning tasks.

Limitations. Our analysis uses scores from the Open LLM Leaderboard, which may have minor evaluation inconsistencies across submissions. We include only base models; instruction-tuned models may show different patterns. The 40-model sample, while diverse in scale and family, cannot capture all architecture types (e.g., mixture-of-experts, encoder-decoder).

Conclusion

Standard LLM benchmarks are highly redundant: 2 principal components explain 97.4% of cross-benchmark variance for 40 models across 11 families. The first component captures scale-dependent "general ability," while the second captures TruthfulQA's orthogonal signal. A minimal evaluation suite of ARC-Challenge, TruthfulQA, and optionally GSM8K recovers 98% of the information in all six benchmarks.

\bibliographystyle{plainnat}

References

• [biderman2023pythia] S. Biderman et al. Pythia: A suite for analyzing large language models across training and scaling. ICML, 2023.

• [black2022gptneox] S. Black et al. GPT-NeoX-20B: An open-source autoregressive language model. BigScience Workshop, 2022.

• [clark2018arc] P. Clark et al. Think you have solved question answering? Try ARC. arXiv:1803.05457, 2018.

• [cobbe2021gsm8k] K. Cobbe et al. Training verifiers to solve math word problems. arXiv:2110.14168, 2021.

• [dey2023cerebrasgpt] N. Dey et al. Cerebras-GPT: Open compute-optimal language models. arXiv:2304.03208, 2023.

• [falcon] TII. The Falcon has landed in the Hugging Face ecosystem. Hugging Face Blog, 2023.

• [hendrycks2021mmlu] D. Hendrycks et al. Measuring massive multitask language understanding. ICLR, 2021.

• [jiang2023mistral] A. Jiang et al. Mistral 7B. arXiv:2310.06825, 2023.

• [lin2022truthfulqa] S. Lin et al. TruthfulQA: Measuring how models mimic human falsehoods. ACL, 2022.

• [open-llm-leaderboard] HuggingFace. Open LLM Leaderboard v1 (archived 2023--2024). \url{https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard}, 2024.

• [open-llm-leaderboard-blog] HuggingFace. Open-LLM performances are plateauing. \url{https://huggingface.co/spaces/open-llm-leaderboard/blog}, 2024.

• [sakaguchi2020winogrande] K. Sakaguchi et al. WinoGrande: An adversarial Winograd schema challenge at scale. AAAI, 2020.

• [touvron2023llama] H. Touvron et al. LLaMA: Open and efficient foundation language models. arXiv:2302.13971, 2023.

• [touvron2023llama2] H. Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023.

• [zellers2019hellaswag] R. Zellers et al. HellaSwag: Can a machine really finish your sentence? ACL, 2019.

• [zhang2022opt] S. Zhang et al. OPT: Open pre-trained transformer language models. arXiv:2205.01068, 2022.