Introduction

As the AI community develops increasingly sophisticated benchmarks to evaluate LLMs, a natural question arises: what makes a benchmark question hard? If difficulty could be predicted from structural properties of the question text alone, this would have important implications for benchmark design, test-set curation, and understanding model capabilities.

Prior work has applied Item Response Theory (IRT) to estimate per-question difficulty from LLM evaluation results [wang2024easy2hard]. However, these approaches require running many models on each question. We ask the complementary question: can we predict IRT difficulty scores from text features alone, without any model evaluation?

We focus on the ARC-Challenge benchmark [clark2018think], a standard science reasoning dataset, using IRT difficulty scores from the Easy2Hard-Bench dataset [wang2024easy2hard], which estimates difficulty from thousands of LLM evaluations on the Open LLM Leaderboard.

Methods

Data

We use the E2H-ARC split of Easy2Hard-Bench [wang2024easy2hard], containing 1,172 ARC-Challenge questions with IRT difficulty ratings normalized to $[0, 1]$. Difficulty scores were estimated using IRT and Glicko-2 models from per-question accuracy data across thousands of LLMs. For reproducibility, we pin the HuggingFace dataset revision to commit 55bc0d2fb10954151e669d2026b87fa896f2fa26.

Feature Extraction

For each question, we extract 12 structural features without running any LLM:

- **question\_length**: Character count of question text
- **word\_count**: Number of words in the question
- **avg\_word\_length**: Mean word length (vocabulary complexity proxy)
- **answer\_entropy**: Shannon entropy over answer option character lengths
- **num\_choices**: Number of answer options (3, 4, or 5)
- **lexical\_overlap**: Jaccard similarity between question and answer words
- **negation\_count**: Count of negation words (not, never, except, etc.)
- **question\_type**: Encoded question starter (what/which/how/why/etc.)
- **flesch\_kincaid\_grade**: Readability grade level
- **unique\_word\_ratio**: Lexical diversity (unique/total words)
- **max\_option\_length\_ratio**: Ratio of longest to shortest answer option
- **stem\_overlap**: Jaccard similarity between question and correct answer

Analysis

We compute Spearman rank correlations between each feature and IRT difficulty. We train a Random Forest regressor (100 trees, max depth 10, min samples leaf 3, seed 42) and evaluate using 5-fold cross-validation with three metrics: $R^2$, mean absolute error (MAE), and Spearman $\rho$. To calibrate whether any signal is meaningful, we also evaluate a dummy mean-prediction baseline on the same folds. Finally, we compute out-of-fold Spearman correlation and run a label-permutation test (200 permutations) to assess whether observed rank correlation exceeds chance.

Results

Feature Correlations

Table shows Spearman correlations between structural features and IRT difficulty. Only negation_count achieves statistical significance ($p < 0.05$), with a weak positive correlation ($\rho = 0.071$). All other features have $|\rho| < 0.04$. However, applying a Bonferroni correction for 12 simultaneous tests yields an adjusted threshold of $p < 0.004$; no feature survives this correction, strengthening the negative-result interpretation.

*Spearman correlations between structural features and IRT difficulty ($n = 1,*172$).}

Prediction Performance

The Random Forest model achieves high training $R^2$ (0.510) but near-zero cross-validated performance (Table), indicating severe overfitting. Relative to the dummy baseline, the random forest provides only modest gains (MAE improvement: 0.227 to 0.223). The out-of-fold Spearman is $\rho = 0.129$; permutation testing yields $p = 0.005$, indicating the signal is statistically non-zero but still weak in practical terms.

Difficulty prediction model performance.

Feature Importance

The Random Forest feature importances (Mean Decrease in Impurity) rank Flesch-Kincaid grade (0.156), average word length (0.128), and answer entropy (0.121) as the top three features. However, given the near-zero cross-validated $R^2$, these importances primarily reflect overfitting patterns rather than generalizable predictive signal.

Discussion

Our key finding is a negative result: structural features of benchmark questions cannot meaningfully predict which questions are difficult for LLMs. The cross-validated $R^2$ of 0.007 means that structural features explain less than 1% of the variance in difficulty. Although permutation testing indicates the rank signal is statistically non-zero, the absolute effect size remains small and practically weak. This is consistent with the intuition that LLM difficulty arises from reasoning demands—the depth of inference, domain knowledge, and logical chains required—rather than from surface properties like question length or vocabulary complexity.

The one significant finding—negation count's positive correlation with difficulty—aligns with known challenges LLMs face with negation. Questions containing "NOT," "except," or "never" require negating a default inference, adding a reasoning step that structural features can detect.

Implications

- **Benchmark design:** Surface-level filtering (by length, readability, etc.) cannot substitute for empirical difficulty estimation via model evaluation.
- **IRT validation:** The fact that IRT scores are not predictable from text properties supports the validity of IRT-based difficulty estimation as capturing genuine reasoning difficulty.
- **Feature engineering:** Future work on difficulty prediction should focus on semantic features (e.g., knowledge graph distance, reasoning chain depth) rather than structural properties.

Limitations

- IRT difficulty scores are derived from LLM (not human) performance.
- Only ARC-Challenge was analyzed; results may differ for other benchmarks.
- Our 12 features are a subset of possible structural features; additional text features (e.g., dependency parse depth, named entity count) might improve prediction.
- The Random Forest may not capture all non-linear relationships; other models could perform better.

Conclusion

We provide quantitative evidence that structural features of multiple-choice questions—including question length, answer entropy, lexical overlap, and readability—cannot predict LLM difficulty ($R^2 = 0.007$, Spearman $\rho = 0.127$ cross-validated). This negative result underscores that benchmark difficulty is a semantic property, not a structural one, and supports the continued use of model-based difficulty estimation methods like IRT for benchmark curation.

\bibliographystyle{plainnat}

References

• [clark2018think] Clark, P., Cowhey, I., Etzioni, O., et al. (2018). Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457.

• [wang2024easy2hard] Wang, J., et al. (2024). Easy2Hard-Bench: Standardized difficulty labels for profiling LLM performance and generalization. NeurIPS 2024, Datasets and Benchmarks Track.

• [ethayarajh2022understanding] Ethayarajh, K. & Choi, Y. (2022). Understanding dataset difficulty with $\mathcal{V}$-usable information. ICML 2022.