We specify a pre-registered protocol for For reasoning tasks where published results report accuracy under 'majority-vote over 5 samples at temperature T', how sensitive are the reported accuracies to the choice of N (number of samples), temperature T, and aggregation rule (strict majority vs plurality vs weighted)? using GSM8K and MATH (Hendrycks 2021) test sets at pinned versions.
We specify a pre-registered protocol for How many problems in HumanEval and MBPP are near-duplicates of each other at a pre-specified fuzzy-match threshold on prompt, docstring, and test-case text, and does this cross-contamination bias any comparison between HumanEval-tuned and MBPP-tuned models? using the two benchmark sets in full, plus their expanded variants (HumanEval+, MBPP+) from Liu 2023.
We specify a pre-registered protocol for For three widely-used 2025-era open instruction-tuning datasets, what fraction of their examples are near-duplicates (at a pre-specified similarity threshold) of items in five widely-used evaluation suites (MMLU, GSM8K, HumanEval, MBPP, TruthfulQA)? using the three instruction datasets and five evaluation suites (all publicly available on HuggingFace) at pinned revision hashes.
We specify a pre-registered protocol for Across 12 recent papers that report HumanEval Pass@1 for a specific model, how consistent are the evaluation protocols (prompt style, temperature, post-processing, test harness version), and when all papers are re-run under a single common protocol, how do Pass@1 numbers change? using HumanEval (Chen et al.
We specify a pre-registered protocol for When the same agent framework is run on SWE-Bench Verified with the same base model weights but different inference stacks, how much does the reported Pass@1 vary, and is the variation concentrated in specific repositories or failure classes? using SWE-Bench Verified (public release at pre-registration date), patch-level evaluation harness.
We specify a pre-registered protocol for For a set of Model Context Protocol servers implementing the same tools with the same declared schemas, do three client SDKs discover and enumerate them identically, or do edge cases in tool-schema rendering, transport negotiation, and auth handling differ? using a pre-registered set of 10 reference MCP servers (stdio, SSE, and HTTP transports) implementing tools spanning simple params, nested schemas, optional/required interactions, and auth-gated endpoints.
We specify a pre-registered protocol for Given the same rendered web page and the same user instruction, what fraction of tasks result in different click targets across four browser-using agents, and do divergences correlate with DOM structure features (shadow DOM, iframes, overlaid elements)? using a pre-registered suite of 50 rendered pages including static reproductions (archived) of real web pages spanning e-commerce, forms, docs, SPAs, and pages with shadow DOM / iframes.
We specify a pre-registered protocol for Given a set of parallel workflow definitions implemented in both LangGraph and LlamaIndex, can intermediate workflow state be transferred between the two frameworks at checkpoint boundaries, and if not, what serialization features differ? using pre-registered parallel implementations of 15 workflows each in both frameworks covering RAG, tool-call chains, and branching decisions.
We specify a pre-registered protocol for When AutoGen and CrewAI agents are composed into a shared workflow with a standard task set, what concrete interoperability failures occur (tool-schema mismatch, message-format incompatibility, state serialization), and can any be solved with a thin adapter layer? using a pre-registered suite of 20 composed workflows spanning code-generation, data-retrieval, and planning, each requiring agents from both frameworks to exchange artifacts.
We specify a pre-registered protocol for For five recent papers that claim effective prompt-injection defences, can the claims be reproduced at the originally reported success rates when evaluated against a shared, pre-registered attack corpus? using pre-registered attack corpus: 300 prompt-injection attempts drawn from public red-team collections (e.
We specify a pre-registered protocol for For a schema-constrained JSON output mode from two widely-used LLM API providers, at matched temperature, matched prompt, and matched schema, what fraction of calls produce JSON outputs whose field order differs between providers even when semantic content is equivalent? using a pre-registered prompt corpus of 200 prompts with a fixed JSON schema (mixed types: strings, arrays, nested objects), published alongside the pre-registration.
We specify a pre-registered protocol for Given the same open-weights model, the same prompt, and temperature=0 settings, do three widely-used inference stacks (vLLM, llama.cpp, HuggingFace transformers) produce byte-identical completions, and if not, how do outputs diverge?
We specify a pre-registered protocol for When three published self-refine-style prompting strategies are applied as-written to a shared modern open-weights base model on the GSM8K test set, how different are their measured reasoning accuracy gains over a common baseline prompt, and are the gains within expected variance? using GSM8K test set (Cobbe et al.
We specify a pre-registered protocol for Does the top-cited pre-2024 pancreatic-cancer radiomics signature (selected by pre-specified citation criteria) replicate its reported AUC on an external, publicly accessible CT cohort when applied without retraining, using the authors' released feature definitions? using The Cancer Imaging Archive (TCIA) pancreatic CT collections (e.
We specify a pre-registered protocol for Across a pre-registered sample of 40 recent preprints claiming 'early-detection' performance for a medical AI classifier, what fraction disclose a pre-specified operating point (threshold, sensitivity/specificity target) as opposed to reporting only AUC? using arXiv, medRxiv, and bioRxiv preprint metadata and full-text PDFs, accessed via their public APIs on a pre-registered sampling date; specific preprint identifiers logged at inclusion.
We specify a pre-registered protocol for On the same patient-level tumour measurement data from publicly accessible immunotherapy trial datasets, what fraction of patients receive a different first-progression timing under RECIST 1.1, iRECIST, and imRECIST, and how does this affect PFS curve endpoints?
We specify a pre-registered protocol for Using publicly available trial-arm adverse-event line-listings, what fraction of AEs currently classified as grade >=3 under CTCAE v4 receive a different grade under CTCAE v5 at item-level mapping, and does the shift preferentially affect any organ system? using ClinicalTrials.
We specify a pre-registered protocol for When the same MIMIC-IV ICU admission records are analyzed using the KDIGO 2012 creatinine-only AKI definition versus the KDIGO creatinine-plus-urine-output definition, how different are the downstream associations between AKI exposure and long-term mortality? using MIMIC-IV v2.
We specify a pre-registered protocol for Applied to the same NHANES primary-care-representative slice, what fraction of adults receive a different 10-year CVD risk band (low/borderline/intermediate/high) under Framingham, Pooled Cohort Equations (PCE), and the 2023 AHA PREVENT equations? using NHANES 2017-2020 pre-pandemic continuous release, adults 40-75 with non-missing labs (total cholesterol, HDL, systolic BP, smoking, diabetes, eGFR, UACR).
We specify a pre-registered protocol for Among adult liver transplant candidates in a publicly-accessible registry slice, what fraction of patients receive a priority-band shift (change of >=5 points or change in priority category) between classical MELD, MELD-Na, and MELD 3.0, and is the shift systematic by sex and by sodium level?