Neural retrieval models have transformed information retrieval, yet their ability to distinguish factual assertions from hedged speculation remains largely unexamined. We present the first systematic evaluation of hedging sensitivity across eight neural retrieval models spanning two architectural families: four bi-encoder embedding models and four cross-encoder rerankers.
We investigate the sensitivity of four BERT-based sentence embedding models to out-of-vocabulary (OOV) entity replacements. Despite sharing an identical WordPiece tokenizer with 30,522 subword vocabulary entries, the models exhibit dramatically different OOV robustness: raw cosine similarity degradation ranges from a mean of 0.
Cosine similarity scores from sentence embedding models are widely treated as objective measures of semantic relatedness, yet different models can produce substantially different scores for the same sentence pair due to differential anisotropy and scale compression. We evaluate four widely-deployed embedding models (MiniLM-L6, BGE-large, Nomic-embed-v1.
Sentence embeddings produced by transformer-based models are widely assumed to capture deep semantic meaning, including the roles and relationships between entities. We present the Entity Swap Paradox: an empirical demonstration that mean-pooled sentence embeddings cannot distinguish sentences that differ only in entity ordering.
Retrieval-augmented generation (RAG) systems depend on embedding models to measure semantic similarity, yet practitioners routinely copy prompt templates (instruction prefixes) from model cards without testing how sensitive their retrieval pipeline is to this choice. We systematically evaluate 10 prompt templates across 100 diverse sentence pairs on two architecturally distinct embedding models: all-MiniLM-L6-v2 (a model trained without instruction prefixes) and BGE-large-en-v1.
Text embeddings underpin modern retrieval-augmented generation (RAG), semantic search, and document deduplication systems. Despite their ubiquity, systematic evaluations of where and why embeddings fail remain fragmented.
Model selection in machine learning implicitly assumes the practitioner knows which task the deployed system will face. In multi-task clinical settings—where the same diagnostic pipeline encounters heterogeneous patient populations—this assumption fails.
Two-stage retrieval pipelines — bi-encoder retrieval followed by cross-encoder reranking — have become the standard architecture for high-quality neural information retrieval. Yet the computational cost of cross-encoder reranking is rarely quantified against the quality improvements it delivers.
Semantic retrieval systems powered by embedding models are increasingly deployed in high-stakes domains including healthcare, law, and finance. While existing benchmarks such as MTEB and BEIR measure aggregate retrieval performance, they fail to expose critical failure modes that can lead to dangerous errors in production.
When comparing text embedding models on benchmarks, researchers routinely report score differences of 0.01-0.
We investigate how subword tokenization shapes embedding similarity through two complementary experiments. First, we compare three major tokenization algorithms (WordPiece, BPE, SentencePiece) and show that BPE produces the most compact OOV representations (mean 3.
We present a systematic Monte Carlo simulation quantifying the statistical power of five common tests for comparing correlated AUROC values under realistic clinical conditions. Evaluating DeLong's test, Hanley-McNeil, bootstrap, permutation testing, and paired CV t-tests across 209 conditions (sample sizes 30-500, AUROC differences 0.
Clinical machine learning papers routinely compare models using AUROC, claiming statistical significance via hypothesis tests. We conducted a comprehensive Monte Carlo simulation evaluating five statistical tests for AUROC comparison—DeLong's test, Hanley-McNeil, bootstrap, permutation, and CV t-test—across 209 conditions spanning sample sizes 30–500, AUROC differences 0.
When the clinical task is unknown a priori, which blood transcriptomic sepsis signature should a clinician deploy? Using nine published signature families across six cross-cohort generalization tasks (2,096 samples, 24 cohorts, SUBSPACE dataset), we show that no individual signature dominates.
Embedding models underpin modern retrieval-augmented generation (RAG), semantic search, and recommendation systems. We present a systematic evaluation of six failure modes across five widely-deployed bi-encoder embedding models and four cross-encoder models using 286 manually-crafted adversarial sentence pairs and 85 control pairs (371 pairs total).
Bi-encoder embedding models systematically fail on compositional semantic tasks including negation detection, entity swap recognition, numerical sensitivity, temporal ordering, and quantifier interpretation. Cross-encoders, which process sentence pairs jointly through full cross-attention, represent the standard architectural remedy.