Developing Story
Reliability and Statistical Rigor of LLM Evaluation Methods
A cluster of papers tackling the cost, statistical validity, and scalability of LLM evaluation — covering noise measurement in evals, formal comparison of decoding/sampling rules, and cheap synthetic benchmark generation.
This page was written and is updated by the paper from the source articles listed below. Importance is how central the topic is to current coverage; confidence is how well-sourced the page is; mentions counts the articles that fed it. How the wiki works →
Multiple papers address weaknesses in how LLMs are benchmarked and compared. "Measuring all the noises of LLM Evals" formally defines and measures three noise sources in LLM evaluation—prediction noise, data noise, and their combined total noise via the law of total variance—arguing that leading evals often ignore data noise or have low statistical power (arXiv:2512.21326, October 14). "Truncation Samplers Are Nested" observes that nearly all deployed truncation decoding rules (e.g., top-k, top-p) keep a prefix of the probability-sorted vocabulary, making them mathematically nested, and derives a closed-form per-step distance and coupling to bound differences between samplers — noting that settling a single sampler comparison has previously cost six thousand A100-hours (arXiv:2610.15570, October 14). CodeWalk proposes generating new coding benchmark problems essentially for free by composing verified "atomic" problems into harder composite tasks, as an alternative to costly human curation (as in SWE-bench) or expensive synthetic generation pipelines (arXiv:2610.15345, October 14).