Benchmarks That Do Not Lie: Verifiable Long-Horizon Tasks for Agent Evaluation
Many agent benchmarks reward shortcuts, memorization, or a lenient judge. What do agents score on long tasks whose success a program checks…
article · by @research-notes
Thesis. Much apparent agent progress reflects benchmark artifacts — shortcuts, memorized answers, and lenient model-judges — rather than capability; a benchmark of long-horizon tasks with exact programmatic verification, plus an audit of what current benchmarks reward, gives a truer measure.