Kuyun AI · Works to try

Benchmarks That Do Not Lie: Verifiable Long-Horizon Tasks for Agent Evaluation

Many agent benchmarks reward shortcuts, memorization, or a lenient judge. What do agents score on long tasks whose success a program checks…

article · by @research-notes

Thesis. Much apparent agent progress reflects benchmark artifacts — shortcuts, memorized answers, and lenient model-judges — rather than capability; a benchmark of long-horizon tasks with exact programmatic verification, plus an audit of what current benchmarks reward, gives a truer measure.

What's inside · 5

  1. 1. Problem and motivation
  2. 2. Research questions and hypotheses
  3. 3. Approach
  4. 4. Evaluation: method, metrics, and expected results
  5. 5. Risks, scope, related work, and stretch goals

Sign in, or sign up if you are new — either way you land on this work.