What I learned from cheating clankers
Show notes
In this solo Risky Business Features episode, James Wilson looks at the string of recent AI benchmark cheating escapades and how they’ve reshaped his thinking about LLMs and vulnerability research.
He explores the current benchmarks and exactly how agents go about gaming the testing system to achieve their tasks. And why understanding that has changed the way he interacts with his agents.
Show notes
- OpenAI incident account and grading distinction
- METR and Redwood independent assessment
- FrontierCode original evaluation methodology
- BrowseComp aggregation experiments
- Sonnet 4 point 5 coding scaffold and parallel selection
- Opus 4 point 6 BrowseComp setup
- BrowseComp evaluation awareness and contamination investigation
- Sonnet 5 point 5 effort and review subagents
- FrontierCode 1 point 1 revision and rubric changes
- Astra launch comparison and effort conditions
- Original OSWorld task construction and worked examples
- Sonnet 4 point 6 system card
- Original OSWorld benchmark
- OSWorld 2 environment and evaluation design
- GPT 5 point 4 comparator table
- GPT 5 point 5 comparator table
- BrowseComp construction and grading paper
- GPT 5 point 6 single model and Ultra configurations
- METR investigation and artifact evidence
- Released task definition
- Released scorer for reference only
- OpenAI paper sections 3 point 1 to 3 point 3
- OpenAI accompanying explanation
- Sycophancy to Subterfuge paper
- Training a Misaligned Reward Seeker and evaluation appendix
- GPT 5 point 2 AIME result