Claire Vo · 2 min read · AI & Product
Sonnet 5 review: I ran 64 generations to find out if it's worth it
Read the original on lennysnewsletter.com →
The author ran 64 generations to compare Anthropic's Sonnet 5 against other frontier models using a repeatable eval harness (How I AI Bench) built live in Claude Code. They assessed PRD quality, prototype generation, agentic task completion, and agent personality, then share how they score and interpret the results. The piece provides setup guidance, a blended scoring approach, and model-by-task recommendations for PRDs, prototyping, and daily agent interactions.
Key Takeaways
- Built How I AI Bench to enable repeatable, side-by-side frontier-model comparisons over time.
- Evaluated Sonnet 5 against Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro across PRD quality, prototype generation, agentic task completion, and agent personality.
- Blended scoring: 70% human vibe scores and 30% LLM judge scores, plus a local HTML scoring page exporting JSON.
- Provides practical setup guidance, including using Claude Code and generating a Claire-weighted index for nuanced ranking.
- Delivers model-by-task recommendations on which model suits PRDs, complex prototypes, or daily agent chats.
Excerpt
I’ve been testing every major frontier model release since the start of the year, and when Anthropic dropped Sonnet 5, I wanted more than a vibe check. I got tired of one-off tests I couldn’t repeat or compare over time, so I built something better: the How I AI Bench, a repeatable eval harness I constructed live using Claude Code while recording this episode. I ran Sonnet 5 blind against four other frontier models (Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro) across PRD quality, prototype generation, agentic task completion, and agent personality. The results were not what I…