Elena Verna · 10 min read · AI & Product
The AI era requires a different kind of experimentation.
Read the original on elenaverna.com →
The article argues that the old approach to experimentation—tiny tweaks, surface-level optimizations, and short-term wins—no longer suffices in the AI era. With AI-driven interfaces collapsing surfaces and development velocity speeding up, experiments should aim for bigger, foundational changes while being selective about what to test and leveraging automation and data platforms to cope with rapid change.
Key Takeaways
- Shift from surface-level tweaks to bigger, foundational bets that drive step-function change.
- AI-era product interfaces are collapsing into fewer conversational surfaces, reducing the need for extensive UI A/B testing.
- Experiment velocity is accelerating, so the impact of changes is shorter-lived and experiments must be faster and more robust.
- Be selective about what you test; not everything is worth experimenting, and leverage automation and data platforms (e.g., MCP, agents) to handle complexity.
- Prioritize integrating data sources and automation to enable scalable experimentation in a rapidly changing AI landscape.
Excerpt
Experimentation is my favorite. I just love it. But this dear old friend of mine is quite different from even a few years ago. And maybe that’s not such a bad thing. The old way of experimenting had its limits: Focus on minor tweaks instead of big bets Optimizing superficial surfaces instead of foundational complexity Avoiding monetization (‘It takes 6 months of decision approvals & build out. If you are lucky.’) Product mostly testing through previous/post, instead of actual A/B tests or feature flags Focusing on fast wins - results measured in 2 weeks Some of this made sense, back…