Gergely Orosz · 26 min read · AI & Product
Building resilient AI-powered software, and planning for when things go wrong
Read the original on newsletter.pragmaticengineer.com →
This article shares excerpts from Sam Newman's new book 'Building Resilient Distributed Systems,' focusing on disaster recovery strategies and AI-powered system resilience. It covers critical DR concepts like RPO, RTO, and various failover patterns, while also addressing the unique challenges AI introduces to system reliability, including model provider resilience and infrastructure shortages.
Key Takeaways
- AI agents writing more code is leading to more deployments and consequently more production incidents, making resilience planning more critical than ever
- Disaster recovery planning must account for cloud region, data center, or site-wide outages with clear RPO and RTO objectives defined upfront
- DR patterns range from backup-and-restore to multisite active-active, each with different cost and complexity trade-offs
- Ongoing GPU and CPU shortages mean infrastructure availability cannot be assumed, requiring prioritization of critical services in DR plans
- AI-powered systems require additional resilience considerations including multi-model and multi-vendor strategies, model change management, and robust observability
Excerpt
Last week’s guest on the podcast was Sam Newman , who had released a book I’ve personally long awaited titled Building Resilient Distributed Systems . It’s an especially timely book for many of us software engineers, as we’re seeing more outages with nearly all services and systems – much of it caused by more code deploys thanks to AI agents writing more code; and more bugs causing more incidents in production. Sam’s new book, Building Resilient Distributed Systems , in print form Sam’s book being “fresh out of the oven,” it naturally also touches on what AI changes as we…