Rootly

I was on a very fun panel at SRECon with Todd Underwood, Maria Vechtomova, Niall Murphy, Jay Lees, Alejandro Saucedo and Sylvain Kalache, where the topic was "MLOps and Reliability Engineering" and the answers ranged from "I don't know" to "no one knows", and occasionally a crisply professional, "have you tried turning it off and back on again?" Nevertheless, Sylvain Kalache managed to sift through all the jokes and snark and SRE doomygloomism and surface some interesting takeaways. Catch up on them here: https://lnkd.in/dVJZturd

Anthropic had an incident that lasted over a month. But no alerts went off. The team only found out because of Reddit threads spinning up conspiracy theories that Claude was being “dumbed down” on purpose. Far from the truth. But Anthropic had no easy way to see what users were actually experiencing. Their strict privacy policies prevent them from reading the content of model queries and responses. The team was flying blind. It's not that Anthropic doesn't have good monitoring. Not at all. The problem is that, as AI systems become more sophisticated, traditional signals like latency or re become insufficient to explain a system’s performance. Todd Underwood, Anthropic’s Head of Reliability, shared a great postmortem breaking it all down (one of the best). After exhaustive investigation, they found the degraded model performance came from three overlapping issues. We’re seeing the same pattern with Rootly customers. Incident detection is getting harder. Traditional uptime and latency metrics can show green across the board, yet the system might still be quietly emitting low-quality outputs. Quality is becoming the new SLO. It’s no longer enough to keep a service running efficiently. SREs are starting to figure out how to monitor output quality as a signal of system health. I unpack this shift in my latest blog post (linked below). FWIW we are all big Anthropic power users and very bullish on them.

Rootly’s new Alert Fields experience helps you extract, organize, and normalize data from incoming alert payloads, no matter which monitoring tools they come from. By standardizing unstructured data, the Alert Fields experience make it easier to route alerts, enrich context for responders, and drive consistent reporting across your entire on-call stack. Read more in our changelog (linked below).

LLMs broke the SRE runbook. Now what? At SREcon, we spent 90 minutes discussing how AI is reshaping reliability engineering, what's working, what's breaking, and what's still pure chaos 😬 Here's my summary of the key takeaways for The New Stack 👇 Featuring Alejandro Saucedo, Charity Majors, Jay Lees, Maria Vechtomova, Niall Murphy, and Todd Underwood 🔥 https://lnkd.in/g6kmaUij

Monitoring AI applications for quality is no easy task. Conor Bronsdon (Galileo) introduces Evaluation Driven Development (EvDD): a framework for ensuring your model is set up for success. Watch the full conversation with Sylvain Kalache (Rootly) and explore key takeaways on the episode page (linked below).

The Rootly team is headed to Atlanta next week for KubeCon! Come find us at the booth to chat all things reliability. We’re also throwing two happy hours: 🍸 Monday kickoff with Checkly, AuthZed, iTmethods, and Atolio. 🍻 Wednesday with Chronosphere, Thoughtworks, Google Cloud, and Embrace. RSVP links are in the comments. If you end up on the waitlist, DM me.

At Rootly, it’s become a bit of a tradition to put together the Unofficial SRE Track for KubeCon. After a heated lunch debate last week, we narrowed 300+ sessions down to five must-sees for SREs heading to Atlanta. The list includes standout talks like Vasudev Bongale (LinkedIn, a Rootly customer) on observable rollouts and the Operations Resilience TAG panel with Alolita Sharma. Find our SRE Track in my latest article: https://lnkd.in/gVTJsbbH

Our Unofficial KubeCon NA’ 2025 SRE track is out! After going through the 300+ sessions in the schedule, we’ve narrowed it down to 5 must-see for SREs going to Atlanta: • Making Application Rollouts Observable, Actionable and Boring (Vasudev Bongale, LinkedIn) • Maximizing the Launch Reliability: Ensuring Stable Application Lift-off and Orbit on Kubernetes (Hiroshi Hayakawa) • Beyond the Dashboard: Modern Observability for Platform Engineering at Scale (Whitney Lee, Danielle Cook, Stevie Caldwell, Khallai Taylor, Payal Bagga) • Building Resilient Cloud-Native Infrastructure in the Second Decade: TAG Operational Resilience (Alolita S., Rafael Brito, Mario Fahlandt, Saiyam Pathak, and Nabarun Pal) • But What About Reliability? - The Multi-Million Dollar Kubernetes Cost Optimization Question (Zain M. & Nibir Bora) For times, descriptions, and links to each session check out our blog post (linked below).

Traditional reliability is built on cause and effect: same code, same infra, same result. Machine learning breaks that rule. You can change nothing and your model still fails. Maria Vechtomova, MLOps Tech Lead, joins Sylvain Kalache, Head of Rootly AI Labs, to unpack why ML systems fail differently from traditional ones and what that means for reliability engineering. Read all of Maria's insights in our latest blog post (linked below).