Chaos engineering has a premise I find genuinely convincing. Tests only check the cases you thought of. Injecting real failure into a real system finds the ones you did not.

It also has a problem sitting right at the front of it, and I have not seen it addressed properly anywhere.

You choose the experiment. Which service to kill, which dependency to sever, which latency to inject. That choice comes out of your understanding of the system, which means the experiments you can design are bounded by exactly the thing chaos engineering exists to expand.

You cannot form a hypothesis about a dependency you do not know exists.

Where teams actually aim

The consequence shows up as a predictable pattern. Teams inject chaos into stateless microservices that recover automatically, and leave the real single points of failure alone. Databases, message queues, API gateways.

That is not cowardice. Stateless services are the parts people understand best and the safest thing to break, so they are what gets chosen. The experiment succeeds, the service recovers, everybody writes it up as a validated resilience property.

Meanwhile the shared queue that four services depend on, one of them in a way nobody has documented, remains untested. The guidance in that same analysis is to use service dependency maps to find critical paths and test those first, which is obviously right and quietly assumes you have an accurate dependency map.

Most teams do not. What they have is a diagram that was true when somebody drew it, plus the parts of the system that whoever is in the room happens to know.

So the practice designed to reveal unknown failure modes ends up sampling from known ones.

The thing being tested is not the system

There is a second confusion worth pulling apart, because it changes what a passing experiment means.

You kill a pod. The service recovers. What did you learn?

You learned that the recovery mechanism you built for that failure works for that failure. Which is worth knowing, and is closer to a regression test than a discovery.

The interesting result is never the injection. It is what happens around it. Did anybody get paged, and how long did it take? Did the dashboards show the failure, or did the service degrade in a way nothing was watching? Did the runbook match what the person on call actually had to do? Did the failure stay where you put it?

That last one is the real experiment. Blast radius containment is what makes this acceptable to run at all, and it is also the hypothesis worth testing: you believe this failure stops here, and the experiment is whether it does.

Which brings you back to the dependency question. If something breaks two hops away from the injection point, that is the most valuable thing the exercise produced, and you will only spot it if you were watching services you did not expect to be involved.

Running the experiment blind

The most common mistake in this space is not aiming badly. It is injecting faults without the observability to see what happened.

An experiment where you cannot measure the response is not an experiment. You broke something in production and it seemed alright, which is a strictly worse position than not having done it, because now you have a written record saying the system is resilient.

The other predictable failure is abandoning the practice after one experiment causes real impact. That is usually a scoping problem rather than a chaos problem, and the standard containment dimensions are worth stating plainly: start with one replica of N, one per cent of traffic, five to ten minutes, internal services before user-facing ones.

Preparing for one

The part of this I have found most useful is not the injection at all. It is the hour beforehand.

Before an experiment, the questions are: what depends on this thing, what does this thing depend on, what is currently healthy, what deployed here recently, and who owns the services in the likely blast radius.

Answering that used to be the reason these exercises got postponed. It meant a dependency diagram somebody would have to verify, a check of what shipped in the last week, and a conversation about who to warn.

Asking Vorr AI what depends on a service and what it depends on takes that from an afternoon to a few minutes, because the relationships between services, infrastructure and deployments are already there rather than being reconstructed. The same goes for checking whether the target is currently healthy, which matters more than it sounds. Running a failure injection into something that is already degraded is how a game day becomes an incident.

The genuinely useful output is the surprise. Every time I have looked at a full dependency picture before an exercise, there has been at least one edge nobody in the room expected. That edge is a better experiment than anything we had planned, and it was free.

The chaos nobody scheduled

Worth flagging a 2026 development that changes the arithmetic here.

Seventy-nine per cent of organisations now have some form of AI agent in production, with the large majority planning to expand. The failure mode described in that piece is worth reading carefully: an agent takes an action that is technically correct given its context, the context is incomplete, and the infrastructure cascades.

That is fault injection. Nobody scheduled it, nobody scoped a blast radius, and it does not fit any postmortem template, so the incident review turns into an argument about whether it was an agent failure or an infrastructure failure.

The framing I took from it is that your system has a finite tolerance for disruption at any moment, and it is consumable. A chaos experiment draws from it. So does every agent action. Two teams running experiments against overlapping dependencies produce a combined blast radius that neither of them planned, and agents acting outside that accounting make it worse.

Which is another argument for the dependency picture being shared and current rather than living in whoever designed the experiment.

The part I have not solved

I should be honest that knowing all of this has not made me good at running these.

We have done a handful. The preparation is genuinely better now, and the dependency picture has turned up something unexpected every time. But there is a gap between finding a surprising edge and building an experiment around it, because the surprising edges are usually the ones nobody wants to touch. That is the whole reason they went unnoticed.

So the honest position is that the dependency graph fixed my aim and did nothing about my nerve. The queue that four services depend on is still untested, and I can now describe precisely why that is a bad idea.

Which might be the more useful outcome anyway. A team that knows exactly which failure it is avoiding is in a different position from one that thinks it has already tested the important things.

The premise was always right. Tests validate what you anticipated, and production finds what you did not. The catch is that the experiment design is also something you anticipated, and the failure modes worth finding are, by definition, in the part of the system nobody has thought about yet.

Author note

What changed my view here was realising I had been treating the injection as the experiment and everything else as setup.

It is the other way round. The injection is trivial. The experiment is whether your understanding of the system survives contact with it, and that is decided before you break anything.

Mohan Gopi, Associate DevOps Engineer at Frigga Cloud, working on Vorr AI across AWS, GCP and Azure. LinkedIn.

You Can Only Break What You Know About