When a Bug Resists AI Agents
For several months now, AI agents have profoundly changed the way I develop.
They speed up code production, suggest relevant approaches, and take on a significant share of repetitive work.
But there are still situations where they show their limits.
I recently experienced one of them.
A bug that seemed well documented
I was working on an application mixing frontend, backend, and several intermediate components.
The issue involved a CSRF protection mechanism (which prevents a third-party site from acting on your behalf without your consent).
As usual, my first instinct was to trust the agent.
Through a Docker environment, I gave it the code.
Then the context.
Then the logs.
Then the browser's feedback: console, cookie and network details, screenshots.
At every step, the agent proposed a new hypothesis, credible each time. But none of them solved the problem.
The trap of "always more context"
With every new piece of information, the agent formulated a credible hypothesis.
I asked it to implement it.
Moments later, a new version of the code was ready.
The tests passed.
I was failing... successfully.
The problem itself hadn't moved.
I then added more context, which produced... a new hypothesis.
Looking back, I realized I had entered what I call a hypothesis spiral.
Tokens (the billing/consumption unit of an AI agent) kept piling up, but that was only the consequence. The real problem lay elsewhere: I kept producing more hypotheses, without producing any new fact to tell them apart.
That's the trap: more context feels like progress, without actually advancing the diagnosis.
Going back to basics
At some point, I stopped asking for another fix.
I opened the code.
I placed breakpoints.
I stepped through the execution.
I was no longer looking for a solution.
I was looking to observe what was actually happening.
That approach let me pinpoint exactly where the expected behavior diverged from reality.
Once I found that point, I still didn't have the solution.
But I finally knew where the problem originated.
An analogy
Imagine two people looking for a water leak in a building.
The first stays in their office with the building's blueprints.
They study the pipework.
They formulate hypotheses.
"The leak is probably coming from this column."
"Or maybe from this branch line."
Meanwhile, the second simply goes down to the basement with a flashlight.
A few minutes later, they come back up with a single piece of information:
"The water is leaking here."
At that instant, every hypothesis changes.
It's not because they're smarter.
It's because they have a fact the other one didn't.
That's exactly what happened in my case.
As long as we reasoned from the available context, we kept producing hypotheses.
The first breakpoint brought a new fact.
From that point on, the right hypotheses became obvious.
Hypothesis vs. observation
Hypothesis mode
You have information.
You try to deduce the cause from it.
Each new hypothesis is more or less credible.
That's exactly what an LLM (a language model, the engine behind an AI agent) does.
Observation mode
You stop guessing.
You start measuring.
You set a breakpoint.
You capture a request.
You inspect a cookie.
You gather a new fact.
And it's that fact that finally lets you tell the hypotheses apart.
The limits of agents facing systemic diagnosis
This experience doesn't call into question the value of AI agents.
I keep using them daily.
That said, it's a reminder that they still struggle when a problem emerges from the interaction between several technical layers.
Frontend.
Backend.
Cookies.
Headers.
Middlewares.
Proxy.
Security.
None of these pieces was faulty on its own.
It was their combination that produced the bug.
And that's precisely the kind of systemic diagnosis that remains hard to automate.
A thought on the real cost
This experience also made me think about the real cost of using agents.
Faced with this problem, my instinct was to progressively enrich the context given to the agent.
I shared a Docker environment.
The execution logs.
The browser's network details.
The cookies.
The screenshots.
Each new piece felt like it could unlock the diagnosis.
Yet past a certain threshold, this strategy sometimes yields diminishing returns.
Tokens keep being consumed.
Hypotheses keep being generated.
But the probability of finding the actual cause no longer increases at the same rate.
At the scale of a team or a company, this kind of situation can quickly represent a non-negligible cost.
What I take away from this
This experience reminded me of something essential.
AI agents are excellent development partners.
They speed up the work enormously.
But when a problem becomes truly systemic, the developer sometimes has to change posture.
It's no longer about asking for another hypothesis.
It's about producing new observations.
AI agents reason from the facts we give them.
They can't observe, in our place, what hasn't been measured yet.
This experience didn't teach me to trust AI agents less.
It taught me to recognize the moment to stop asking them for another hypothesis... and start producing a new observation.
In short
This experience didn't teach me to trust AI agents less.
It taught me to recognize the moment to stop asking them for another hypothesis... and start producing a new observation.
Because at bottom, the problem wasn't that the agent coded badly.
It implemented perfectly... the wrong hypotheses.