> ## Content Index
> Fetch the complete content index at: https://notes.dawidbalut.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# AI Security Agents Fail When You Build Six Products at Once
- URL: https://notes.dawidbalut.com/ai-security-agents-fail-when-you-build-six-products-at-once/
- Published: 2026-07-07T18:53:00.000Z
- Updated: 2026-07-08T18:53:53.000Z
- Description: Discovery, validation, containment, recall, cost control, and patching are different problems. Treating them as one is how agentic security projects die early.
- Author: Dawid Balut

I think a lot of people get disappointed with AI agents in security because they accidentally try to build six products at once.

They want the agent to find more bugs, reduce false positives, not miss true positives, avoid unsafe actions, produce evidence engineering trusts, learn from previous runs, improve cost efficiency, and start chaining vulnerabilities.

And then inevitably someone adds another hard problem to the pile: “Can it just patch the issue too?”  
That is where things really start going sideways.

Those are not the same problem. They barely even have the same constraints.

The thing that worked for us was not trying to build the final version from day one. The first question was much simpler - can we even prove that this thing can find anything real?  
Not everything. Not perfectly. Not with beautiful recall metrics.

Just something real enough that a skeptical engineer could not reasonably wave away as AI hype.

You are not competing with SOTA harnesses or building a commercial unicorn on day one. Internal harnesses have a different advantage: they can go deep into one environment, one product, one threat model, and weird company-specific failure modes.  
That is a very different game than building something generic enough to sell.

Once discovery worked, the next problem was containment. Do not give the model a blank shell and hope it makes good choices - it won’t. Give it a narrow set of approved tools and workflows. Make the intended path the easiest path. If it tries to go around it, block it.

Then add the boring but critical pieces: scope boundaries, command restrictions, network limits, passive-only modes where needed, evidence capture, deduplication, session rules.

Only after that does stronger recall become worth serious investment. Early on, false positives are usually more damaging than false negatives.

A missed issue is bad, but a stream of vague “maybe exploitable” findings kills trust much faster. Nobody wants another scanner that produces homework.

So the bar that works is deterministic evidence, not model confidence.

Hypothesis → live result → reproduced result → real impact.

And I would keep patching as a separate problem for much longer than people think.

Discovery and remediation have different owners, risks, review paths, and blast radius. If the community figures out safer autonomous patching, great. Add it later as another capability. But collapsing discovery, validation, and remediation too early makes the whole system harder to trust.

My current view: doing discovery and exploit validation well already puts you ahead of most attempts.

Start narrower.  
Find one real thing.  
Constrain the action space.  
Raise the evidence bar.  
Reduce noise.  
Then scale.

The model matters, but the workflow matters more.

Most teams will not fail because the model is useless. They will fail because the first version creates too much noise, too much risk, or too much organizational friction before it becomes useful.