Juan Piaggio · 2026-09-13 · 7 min read · ai · governance · risk
The question we get most often now is some version of "what stops it from going rogue?" It is a fair question, and it is pointed at the wrong thing. In every agent incident we have seen up close — our own included — the agent did precisely what it was asked. The damage came from how far it could reach while doing it.
"Rogue" imagines defection: a system that forms its own agenda and turns. What actually lands in the postmortem is duller and more expensive. An agent with a credential broader than its job. A rule that was correct for one ticket and catastrophic across fourteen hundred. A retry loop that treated a downstream timeout as a reason to try harder. None of that requires the model to want anything.
Three shapes cover most of them.
Over-broad reach. The agent was given a key that works everywhere, because scoping it to one workspace was a day of work and the demo was Thursday. The agent never left its instructions. Its instructions just happened to be executable against production.
Scale mismatch. A human applying a slightly wrong judgment call makes one slightly wrong decision and usually notices by the third. An agent applying the same judgment makes it a thousand times before lunch, uniformly, with no drift toward "wait, this feels off." Consistency is the whole value proposition and also the failure mode.
Silent failure. This one is ours. Our worst agent incident was not an agent doing too much — it was a stack of four bugs that meant no tool-using run ever reached the model at all. Two hundred passing tests reported a healthy system. We found it by reading the stored reasoning of real production runs, not by running the suite again. An agent that quietly does nothing while reporting success is a governance failure too. It just doesn't make headlines.
None of the three are solved by a better-aligned model. They are solved by bounding what a misfire can touch.
You cannot inspect an agent's motives, and you cannot test your way to confidence about them — the input space is open-ended, and the failures that matter are the ones you did not imagine. So stop trying to control the thing you cannot measure and control the thing you can: what an action is able to affect, and whether you can undo it.
That reframe is freeing, because blast radius is ordinary engineering. It is scopes, allowlists, thresholds, and a log. Nothing about it is speculative.
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 150" role="img">
<title>Containment bounds what a misfire can touch — any gate can hold the action, and the ledger records it either way</title>
<rect x="1" y="40" width="176" height="56" rx="10" fill="var(--primary-50)" stroke="var(--primary-600)" />
<text x="89" y="73" text-anchor="middle" font-size="14" fill="var(--ink-1)">Agent proposes</text>
<path d="M177 68 H198" stroke="var(--text-muted)" stroke-width="1.5" fill="none" />
<path d="M198 63 l10 5 -10 5 z" fill="var(--text-muted)" />
<rect x="207" y="40" width="176" height="56" rx="10" fill="var(--sand-1)" stroke="var(--border)" />
<text x="295" y="62" text-anchor="middle" font-size="11" fill="var(--text-muted)">REACH</text>
<text x="295" y="83" text-anchor="middle" font-size="14" fill="var(--ink-1)">Workspace scope</text>
<path d="M383 68 H404" stroke="var(--text-muted)" stroke-width="1.5" fill="none" />
<path d="M404 63 l10 5 -10 5 z" fill="var(--text-muted)" />
<rect x="413" y="40" width="176" height="56" rx="10" fill="var(--sand-1)" stroke="var(--border)" />
<text x="501" y="62" text-anchor="middle" font-size="11" fill="var(--text-muted)">ACTION</text>
<text x="501" y="83" text-anchor="middle" font-size="14" fill="var(--ink-1)">Allowlist + floor</text>
<path d="M589 68 H610" stroke="var(--text-muted)" stroke-width="1.5" fill="none" />
<path d="M610 63 l10 5 -10 5 z" fill="var(--text-muted)" />
<rect x="619" y="40" width="176" height="56" rx="10" fill="var(--primary-50)" stroke="var(--primary-600)" />
<text x="707" y="73" text-anchor="middle" font-size="14" fill="var(--ink-1)">Recorded, reversible</text>
<path d="M295 96 V116" stroke="var(--warning)" stroke-width="1.5" fill="none" />
<path d="M290 116 l5 10 5 -10 z" fill="var(--warning)" />
<text x="295" y="142" text-anchor="middle" font-size="11" fill="var(--warning)">held for a human</text>
<path d="M501 96 V116" stroke="var(--warning)" stroke-width="1.5" fill="none" />
<path d="M496 116 l5 10 5 -10 z" fill="var(--warning)" />
<text x="501" y="142" text-anchor="middle" font-size="11" fill="var(--warning)">held for a human</text>
</svg>
Bound the reach. The agent's identity should be the tenant's identity, enforced at the data layer rather than remembered in a prompt. In Meshworq every agent action resolves against the workspace on the verified token, and no tool accepts a workspace argument — there is no string an agent can emit that widens its own scope, because the scope is not an input.
Bound the action. Give the agent a curated allowlist, not a general-purpose API. Five well-understood actions with a confidence floor beat forty capabilities you have never watched fail. Capability you have not observed in production is not capability; it is exposure.
Bound by risk, not by rule. Blanket approval gates train people to click through. Score each proposal instead and route by the score. Ours combines the agent's own confidence with how long the item has been waiting: anything at or above 0.67 is high risk and goes to a human, while the low-risk remainder flows. The point is that scarce human attention lands where the uncertainty actually is.
Bound the consequence. Every action carries a correlation ID into an audit ledger, so "what did it do, and why" is a query rather than an investigation. Pair that with a kill switch you have actually thrown — a feature flag that has only ever been tested on the way on is not a kill switch, it is a hypothesis.
Containment costs capability. An agent that can only touch one workspace, take five actions, and act unilaterally below 0.34 risk will do less than one you hand the keys to. That is the trade, and pretending otherwise is how teams end up with governance theater: a heavy approval queue nobody reads, guarding an agent nobody trusts.
The way out is not to loosen everything at once, but to widen the bounds as evidence arrives. Watch the acceptance rate on a curated action for a month. If humans approve it 97% of the time, that gate has stopped being a control and become a tax — promote it. If they approve it 60% of the time, you have learned something far more valuable than an incident would have taught you, and you learned it for free.
Nobody is going to hand you a model you can prove is well-intentioned, because intent is not the kind of thing that admits proof. Reach, permitted actions, risk thresholds, reversibility, and a ledger all are. Build those four bounds and the question stops being "what if it goes rogue" and becomes "what is the worst thing that happens if it is wrong" — which is a question with an answer, a number, and a deploy that changes it.