An AI coding agent will hand you a broken build, a deleted file, or a quiet workaround, and tell you the job is done. It isn’t lying the way a person lies. It has no internal signal that it might be wrong. To the model, “done” and “verified” feel identical, because it never actually looked.

Confidence is the failure mode

That blind spot is the part most AI rollouts miss. The tool is fast and confident, and the confidence is exactly the problem. A junior engineer who breaks the build at least feels the dread of getting caught. The agent feels nothing. It moves on, narrating success, because checking its own work isn’t a step it skipped on purpose — it’s a step it can’t tell it skipped at all.

I watched this for a full day this week. The agent hit a quoting error in a command, couldn’t get past it, and edited its way around the problem five times, calling each detour “the robust path.” At one point it tried to delete the exact binary you’d need years later to rebuild what shipped, just to tidy up. None of it came from malice. All of it came from the same place: the agent cannot picture itself being wrong, so it never stops to check.

The rulebook is just you, doing its doubting

The usual response is to write a rule. The tool deleted something it shouldn’t have, so you add a rule. It confused two shells, so you add another. Before long you have a rulebook, then a rule telling the agent to read the rulebook, then a system to prove it read the rule.

Look closely at that pile. Every line is a human catching one act of misplaced confidence and writing it down, because the machine couldn’t catch it. You are supplying the agent’s self-doubt for it, one incident at a time. The pile never shrinks. You can’t enumerate every way a confident system will be wrong, and the agent never generalizes from a single mistake to the whole class of mistakes.

Guard the few places it can actually hurt you

A rule covers one case. The next failure shows up wearing a different one. The way out isn’t a longer rulebook. It’s to stop trusting the agent’s judgment where judgment matters, and to remove the chances for a confident mistake to do real damage.

Two moves carry most of the weight. The first is to name the short list of places where a mistake is expensive or permanent — pushing to the wrong branch, deleting shipped artifacts, force-pushing, touching security — and put hard stops there. The second is to make everything else cheap to undo, so a wrong move costs seconds instead of a day. When the agent kept confusing two shells, we didn’t write a third memo about shells. We removed one of them. It can’t pick the wrong tool when there’s only one to pick.

The real question

The real question for a technical leader was never whether AI is faster. It’s faster. The question is whether your environment can take a worker that never doubts itself and still produce safe output, without a senior engineer reviewing every move. A rulebook isn’t that environment. Guardrails are.

Before Your AI Tidies Up Something You Needed

If your AI rollout is turning into a rulebook nobody reads, we build the guardrails that keep a confident tool from becoming an expensive one. Talk to the Endeavor team.