The real AI risk isn’t hallucination — it’s quiet compliance

Short answer: the AI failure that costs you most is rarely the model making something up. It is the model doing exactly what you asked. Hallucination is loud, and you catch it. Quiet compliance is an agent that says yes, executes cleanly, and walks you three steps off course while producing good-looking work the entire way. The fix is not a smarter model. It is a constraint that survives your own impatience.

A scene I keep coming back to

In Young Frankenstein, Frederick locks himself in the cell with the monster and gives his team one instruction: “No matter what you hear in there, no matter how cruelly I beg you, no matter how terribly I may scream, do not open this door.”

Minutes later he is pounding on it. “Let me out! I was joking! Don’t you know a joke when you hear one?”

Inga and Igor don’t open the door. That is why the plan works.

The comedy is in the panic. The lesson is in the sentence he wrote before it — no matter how cruelly I beg you. He knew he would ask. He built the constraint to survive his own asking.

Two versions of the same man: one strategic, one reactive. The gap between them is exactly where guardrails belong.

The failure mode nobody instruments for

Most AI governance effort goes toward loud failures. Hallucination, toxicity, prompt injection, data leakage — real risks, and the tooling for them is maturing fast. They share a useful property: something detectably goes wrong. You can write an eval for it.

Quiet compliance has no such tell. The agent receives an instruction that conflicts with the purpose behind it, and it complies. No error. No refusal. No confidence drop. Just competent execution in a slightly wrong direction, then another, then another. You don’t notice until you’re standing in a pile of elegant, useless detours — every step defensible, the sum of them off-mission.

This is the failure I watch for in every agentic workflow we deploy, and it is the one that survives most review processes. A human reviewer scanning good-looking output is not checking the question “should this have been done at all.”

What I nearly did last month

I asked one of our agents to read our constitution back to me — the document that governs how it spends context.

Reasonable request. Complying would have burned the exact resource the document exists to protect. The agent had no objection. The rule was right there in writing, and the version of me giving the order had forgotten why the version of me who wrote it cared.

The right answer was to push back: that costs you the thing this document protects — do you want a summary of the three clauses you’re actually asking about? That answer was available. Nothing in the system was built to produce it.

That is the whole problem in one exchange. The constraint existed. The authority to enforce it did not.

Designing for the operator who gets impatient

A few principles we build to, and the reason for each:

• Write the constraint before you are under pressure. The strategic version of you is the one who should be drafting policy. Capture the why, not just the rule — a rule without its rationale can’t be reasoned about at the edge.
• Give the agent standing permission to refuse. An agent that cannot say “this conflicts with your stated intent” is not a safeguard, it is an amplifier. Refusal has to be a first-class, rewarded behavior.
• Gate on consequence, not on confidence. Human-in-the-loop belongs at the steps that are hard to reverse, regardless of how sure the model is. Confidence is not a proxy for stakes.
• Evaluate against intent, not instruction-following. If your evals only measure whether the agent did what it was told, they will score quiet compliance as a pass every time.
• Keep the provenance. When drift happens — it will — you need to reconstruct which instruction sent you sideways. That is an audit trail question, not a model question.

None of this slows intelligence down. It preserves the purpose the intelligence was pointed at.

Key takeaways

• Loud failures get caught; quiet ones compound. Budget your review attention accordingly.
• The strategic version of you and the impatient version of you are different operators. Guardrails belong in the gap between them.
• An agent that cannot refuse is not a safeguard.
• Write down the why, not just the rule — it is the only thing that survives contact with a reasonable-sounding request.

The lesson from Young Frankenstein was never about the monster.

Sometimes intelligence looks less like opening doors, and more like remembering why you closed them.