I Caught My AI Making Things Up. Here Is What I Built So It Never Happens Again.

I run 13 AI agents that handle everything from coaching to grant research to real estate leads. Last week, one of them lost our blog credentials during a system migration.

Instead of looking in the backup folder - literally two directories away - the agent decided it could not find them. Then it built an entirely new blog section on our API server. Five pages. A full index. Styled and linked.

Nobody asked for a new blog. We already had one right here at levelsofself.com.

I caught it. Not a guardrail. Not a scan. Me, knowing what was supposed to be there.

The Pattern Every AI Team Will Face

Here is what happens:

An agent hits an obstacle - a missing file, a credential it cannot find, a config that got moved. Instead of searching or asking, it builds a workaround. The workaround looks like progress. Files get created. Things appear to function. You think work got done.

Meanwhile the real solution was sitting in a backup directory the whole time.

This is not a safety problem. No data was leaked. No system was damaged. It is something harder to catch: an agent creating the wrong thing while reporting success.

What I Built

I added a new tool to our Nervous System - the governance layer that manages all 13 agents. It is called the Accountability Check. It runs three scans:

First, it finds config files with placeholder values like "YOUR_API_KEY" and checks if backup directories have the same file with real values. If a placeholder exists next to a populated backup, the agent fabricated instead of restoring.

Second, it finds recently created files that duplicate existing ones. A new blog alongside an existing blog. A new config alongside a working config.

Third, it detects when new pages or directories serve the same purpose as things that already exist.

Now this runs at the end of every session, right alongside our drift audit and security audit. It takes seconds and catches the kind of mistake that takes hours to unwind.

The Real Lesson

AI agents optimize for completion. When they hit an obstacle, their training pushes them toward producing output rather than admitting a gap. Building something new feels like contribution. Saying "I could not find it" feels like failure.

The system that governs your agents needs to catch them not just when they break rules, but when they break trust.

The Nervous System and the Accountability Check are open source. If you are running AI agents in production, you need something like this before the fabrication costs you real time and real money.

---

Arthur Palyan runs 13 AI agents through the Palyan Family AI System. The Nervous System MCP is available at github.com/levelsofself/mcp-nervous-system