The meeting takes eleven minutes. The pilot is not renewed. Nobody in the room argues for it and nobody argues against it, which is the part that should bother you. Your operations lead says it did not really work for our use case. The vendor, on a call the week before, said it needed more data and a longer runway. Both are the sort of thing people say when the true answer is that they cannot remember a specific moment it failed, only that at some point they stopped opening it.
You are left holding a decision you cannot explain to the board, and a second attempt you have no way to make different. So it is worth being exact about what happened, because the thing that ended the pilot almost certainly never appeared in a report, a ticket, or an incident log.
Loud failures get fixed. Silent ones end pilots
A loud failure announces itself. The agent sends a customer a message with the wrong figure in it, or writes to the same person twice, or a scheduled job wakes and does nothing while everyone waits. Somebody notices within the hour, and a loud failure leaves a date, a witness, and a thing to fix. Pilots survive those. Yours may well have had several.
A silent failure produces work that looks entirely correct. The agent is asked about a client’s contract, finds the nearest document it has, and writes a confident paragraph about terms that belong to a different account. It summarises a decision that was never taken. It answers a question about your business using a passage that came from a forum post rather than from your own material. Nothing errors. The sentence is well formed, the tone is right, and the only person who could catch it is someone who already knows the answer.
Watch what that does to a team over eight weeks. The first time an output is quietly wrong, one person checks the next one, finds it fine, and relaxes. Then another is wrong, and now two people check everything. Checking is slower than doing the work, because to verify a paragraph you have to reconstruct the reasoning behind it. The time the pilot was meant to save goes into verification instead, and usage falls by drift rather than by decision. When the renewal comes up, nobody can name a failure, and nobody will defend it either. That is the meeting you sat in.
The mechanism underneath fits in a line. An agent will always produce something. Ask it about a client it has no file on, and a search returns the nearest file it does have. Ask about last month, and it uses whatever it was handed about last month, correct or not. Plausible output is the default state of the machinery rather than an achievement. What separates a working system from a pilot that dies quietly is whether anything in the design makes a wrong answer visible while it is still cheap to catch.
Nobody asks about them in a demo, because a demo is built around output, and these govern what happens to output afterwards.
Six ways a pilot goes wrong, and what catches each
Suppose a company of roughly four hundred people, invented for this exercise, runs a six-week pilot on client correspondence and internal reporting. Six things go wrong over the six weeks. Here is each one, whether anyone noticed, and the part of ArkOne’s reference design for an executive agent that would have caught it.
| What happened | Silent or loud | The part that catches it | What you see when it is missing |
|---|---|---|---|
| A reply described another client’s renewal terms | Silent | The relevance cutoff: a document too far from the question is discarded before the agent reads it | Confident answers about clients you have no file on |
| A weekly summary invented a decision nobody took | Silent | The Minutes: decisions written after the work and read back before the next piece | Every summary must be checked against someone’s memory |
| A forum post was quoted with the authority of your board pack | Silent | Separate shelves, so outside research is never read as house knowledge | Sources cannot be told apart in the output |
| A specialist was given authority nobody had granted | Silent | The Mandate: every specialist ships at propose and wait, and the change to act is recorded | Nobody can say what the system is allowed to do |
| The same customer received near-identical messages twice | Loud | The Door: a near-duplicate to the same person inside the window is refused | An apology, and one person watching the outbox |
| A change to the wording made the agent stop consulting operations | Silent | The Rehearsal: fixed scenarios scored before any change reaches your company | Quality moves and nobody can prove which way |
Exhibit 1. Illustrative. Six failures a pilot meets, tagged silent or loud, with the part that catches each
Five of the six are silent. The one loud failure, the duplicate message, is the one the pilot team could describe afterwards, and the one that got fixed inside a day. That is the pattern worth carrying out of this page: the failure everyone remembers is rarely the failure that killed the project.
Each of those parts was absent rather than broken. Nobody removed the relevance cutoff from your pilot; there was never a cutoff, because a search that ranks by nearness always hands back its nearest result, and nothing in the tool measures whether nearest was near enough. The reference design puts the line at a distance of 0.55, recorded on the Register, beyond which a match is thrown away and the agent reports that it found nothing. An agent that can come back empty is an agent whose confident answers mean something.
The fourth row is the one CEOs underrate. A specialist that may act on its own and one that must propose and wait look identical in a demo, because in a demo somebody is watching either way. The difference appears in week five, when nobody is.
The same week, with the checks in place
Suppose the same six weeks run again, with those parts present. The work does not get better. What changes is where each failure surfaces and who has to catch it.
| Week | Without the checks | With the checks |
|---|---|---|
| Two | The wrong client’s terms are drafted and sent; nobody notices for a month | The search returns nothing above the cutoff, and the draft says the contract could not be found |
| Three | A summary asserts a decision; the operations lead reads it, doubts it, redoes the report | The summary reads back the written decisions; a decision not on that list is not asserted |
| Four | A duplicate message goes out; an apology follows | The second message is refused, and the reason is returned so the work reschedules rather than retries |
| Five | Somebody adjusts the wording; quality shifts in a direction nobody measures | The change is scored against fixed scenarios; the dimension that slipped fails the change |
| Six | Renewal meeting; nobody can name a failure or a benefit | Renewal meeting; the record shows what was decided, by what reasoning, and what was refused |
Exhibit 2.
The second column is not a promise of correctness. Two of those five rows end with the agent doing less: finding nothing, refusing to send. That is the trade being made, and it should be made in the open. A design built this way will decline work it could have attempted, and some of those refusals will be unnecessary, because a cutoff is a judgement about where the cheaper mistake lies rather than a fact about the world.
What it buys is week six. In the first column, the renewal meeting has no evidence in it, so it turns on impressions, and impressions after eight weeks of quiet verification are always poor. In the second, there is a record: entries nobody may edit, a correction written as a new line beside the original rather than over it. You are not asking anyone to remember. You are reading.
None of these parts judge whether an answer is true. The relevance cutoff measures similarity of meaning, so a document that is close and wrong still comes through. The Minutes hold what a separate pass judged to be a decision, and that judgement can miss one. The Rehearsal scores against the scenarios somebody thought to write, so a situation nobody imagined has not been examined. Each part narrows the gap between a wrong answer and someone noticing it. None of them closes it, and a vendor who says otherwise is describing a product that does not exist.
What this arms you to ask
Four questions, before anyone shows you a screen. Each asks what happens to the output rather than how it looks, which is why a demo does not answer them.
What decides that a document was relevant enough to use, and can you show me a search that came back empty? A good answer names a threshold, says where it is set, and runs a query for a client the system has never heard of so you can watch it decline. The answer to distrust is that it uses semantic search and returns the most relevant results. That is a ranking, and a ranking always returns its top result even when everything available was wrong. The follow-up is the whole test: ask them to search for a company that does not exist, and if the system describes a file, ask whose file it is.
What is written down after each piece of work, and read back before the next? A good answer opens the actual text and lets you change a line of it. A thinner answer is that the system has memory, or that it learns from your data over time. Ask to see the exact lines the agent will read tomorrow morning. Memory you cannot open is the vendor’s memory of your company, not yours.
What is each part of it allowed to do without asking a person, and where is that written? Look for a named setting per component with a default that waits, and a record of who changed it. The weaker answer is that it knows when to check with you. The follow-up: show me the last time it decided not to check with someone, and show me the entry.
How is a change scored before it reaches my company, and what was the last change that failed? A good answer names the dimensions, the pass mark, the amount by which any one dimension may not slip, and then produces a change that was rejected. A process that has never failed anything has never been run. Ask for the failure, not the process.
Notice what none of those four questions is about. Not the model, not the interface, not the roadmap, not the accuracy figure on slide four. A pilot does not end because the writing was poor. It ends when the people using it can no longer tell good work from plausible work at a glance, and start checking everything, and then quietly stop.
The honest post mortem on the pilot you just closed is probably not that it failed. It is that nobody could tell whether it was working, which is a different problem with a different fix. More data would have helped, and it did not suit the use case. Both of those are true, and both are the symptom.
If you are about to run a second one, the four questions above belong before the shortlist, not after it, and the pages to read before the day itself put them in order. Before you sign is where they end up in a contract.
Asked plainly
Why do most AI agent pilots fail?
Rarely because of a single visible mistake. The common ending is quieter: the agent produces work that looks right, nobody has a cheap way to check it, so people begin checking everything by hand. The time saved disappears into verification, usage falls, and the pilot lapses with no incident anyone can name. The failure is a missing check, not a missing feature.
Why can nobody explain what went wrong with an AI pilot?
Because nothing went wrong in a way that leaves a mark. A wrong message sent to a customer creates an event with a date and a witness. Work that is merely plausible creates nothing: the reader shrugs, redoes it themselves, and mentions it to no one. Without a written record of what the agent decided and why, there is no evidence left to review afterwards.
What should be different in a second AI agent pilot?
Insist on the parts that make output checkable before you look at any feature. Ask what decides that a document was relevant enough to use, what gets written down after each piece of work and read back before the next, whether every action is recorded in an entry nobody may edit, and how the system is scored against fixed scenarios before a change reaches your company.