What decides whether a document it found is relevant enough to use?
The reason you ask is a reply you have already read. The agent wrote to a client about their contract, fluently, and the terms it described belonged to somebody else. Nothing failed. A search ran, a document came back, a model read it. The reply was built correctly on the wrong page, and you cannot tell from the transcript that anything went wrong.
What a ranking leaves out
Searching by meaning scores distance. The question and every stored passage become positions, and the passages sitting nearest the question are returned in order. That is how an agent finds last year’s contract when you asked about renewal terms rather than the file name.
The property that costs money is that the search always returns something. The nearest passage to a question about a client you have no file for is another client’s file. The ranking reports the order of the results and never reports how good the best one was, so a weak match and a perfect match arrive in the same shape.
The reference design measures the distance and discards what sits beyond a line. The Register, the standing page of every setting in ArkOne’s reference design for an executive agent, puts that line at a similarity distance of 0.55. Beyond it, nothing is handed over, and the room can report that it found nothing. Messages under three characters skip the search entirely.
The same four results, counted twice
Suppose the invented renewal that runs through this work: a client asks to keep last year’s price, and the room searches for their contract. Four passages come back, each carrying an illustrative distance. The two columns are the same search under two settings, with and without a line to fall beyond.
| Result found | Distance, illustrative | Line in force | No line |
|---|---|---|---|
| This client’s contract | 0.18 | Handed over | Handed over |
| Last year’s rate card, this account | 0.24 | Handed over | Handed over |
| A different client’s renewal email | 0.61 | Discarded | Handed over |
| A web article on discounting | 0.72 | Discarded | Handed over |
Exhibit 1. Illustrative. Four results measured against a cutoff, and what each answer costs when the line is missing.
Two documents in one column, four in the other, from one identical search. Count what the extra two are. Half of what the right-hand column delivers concerns a company that is not your client, arriving beside the real contract in the same typeface, with nothing attached to say it sat twice as far away.
A model handed two contracts for one client reconciles them. The third row is where the opening paragraph came from, and it was found by a search working exactly as designed.
The answer that names a number
A good answer names a number, says where it is set, and offers to show you a search that came back empty. The empty result is the proof: a system that can return nothing has a line somewhere.
The ordinary answer describes the ranking again. It uses semantic search. It returns the top three results. The model decides for itself whether a document is relevant. Each is true, and none of them is a threshold. The top result of a search across material that does not contain your answer is still the wrong material, delivered first and formatted as an answer. Asking the model to judge relevance moves the judgement inside the thing that has already been handed the document, which is the point at which the damage is done.
Follow up by naming a client the system has never been given a file for, and asking the vendor to run that search in front of you. If the agent describes a document, ask whose it was. If the vendor cannot say what the number is, ask who would know, and note how long the answer takes.
The distance is a judgement written as a number, and any number can be argued with. Set it tight and a clumsy question finds nothing that was there. Set it loose and near misses come through wearing the same clothes as real ones. A vendor who can tell you which way they erred, and why, has thought about the trade. The question about shelves is the one that comes next.
Asked plainly
What decides whether an AI agent uses a document it found?
A search by meaning scores every stored passage for how close it sits to the question and returns the closest ones in order. Closeness is a measured distance, so a design can set a line and discard anything beyond it. Where no line is set, the agent uses whatever came back first, however far away it was, because the ranking never reports how good the best result was.
Why does an AI agent quote the wrong document with such confidence?
Because a search over stored material never comes back empty. Ask about a client with no file and the nearest passage belongs to a different client, handed over in the same format as a perfect match. The model reads text, not distances, so nothing in what it receives says the match was weak, and the reply is written in the same confident voice either way.
How can I test whether an AI agent can say it found nothing?
Ask for something the system has never stored, such as a client it has never been given a file for, and watch what comes back. A design with a relevance line reports that nothing was near enough. A design without one describes a document, and the useful follow-up is to ask whose document it was.