How was it examined before it met my company, and what is the pass mark?
Put it at the point in the demonstration where somebody mentions improvements. A slide or a sentence promises the agent gets better every month, and everybody nods, because who would want the other thing. That is the moment. An agent that improves monthly changes monthly, and you are asking what the next change has to get past before it speaks to your customers on a Thursday.
What a change has to survive
Instructions get sharpened. A specialist is told to be more direct, quicker to a number, less inclined to check with somebody else. Tomorrow it meets your company with that in force, and nothing about the edit tells you what else moved.
The Cabinet, ArkOne’s reference design for an executive agent, puts a standing examination between the edit and the meeting. A written bank of scenarios is put to the design and each answer is marked on five dimensions: does it sound like an executive, get the domain right, use your company’s own situation, consult the right specialists, say what to do next. The marking is done by a model that is not the one being examined. Two rules turn scores into a verdict, both on the Register: an average of 3.5 of 5 to ship at all, and a refusal of any change where one dimension slips by more than 0.1 points, whatever the average says. A pass mark is a floor, speaking only about the scenarios somebody thought to write.
Four things a marking scheme can show
Suppose you ask about one change: the specialist told to be more direct. Four things about that change should be showable, in a demonstration invented for the exercise.
| What you ask to see | What it looks like when it exists | Why this one |
|---|---|---|
| The scenarios | A list you can read, fixed before the change, run again after | An examination written after the answer is a description |
| The dimensions, and who marked them | Named dimensions, a marker that is not the thing marked | A model reading its own work finds its own habits reasonable |
| The floor, and the slip one dimension may take | Two figures, not one | An average is where a bad trade hides |
| The last change that failed, and the dimension that failed it | A change, a row, a date | A gate nothing has failed has never been shut |
Exhibit 1. Illustrative. One change asked about in a demonstration, and the four things a marking scheme must be able to show.
The third row is the one buyers skip and should not. A change making the room sound more assured while it consults fewer specialists lifts the average and costs you the check that made the answer right. Two figures catch that trade. One figure rewards it.
The fourth row is the whole question compressed. Anybody can hold an examination; the interesting property is having been failed by one, because that is the proof the marking scheme has teeth rather than a threshold set below wherever the product already sits.
The number, and what stands behind it
A good answer names the dimensions, the floor, and the slip allowed on any one dimension, then goes looking for the change that failed last. The looking is the signal. A team that can find it knows where those records are because somebody once had to explain one.
Where the examination is thinner, the reply describes effort rather than a mark: that the system is tested extensively, or evaluated against recognised benchmarks. Both may be true and neither is a gate, because neither carries a number your change must clear. The follow-up is one sentence, the last one this list asks you to memorise: show me the most recent change that did not ship, and what stopped it.
Then a narrower one, for the vendor who clears that. Ask what happens when a change passes on average and slips on one dimension. If somebody looks at it and decides, you have a review, which is a real thing and a different one, and the person deciding is under pressure to ship. Ask what they are measured on.
The last of the thirty-four
A vendor selling an agent is selling a room full of decisions somebody made, and most of those decisions are invisible from a demonstration, which is why they have to be asked for out loud. What you are listening for is always the same: whether a number exists, where it is written, and who would notice if it changed.
The examination and its limits are in the paper on the Rehearsal, and every setting quoted across these questions is stated once, with its reasoning, on the Register. The first of the thirty-four remains the cheapest to ask, in the silence while a demonstration is thinking. Twelve of these thirty-four fit on one side of a page, for the meeting itself.
Asked plainly
How should an AI agent be tested before it reaches my company?
Against a written bank of scenarios, scored on named dimensions, with a mark that must be reached before a change ships. What makes it an examination rather than a demonstration is that the scenarios are fixed in advance, the same ones are used before and after each change, and the scores are kept so a change that helped one dimension and hurt another is visible.
Can an AI agent get worse when a vendor improves it?
Yes, and an average score is where that hides. A change making an agent sound more assured while consulting fewer specialists can raise the overall mark and lower the protection at the same time. The defence is a rule saying no single dimension may slip by more than a set amount, whatever the average does.
Who should mark an AI agent's answers in testing?
Not the model being examined. A model reading its own work finds its own habits reasonable, so a separate model is given the scenario, the answer and a written marking scheme, which makes the scores colder and more consistent. It remains an opinion, so the marking scheme is the thing to read rather than the number it produces.