Questions for the Vendor

The questions that separate a Cabinet from a chatbot with a logo.

35 pages, in reading order.

Questions for the Vendor, No. 1How many times can it go around before it must answer me?Ask how many rounds it may take before it must answer you, and watch whether the vendor reaches for a number or for reassurance.4 minute read
Questions for the Vendor, No. 2What does it do when it wants something it has no tool for?Push the agent one step past what it can do, then read its reply and its transcript side by side.4 minute read
Questions for the Vendor, No. 3Which of its actions cannot be undone?A vendor who has sorted the agent's actions by what can be taken back answers in seconds; a vendor who has not, improvises.4 minute read
Questions for the Vendor, No. 4Which of your safeguards are code, and which are instructions to the model?A safeguard in code cannot be argued with; a safeguard written into the prompt is a request the model weighs against everything else.5 minute read
Questions for the Vendor, No. 5What sits between the model deciding to message someone and the message leaving?Between the decision to send and the message leaving there is either a list of checks or a gap, and the vendor knows which.4 minute read
Questions for the Vendor, No. 6What happens if two copies of it run at once?A job fires once because taking it and marking it taken happen in one step, not because only one copy happens to be running.4 minute read
Questions for the Vendor, No. 7Can it sign a message as one of my staff?An agent should carry its own identity on everything it sends; a colleague's address is how it recognises them, never a name it may borrow.4 minute read
Questions for the Vendor, No. 8Show me the line where it is told who it is talking toAn agent knows who is speaking only because a line of text told it; leave that line out and it addresses everyone as a stranger.4 minute read
Questions for the Vendor, No. 9What does it know about my company tomorrow morning that it knew tonight?What an agent knows tomorrow is whatever was written down tonight and read back; nothing else survives the night.5 minute read
Questions for the Vendor, No. 10Where does what it remembers live, and can I read it?What an agent remembers should sit in a store your company owns, as lines a person can open, search and correct.5 minute read
Questions for the Vendor, No. 11What decides whether a document it found is relevant enough to use?A search always returns something, so the question is what number decides that something was near enough to use.4 minute read
Questions for the Vendor, No. 12Where does my company's material sit relative to yours?Your contract, the vendor's method and a web page are three different kinds of sentence, and only separate shelves keep them so.4 minute read
Questions for the Vendor, No. 13What is in the part of the prompt that never changes, and what is in the part that does?The briefing is re-read every turn, and its order, not its length, is what decides the reading half of your bill.4 minute read
Questions for the Vendor, No. 14Show me the bill for the same conversation with caching on and offTwo figures for one conversation is a small ask, and a vendor who cannot produce them has never priced their own design.4 minute read
Questions for the Vendor, No. 15Which model answers which kind of question, and who decided?One conversation is several calls, and a design either has a written table saying where each one goes or it does not.4 minute read
Questions for the Vendor, No. 16Can I move it to a different model provider next quarter, and what quietly stops working when I do?Moving providers is rarely the hard part; the hard part is the small extras that travel badly and the way they fail.4 minute read
Questions for the Vendor, No. 17What can it do at three in the morning with nobody watching?A schedule is a trigger, and the useful answer says what the trigger is allowed to set off while everyone is asleep.4 minute read
Questions for the Vendor, No. 18Which actions wait for a human, and what happens when the human is slow?Every pause needs a rule for silence, and the only safe rule is that silence decides nothing at all.4 minute read
Questions for the Vendor, No. 19After the human answers, does the job finish itself?Approving a thing and doing it are two acts, and a design that merges them removes your last chance to look.4 minute read
Questions for the Vendor, No. 20Show me the record of the last thing it did and whyAsk for one entry and watch who has to be called, because the answer is about custody rather than about logging.4 minute read
Questions for the Vendor, No. 21Can a failure to write that record stop it from working?Ask what the agent does when its own audit write fails, and find out whether the ledger records the work or gates it.4 minute read
Questions for the Vendor, No. 22How many specialists can it consult at once, and what stops it consulting all of them every time?Ask how many specialists it may ask in one breath, then ask what restrains it, because a ceiling is not a brake.4 minute read
Questions for the Vendor, No. 23Can I change a specialist's instructions without a redeploy?Ask to change one line of a specialist's instructions during the demonstration, and see whether policy is yours or the vendor's release schedule.4 minute read
Questions for the Vendor, No. 24How do you test that it refuses a bad instruction?Ask how refusals are tested, and ask to see the last change the test refused to let through.4 minute read
Questions for the Vendor, No. 25How many messages will it send one person in an hour, and does it know when they are off?Ask for the number of messages one person may receive in an hour, and where the agent learns that they are on leave.4 minute read
Questions for the Vendor, No. 26When a rule is broken, what is the model told?A guard that refuses in silence teaches the agent nothing, so it tries the same thing again until something gives way.4 minute read
Questions for the Vendor, No. 27Whose account does it act from when it sends an email?The account an agent borrows to send with is also the account it can read, search and delete from.4 minute read
Questions for the Vendor, No. 28How much of this system is the agent, and how much is everything around it?The loop that reasons is a fraction of a working system; the guards, the ledger, the connections and the screens are the rest.5 minute read
Questions for the Vendor, No. 29For each part of my business, can it act, propose, or must it ask?Authority is one word against each department, chosen by a person and read by code the model cannot argue with.4 minute read
Questions for the Vendor, No. 30Who is on the list of people it may speak to, and what happens when a stranger emails it?A short list of named people is the difference between an agent your company uses and an address the internet can talk to.4 minute read
Questions for the Vendor, No. 31When it pauses for my approval and my answer is vague, what does it do?A half approval either goes back to you as a question or gets read as a decision, and the vendor knows which.5 minute read
Questions for the Vendor, No. 32Before it builds me a board pack, do I see the plan or only the result?A plan shown before the work costs one cheap turn; a plan explained afterwards is a receipt.5 minute read
Questions for the Vendor, No. 33What does it watch when nobody messages it, and what stops a noisy source from ringing all day?Between a source changing and your phone ringing there is either a count you can be shown or nothing at all.5 minute read
Questions for the Vendor, No. 34How was it examined before it met my company, and what is the pass mark?An examination you can read and a change it once refused are worth more than any score a vendor quotes you.5 minute read
Questions for the Vendor, No. 35The twelve questions, on one pageTwelve questions, one printable side, each with a good answer you can recognise and a weaker one you will hear instead.6 minute read