Retention, training, business associate agreements, audit logging, access control, and evaluation evidence. Written to be forwarded to whoever runs your procurement.
Vendor selection is where most organizations make their expensive AI mistake, and the meeting is usually structured to prevent hard questions: a polished demo, an enthusiastic champion, and a deadline. These six questions restore the balance. Send them ahead of the meeting so the answers arrive in writing.
The last one is the one that separates serious vendors from the rest.
Why it matters: Retention determines what exists to be subpoenaed, breached, or reviewed later. A vendor who cannot answer precisely is telling you they have not thought about it.
What a good answer looks like: A specific retention window, stated in the contract, with a zero-retention option available.
Why it matters: This is a different question from retention, and vendors sometimes answer the one they prefer. Training on your inputs means your material can influence outputs shown to other customers.
What a good answer looks like: A contractual commitment that customer data is excluded from training, not a statement on a marketing page.
Why it matters: For anyone handling protected health information this is the threshold question. For firms, the equivalent is whether they will accept confidentiality obligations that match what you owe your clients.
What a good answer looks like: Yes, without renegotiating the core obligations, and their standard agreement is available to review before you commit.
Why it matters: When something goes wrong you need to reconstruct who did what. Without logs, an incident investigation becomes a set of interviews.
What a good answer looks like: Per-user activity logs, exportable, retained long enough to be useful in an investigation.
Why it matters: In regulated settings not everyone should reach everything. Matter teams stay separated, and clinical data has a narrower audience than internal operations material.
What a good answer looks like: Role-based permissions and the ability to scope access by group, configurable by you rather than by their support team.
Why it matters: This is the question almost nobody asks, and the answer separates vendors who measure their system from vendors who demo it. Ask how they know it works on tasks like yours, not whether it is impressive in a scripted demo.
What a good answer looks like: Documented evaluation on tasks resembling yours, with a stated methodology, known failure modes, and honesty about where accuracy drops.
Send the six questions before the demo and ask for written answers. You will learn a great deal from which ones come back with specifics and which come back with reassurance. Then keep the answers, because when the tool is up for renewal, or when someone asks how the decision was made, the record is the whole defense.
Share this with whoever is running your procurement. It is written to be forwarded.
For a structured way to put this into practice, see AI Advisory Retainer.