AI Vendor Evaluation Checklist: Scoring Tools and Partners

An AI vendor evaluation checklist scores every candidate on the same nine criteria, weighted before anyone sees a demo. The nine are production evidence, eval practice, data handling and EU residency, security, pricing transparency, ownership of code and prompts, exit terms, team seniority and references. Score each from 0 to 5. Treat a few answers as disqualifying, whatever the total says. The same checklist works for an AI product and for a partner who builds one for you.

Fix the weights before the first call. AI demos are very good, and a good demo bends every score that comes after it.

What vendor selection means when the vendor sells AI

Vendor selection is the process of choosing an outside supplier against criteria you set in advance. You define the need, shortlist candidates, score them the same way and sign a contract you can live with and leave. For ordinary software, the process is mature. AI makes it harder in four places.

• The output is probabilistic. A tool that handles your case today can fail next month, after its provider swaps the model underneath.
• Your data travels further, through the vendor, a model provider and a host, each with its own terms.
• Pricing follows usage. A per-seat price can hide a per-call cost that appears when volume grows.
• Prompts, eval sets, retrieval pipelines and fine-tuned weights are assets. Contracts written for ordinary software often ignore them.

There are also two kinds of AI vendor, and buyers often evaluate one as if it were the other. An AI product or SaaS tool is software someone else built and runs, such as a support assistant or document extraction. You judge the software, the company and the contract. An AI development partner builds something for you. You judge people, process and what you will own at the end.

The nine criteria apply to both, with different emphasis. A SaaS vendor runs the system with your data inside it, so security and data handling matter most. With a partner, the people and the contract are the product. We are a development partner ourselves, so we wrote down the criteria we would want to be judged on.

The AI vendor evaluation framework as a scoring matrix

Score each criterion from 0 to 5 and multiply by its weight. Divide the sum by 5 for a score out of 100. The weights below suit a development partner. For a SaaS tool, move five points from team seniority to security and five to data handling.

Evidence: production, evals, people and references

Production evidence carries the most weight because it is the hardest to fake. Ask for a system that has run in production for at least six months, with real users. Then ask what broke. Every production AI system has broken somewhere. A vendor who says nothing went wrong has not shipped much, or is not telling you.

Eval practice separates products from demos. An eval suite is a set of real inputs with known correct outputs, run whenever the prompt, model or code changes. Ask a SaaS vendor how they knew quality held the last time their model provider shipped a new version. Ask a partner when they write the evals. The right answer is before or alongside the build, running in CI. We will not ship an AI feature without one, and you should not buy one without one.

Better still, bring your own test set. Fifty real cases with answers your team agrees on tell you more than any vendor benchmark. A vendor who resists a trial on your data is telling you something.

Team seniority matters most with a partner. Ask who will write the code, and put those names in the contract. Agencies often pitch with a senior architect and deliver with a team you never met. Two senior engineers who have shipped this before usually beat a large team learning on your budget. With a SaaS tool, seniority shows up as support. Ask who answers when the product gives your customer a wrong answer, and how fast.

References carry the least weight because the vendor chooses them. Ask for a list and pick yourself. Then ask each one what the vendor got wrong, and how it was handled.

Data handling, security and EU residency

For an EU company, data handling is where most AI vendors lose points. 'We are GDPR compliant' is a claim about the vendor, not an answer about your data.

You want specific answers. Where is inference processed, and in which region? Which subprocessors touch the data, including the model provider? Are prompts, files and outputs retained, for how long, and are they used for training? Does a data processing agreement under GDPR Article 28 cover all of it? If the vendor's role is unclear, the EDPB guidelines on controllers and processors settle it.

EU residency is achievable for most use cases now. The major models run on EU region endpoints through AWS Bedrock, Azure OpenAI and Google Vertex AI. Open weights models can run on hardware you control. A vendor that cannot say which it uses has not thought about it.

Ask about the EU AI Act, but ask the right question. Classification follows your use case, not the vendor's model or hosting, so no vendor can certify your deployment. A good one shows how its product supports your duties. Telling people they are talking to an AI has been required since 2 August 2026. Hiring, credit and education use cases face high-risk obligations from 2 December 2027. Those need logging, version history and real human oversight in the product.

Security covers the usual ground plus one new item. The usual ground is SOC 2 Type II or ISO 27001, SSO, role-based access, audit logs and a recent penetration test. The new item is prompt injection. If the system reads documents or email and can act, ask how hidden instructions are stopped from triggering actions. Smaller development partners rarely hold certifications. Ask how they handle your credentials and secrets, and whether their access ends with the project.

Price, ownership and the exit

Pricing transparency means you can predict the bill at your volume and at ten times it. AI tools often price per seat with a hidden usage cap, or per call with the volume left to you. Get both numbers in writing. From a partner, read the line items. Without evals, observability and a runbook, you are paying for a demo. Ask what the system will cost to run each month, and how they got there.

Ownership is where AI contracts are weakest. Standard software contracts assign the code and forget the rest. Yours should name code, prompts, eval sets, retrieval pipelines, embeddings and fine-tuned weights. From a partner, those become yours as each phase is paid. Watch for proprietary middleware. If the build only runs on their licensed runtime, you bought a subscription that looks like a build.

Negotiate exit terms while everyone is still friendly. You want a documented export of data, configuration, prompts and logs. You want a defined window to take it, and written confirmation of deletion. The EU Data Act requires cloud and SaaS providers in its scope to support switching. From 12 January 2027 it also bans switching charges. Do not rely on that alone. Put the export format in the contract.

With a partner, exit means a handover another team can pick up: documentation, a runbook and admin access to every repository and account. The EU model clauses for AI procurement target public buyers, but private contracts can borrow their wording.

Questions to ask on the call

Send these in writing before the call. A vague written answer says more than a vague spoken one, because someone had time to think.

For an AI product or SaaS tool

1. 01Where is our data processed, in which region, and by which subprocessors?
2. 02Are our prompts, files or outputs used to train any model, yours or your provider's?
3. 03When your model provider releases a new version, how do you know our quality did not drop?
4. 04Can we run a trial on our own data, with our own test cases?
5. 05What will this cost at our volume, and at ten times our volume?
6. 06How do we export everything when we leave, in what format, and how long do we have?
7. 07How does the product tell our users they are talking to an AI?

For an AI development partner

1. 01Show us a system you built that has been in production for six months or more. What broke?
2. 02Who will actually do the work, and can their names go in the contract?
3. 03When do you write the eval suite, and does it run on every change?
4. 04Who owns the code, prompts, eval sets and any fine-tuned weights once we pay?
5. 05What will this cost to run per month, and how did you calculate it?
6. 06What happens if we stop after the first phase?
7. 07When did you last tell a client not to build something?

The last question matters most. A partner that has never advised against a build is selling hours, not judgment. Listen for a specific story.

Red flags that end the evaluation

Some answers are not a low score. They end the conversation.

• The demo only runs on their sample data, and they resist a trial on yours.
• 'GDPR compliant' or 'enterprise-grade security' is the full answer to a specific question.
• Training on your data is on by default, with the opt-out buried in settings.
• An accuracy percentage with no test set, no definition of correct and no date.
• A fixed build price before anyone has looked at your data or systems.
• The build runs only on the vendor's proprietary runtime, under a license you pay forever.
• Code and prompts become yours only at the end of the whole program, or never.
• The delivery team is unnamed, or changes between pitch and kickoff.
• A claim that the product makes you EU AI Act compliant, whatever your use case.
• Every reference is a pilot. None are in production.

Running the evaluation in two weeks

For most AI purchases, two focused weeks are enough if the steps run in order.

1. 01Write the need in one paragraph and list your disqualifiers. Get both signed off before any vendor call.
2. 02Shortlist three vendors. With more, nobody scores them properly.
3. 03Send the written questions with one deadline for everyone.
4. 04Run a trial on your own data, against your own test set.
5. 05Have two people score each vendor separately, then discuss the gaps.
6. 06Call the references you chose.
7. 07Negotiate ownership and exit before price. Price is easier to move later.

Want engineers who build these systems to run it with you? Vendor and build or buy questions are part of our consulting work. It ends in a written document, and you can take it to any supplier.