Free template
AI vendor scorecard
Use this to compare AI vendors side by side and reach a choice you can defend to finance, security and procurement. Copy it into a document or spreadsheet and change whatever doesn't fit your situation.
Fields in [square brackets] are yours to fill in.
Before you score anything
Write these down first. If you can't, the scorecard is too early.
- Goal: [the business result you need, in one sentence]
- Metric: [the one number that would prove it worked]
- Decision owner: [who owns the budget and makes the call]
- Who else must agree: [for example finance, security, procurement, the team that will use it]
- Vendors compared: [two or three is usually enough]
Step 1: Set your must-haves
A must-have is a yes or no question. A vendor that fails one is out, however well it would score elsewhere. Keep the list short, so that each item is a real deal breaker.
| Must-have | [Vendor A] | [Vendor B] | [Vendor C] |
|---|---|---|---|
| [Can meet our data location and handling rules] | Yes / No | Yes / No | Yes / No |
| [Works with the systems we must connect to] | Yes / No | Yes / No | Yes / No |
| [Will go through our security review] | Yes / No | Yes / No | Yes / No |
Step 2: Agree the weights
Agree the weights before any demo, and get the decision owner to sign off on them. Weights set after a demo tend to drift toward the favorite.
Start from these five criteria and adjust. The weights must add up to 100%.
| Criterion | The question it answers | Starting weight |
|---|---|---|
| Fit to the goal | Will this move the metric we chose? | 30% |
| Time to value | How soon, and with how much work from us, would we see a result? | 20% |
| Security and data | Would our security team be comfortable, and is our data handled the way we need? | 20% |
| Total cost | What does it cost over the full term, including the work on our side? | 15% |
| Vendor strength | Will this vendor be there and able to support us through the term? | 15% |
Some reasons to move the weights:
- Regulated or sensitive data: raise security and data.
- A deadline you can't move: raise time to value.
- A tight budget: raise total cost, and make sure it includes your own team's time.
- A long contract or a core process: raise vendor strength.
Keep fit to the goal the largest weight unless you have a clear reason not to. A cheap, quick tool that doesn't move the metric is still a cost.
Step 3: Score from 1 to 5
Score each vendor on each criterion, using evidence rather than impressions. Agree what the numbers mean up front:
| Score | Meaning |
|---|---|
| 5 | Clearly meets the need, and we have seen proof on our own work or data |
| 4 | Meets the need, with good evidence from the vendor or its customers |
| 3 | Partly meets the need, or meets it with gaps we can live with |
| 2 | Meets the need only with significant work or workarounds |
| 1 | Doesn't meet the need, or the vendor gave no evidence |
Have the people closest to each criterion score it: security scores security, finance scores cost. Discuss any large disagreements, and note what changed the score.
Step 4: Ask each vendor for the same evidence
Send every vendor the same list. A vendor that won't answer a question has told you something, so score it accordingly.
| Criterion | Ask for |
|---|---|
| Fit to the goal | A demo on your use case rather than the standard script. A trial on your own work or data. Two current customers with a similar problem whom you can speak to directly. |
| Time to value | A plan for getting started: the steps, who does each one, and what they need from your team. What a customer needs in place before work can begin. |
| Security and data | Their security documentation and current certifications. Where your data is stored, who can access it, how long it is kept, and whether it is used to train models. |
| Total cost | A full written price for your expected use over the term, including implementation, support, usage-based charges, renewal terms and what happens to the price as you grow. |
| Vendor strength | Who would work on your account. How they handle support and product changes. Enough about the company's ownership and finances to judge whether it can support you through the term. |
Then run a proof of value with your top choice. Set the pass mark before it starts, so that the result is a clear yes or no. How to run an AI pilot that leads to a decision covers this step.
Step 5: Fill in the scorecard
For each vendor: multiply each score by its weight, then add up the results.
Goal: [goal]
| Criterion | Weight | [Vendor A] | [Vendor B] | [Vendor C] | Evidence and notes |
|---|---|---|---|---|---|
| Fit to the goal | [ ]% | [1 to 5] | [1 to 5] | [1 to 5] | [ ] |
| Time to value | [ ]% | [1 to 5] | [1 to 5] | [1 to 5] | [ ] |
| Security and data | [ ]% | [1 to 5] | [1 to 5] | [1 to 5] | [ ] |
| Total cost | [ ]% | [1 to 5] | [1 to 5] | [1 to 5] | [ ] |
| Vendor strength | [ ]% | [1 to 5] | [1 to 5] | [1 to 5] | [ ] |
| Weighted score | 100% | [ ] | [ ] | [ ] |
Next step: [proof of value with the top vendor, then the business case, or stop]
Step 6: Test the result
Before you call it, check that the answer holds up:
- Change the weights. Move a few points between criteria. If the winner changes with a small shift, the vendors are close, and that is what the proof of value should settle.
- Look at the low scores. A strong total can hide a score of 1 on something that matters.
- Remember "none of them" is an option. If no vendor would move the metric, say so. Stopping here costs far less than stopping after rollout.
The score supports the decision and doesn't make it. If the team's preference differs from the numbers, find out why. Either a weight is wrong or the preference is, and it is worth knowing which.
Filled-in example
This example is made up. The company, the vendors and every score are invented to show how the math works. They say nothing about any real product.
Goal: Our support team spends less time looking for answers.
Metric: Average time to resolve a support ticket.
Must-haves: All three vendors passed.
| Criterion | Weight | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|
| Fit to the goal | 30% | 5 | 3 | 2 |
| Time to value | 20% | 4 | 3 | 5 |
| Security and data | 20% | 4 | 4 | 3 |
| Total cost | 15% | 3 | 4 | 5 |
| Vendor strength | 15% | 4 | 5 | 3 |
| Weighted score | 100% | 4.15 | 3.65 | 3.40 |
The math for Vendor A: (5 × 0.30) + (4 × 0.20) + (4 × 0.20) + (3 × 0.15) + (4 × 0.15) = 1.50 + 0.80 + 0.80 + 0.45 + 0.60 = 4.15.
The weight test: the team moved 10 points from fit to the goal over to total cost (fit 20%, cost 25%). The scores became Vendor A 3.95, Vendor B 3.75 and Vendor C 3.70. Vendor A still led, but by less, so cost became a topic for the business case.
The low score: Vendor C's 2 on fit ruled it out, even though it was the fastest and cheapest. It wouldn't have moved the metric.
Next step: a proof of value with Vendor A on real tickets, with the pass mark agreed before it started, then the business case.
Want help running this?
countach.ai runs evaluations like this with buyers, with any vendor that fits the problem. See how we work, or take the decision check to see where you are in the decision.