How to run an AI pilot that leads to a decision

A pilot that skips those steps can look busy and even popular, and still leave no one able to say whether to continue. Here is how to avoid that.

Why pilots stall

Pilots rarely stall because the technology does nothing. They stall because nobody agreed what success looked like, so every result is open to interpretation. These are the usual causes:

• No decision attached. The pilot was started to learn about AI in general, so nothing depends on how it ends.
• No owner. Someone is running the pilot, but no one with a budget is waiting for the answer.
• No baseline. The team feels faster, but no one measured how long the work took before.
• Late objections. Security, finance or procurement first hears about the project when it's time to expand.
• A protected test. The pilot ran on clean data with enthusiastic volunteers, so it says little about the wider rollout.

Each of these can be fixed before day one.

Step 1: Write the decision down first

Start with one sentence: "At the end of this pilot, we will decide whether to ___." For example, whether to roll a tool out to the whole support team, or whether to buy it at all.

If you can't finish that sentence, you may not be ready for a pilot. You may need to frame the problem first, or compare your options. A short checklist like our decision check can show you which stage you are at.

Step 2: Name the owner and everyone who has to agree

The decision needs one owner, ideally the person who holds the budget. Then list who else has to say yes, which usually means finance, security, procurement and the manager of the team that would use the tool.

Talk to each of them before the pilot starts, and ask one question: "What would you need to see to say yes?" Security may need to review how data is handled before the pilot can touch real data. Finance may want the result expressed in cost or revenue, so measure it in a way that makes that possible. Those answers will shape the pilot's design.

Step 3: Pick one metric and measure the baseline

Choose the one number that would prove it worked, and tie it to the business problem. Time to resolve a support ticket is a useful metric. Number of prompts sent is not, because it measures activity rather than results.

You can track a few secondary measures, but only one should decide the outcome. Then measure the baseline before anyone gets access. You won't be able to reconstruct it later, and without it you can't show what changed.

Step 4: Set the pass mark in advance

Agree the result that counts as a pass, and write it down before the pilot begins. It's worth setting two lines:

• Go: at or above this result, you move ahead.
• Stop: below this result, you end it.

Anything between the two means you change something and decide whether a further test is justified. Base the numbers on what the project would need to deliver to be worth its full cost, including licenses and your team's time. The right figure depends on your business, so work it out rather than borrowing one.

Step 5: Keep the scope small, but keep it real

Pick one team and one workflow. Then make the test as close to everyday conditions as you can:

• Real work, rather than a sandbox with made-up examples.
• Real users, including a few people who are skeptical and not just volunteers.
• Real data and access, approved before the pilot starts, so you don't spend the time waiting for permissions.
• Real constraints, including the systems the tool has to connect to and the security rules it has to follow.

Set an end date and a date for the decision meeting, and put both on the calendar now. The right length depends on how long the work takes to show a result, but it should have a fixed end. Without one, a pilot can drift on until it becomes an unofficial, unmeasured rollout.

Step 6: Watch usage and results separately

Track two things during the pilot, because they tell you different things.

Usage tells you whether people are actually working with the tool. Results tell you whether the metric moved. High usage with no change in the metric means people like the tool, but it isn't doing the job. A good result from a few heavy users raises the question of whether it can spread. Low usage tells you to find out why before you judge the tool.

Check in during the pilot and fix practical problems, such as access issues or missing training. Don't move the pass mark.

Step 7: Finish with a decision meeting

At the meeting you set in Step 5, compare the result with the pass mark and make one of three calls:

• Go. The result cleared the bar and the people who needed to agree are on board.
• Change. The result was promising, and there is a specific change worth testing. Name it, and set a new end date.
• Stop. The result fell short. Write down what you learned and close it. A clean stop is a useful result, because it frees up budget and attention for a better option.

Record the decision, the reasons and the numbers in a short document. People will ask later, and it stops the same debate from starting over.

After a yes: get ready for rollout

A positive result doesn't mean a project is ready to roll out. Before you start, you need a charter with named owners, data and access in place for the wider group, success metrics carried over from the pilot, and a rollout plan with teams, order and milestones agreed with the vendor.

We call this stage Phase 0. Whatever you call it, don't skip it. It's what turns a good pilot result into a live project.

If you would like a hand with any of this, here is how we work. We run the same process with whichever vendor fits your problem.