Home › Blog › From pilot to production

AI projects that stall at the pilot: the cause is the data

It is very common for a pilot to go well and then have no next step. The cause is usually that the pilot ran on the cleanest available data. Here is how to design a pilot that can actually spread.

5 design points Conditions for rollout The data trap

The short answer

The decisive difference between a pilot and production is the state of the data. A pilot runs on tidy cases someone selected. Production meets real data — inconsistent formats, missing fields, exceptions mixed in.

So a 90% success rate in the pilot becomes 60% after rollout, and that is where the project stops. The fix is to deliberately feed dirty data in during the pilot.

1. Do not let the owner choose the pilot data

The most common mistake. Ask someone for "a few example files" and they hand you the best-organised ones. Accuracy validated on that data differs substantially from accuracy in operation.

The fix is random sampling. Pull thirty items at random from the last three months and use them as they are. Broken formats, missing fields and exceptions come along naturally — and that is your real performance.

2. Measure the exception rate up front

The practical value of automation is what percentage passes through without a human. If 90% passes and a person reviews 10%, that is a large saving. If only 60% passes, the review burden remains and the felt benefit disappears.

The number to record in the pilot is that pass rate — not accuracy, pass rate. And classifying the failures by type gives you, directly from that list, exactly where production needs shoring up.

3. Decide the rollout targets before the pilot

Without deciding where it goes next if it succeeds, the pilot output gets built for that one team. Team names end up baked into prompts, exception rules get shaped around that team’s habits, and reuse becomes hard.

Naming three rollout teams up front changes the design: you pull the parts that vary per team out into configuration and separate the shared logic. That is the most practical device for connecting a pilot to production.

4. Test the owner-absent scenario

A pilot usually has one enthusiastic owner attached to it. If you never check whether it runs without them, it stops the moment that person moves on after rollout.

The test is simple: in the pilot’s final week, remove the owner and have someone else run it. Can they operate it from the documentation alone, and do they know what to look at when something errors?

5. Calculate the rollout cost in advance

Training — How many hours per team. The pilot owner learned while building; rollout teams receive a finished thing.

Permissions — More departments means designing who can see what — work that did not exist in the pilot.

Standardising exceptions — Unifying each team’s different exception rules, or separating them into configuration.

Running cost — API usage grows with team count. Estimating it as pilot cost × number of teams usually falls short.

Redefine what pilot success means

"It works" is too weak a success criterion for a pilot — that was already established at the demo stage. The purpose of a pilot is to judge whether it can spread.

So there should be four criteria: is the pass rate on a random sample above target, are the failures classified by type, can it be operated without the owner, and has the rollout cost been calculated. Satisfy those four and production stops being a technical question and becomes purely a budget one.

Frequently asked questions

How long should a pilot run?

Two weeks as a baseline; three to four with random-sample validation and the owner-absent test included. Longer than that and it stops being a pilot and becomes a small production build, which raises the cost of stopping.

If the pass rate comes out low, should we stop?

No. Classifying why it is low is the output of the pilot. If two of three failure types are data-format problems, that is a cleanup task rather than an AI problem, and it is likely solvable.

Can pilot output be reused in production as-is?

It depends on the design. If you named the rollout targets up front and pulled per-team differences into configuration, much of it carries over. Otherwise it needs rewriting.

What if the rollout cost is higher than expected?

Reduce the number of teams or split it into stages. Rolling out to every department simultaneously almost always fails. Adding two teams at a time and re-measuring the pass rate at each stage is safer.

A pilot that can actually spread

We build these five into the two-week prototype

Random-sample validation, pass-rate measurement and the owner-absent test, run as a fixed-scope, fixed-price prototype. Tell us your situation and we will review it and reply.

Get in touch