An AI pilot is not a strategy. Here is what to decide first.

An AI demonstration can make a possibility feel close. A system summarises a technical document, answers a maintenance question or drafts a customer response. The harder judgement is whether that capability belongs in the business, and what must change for it to be useful.

A pilot should reduce uncertainty about that judgement. Before spending money on one, decide which uncertainty matters. Otherwise, a successful demonstration can leave the original business question unanswered.

Start with the work you want to improve

Describe the operational problem in language the person accountable for it would recognise. Perhaps engineers spend too long finding relevant information before preparing a quotation. Perhaps maintenance planners struggle to assemble a reliable equipment history. Identify the consequence: delays, rework, missed opportunities or decisions made with incomplete information.

Then establish how the work happens today. Who does it? What takes time? Where do errors enter? What can the business measure without introducing a new reporting burden?

This gives the pilot something useful to test. It also exposes alternatives. Better document control, a simpler approval process or a conventional search tool may address enough of the problem at lower cost. Include those options in the assessment.

Be specific about the benefit. Time saved has value when it releases a constrained person for useful work, improves service or avoids an actual cost. A claim about hours saved needs an explanation of how those hours will change the business.

Decide whether the information is usable

Access to a large document collection does not establish that it is suitable for the proposed task. Versions may conflict. Critical knowledge may sit with experienced staff. Records may describe conditions that no longer apply.

Ask which sources the system can use, who owns them and how their reliability will be assessed. Decide how access permissions will work, what information can leave the organisation and how sensitive material will be handled. These questions affect the operating design and the cost of making it work.

The pilot should include representative information, including awkward or incomplete examples. A result obtained from carefully selected documents says little about performance against the material staff will encounter on a busy day.

Where data is inadequate, make the prerequisite visible. A limited information-clean-up exercise may be the right first investment. Deferring the AI work can be a sound decision when the inputs cannot yet support it.

Give someone responsibility for the outcome

Name a business owner with enough authority to change the workflow. The technology team can assess the capability and its constraints. The operational owner needs to decide whether the output is useful and how it will affect people’s work.

Agree who reviews results, resolves exceptions and responds when the system produces an unreliable answer. Staff need a way to challenge an output and continue working when the tool is unavailable.

For work with material safety, quality or commercial consequences, define the required review before testing begins. The effort of that review belongs in the value assessment. A faster draft is less useful if checking it takes longer than doing the original task.

Test the operating change as well as the capability

Consider a hypothetical engineering-services business evaluating AI to help prepare quotations. The demonstration produces a plausible scope from previous proposals. The actual bottleneck, however, is securing an engineer’s review of unfamiliar requirements.

A useful pilot would test whether the draft gives that engineer a better starting point. It would include unusual requests, outdated source material and cases where the system should flag uncertainty. The assessment would cover review effort, missed requirements and whether the quotation reaches the customer sooner.

It would also identify who maintains the source material and how the tool fits the existing quotation process. If users have to copy information between several systems, that additional work must be counted.

The business could then decide whether to expand the approach, narrow it to familiar quotation types or improve the review process first. Each result would teach it something relevant to the investment decision.

Agree the decision the pilot will support

Before kickoff, write down what would justify proceeding and what would cause you to revise or stop. Set criteria around usefulness, reliability, user effort, ongoing cost and readiness to operate the capability.

Avoid allowing enthusiasm to redefine success after the results arrive. If the pilot answers a different question from the one originally agreed, record that and decide whether further testing is worth funding.

A successful pilot should leave leaders able to answer:

What has improved against the current way of working?
Which limitations remain, and who will manage them?
What will adoption, support and ongoing operation require?
Is the next commitment justified by the evidence?

Start by putting the proposed business outcome, accountable owner and next investment decision on one page. Use that page to decide whether a pilot is the right next step. The capability earns its place when the business understands how it will use it and why the commitment makes sense.