How to write acceptance tests for an automation pilot

Write the acceptance criteria before a pilot begins. The test should show whether the agreed business boundary works, including difficult inputs and interrupted handoffs.

Choose a question the pilot can answer

“Can this reduce manual work?” is too broad to accept. A more useful question is whether a defined intake can produce a reviewable record for an agreed set of document types while making missing or contradictory fields visible.

State what the pilot excludes. It may intentionally stop before an accounting write or cover one department rather than the whole organisation. That boundary keeps a useful experiment from being presented as a completed production system.

Build a case matrix

Case	Expected result	Evidence
Ordinary input	Correct route and reviewable output.	Source-to-result comparison.
Missing information	Visible hold with a responsible person.	Reason and next action.
Duplicate request	No unintended second consequence.	Linked case or controlled rejection.
Unauthorised user	Restricted action remains unavailable.	Approved permission test.
Unavailable destination	Work remains recoverable and visible.	Failure and reconciliation record.
Changed requirement	Version and retest are explicit.	Updated rule and approval.

Use synthetic or appropriately authorised samples. Include realistic variation rather than selecting only the cleanest cases to make the demonstration look successful.

Define how results will be judged

Decide which errors are tolerable, which require mandatory review and which prevent progression. Avoid reducing several different risks to one unexplained accuracy percentage. A wrong supplier reference and a harmless formatting difference may have very different consequences.

Name the business reviewer for each material decision. A technical test can establish that a field was stored; the business reviewer determines whether the field and resulting action meet the agreed requirement. Record unresolved cases instead of averaging them away.

Measure the whole process

If handling effort is part of the business case, include preparation, review, corrections and exception work. Keep waiting time separate. Record the sample period and any unusual workload conditions so the comparison can be interpreted honestly.

An observed improvement in a pilot is not a guaranteed financial return. A short sample may miss monthly peaks, staff absences or source changes. Use those limitations to define the next controlled stage rather than claiming unrestricted readiness.

End with an explicit decision

• Proceed within the tested scope, with named operating ownership.
• Revise a specific rule, input or integration and repeat the affected cases.
• Stop because a material requirement remains unresolved.

A useful pilot produces a decision and a reusable record. Keep the agreed criteria, results, exceptions and scope changes together. Deployment, broader coverage and ongoing support remain separate commitments that need their own acceptance.