Start with a role charter
A role charter should answer five questions:
- What work is in scope?
- Which sources may it use?
- What can it change or send?
- What must it stop for?
- Who reviews the result?
Use specific tasks, such as preparing an exception queue, rather than broad instructions like “manage finance.”
Set permissions in layers: read approved sources; prepare drafts or recommendations; execute only low-risk, reversible actions that have explicit approval; and require human approval for money movement, employment decisions, legal conclusions, access changes, or external commitments. Do not infer a tool's capabilities from the title “AI employee.”
Choose KPIs that expose quality and risk
- Completion rate: Completed runs divided by eligible runs, with exclusions documented.
- First-pass acceptance: Outputs accepted without material correction divided by reviewed outputs.
- Exception precision: Escalations that a reviewer confirms needed human handling divided by all escalations.
- Rework rate: Outputs reopened or corrected after acceptance divided by completed outputs.
- Escalation completeness: Escalations containing required context and evidence divided by reviewed escalations.
- Cycle time: Elapsed time from eligible input to accepted result, compared with the existing process.
- Control failures: Unauthorised actions, missed stop conditions, or broken audit trails. Set tolerance with the risk owner; do not hide these inside an average.
Measure a baseline before setting targets. A higher completion rate is not progress if errors, reversals, or unresolved exceptions rise. Keep denominators and exclusions visible so a team can reproduce each metric.
What do these KPIs look like in a real Zamp deployment?
We track the same measures in our own deployments, so here are real values instead of targets. Each is tied to the deployment it came from.
KPI | Example from a deployment |
|---|---|
Decision rate and manual-review rate | Biopharma procurement: week one, the agent recommended a decision on 36% of requests and sent 64% to manual review. After reviewed corrections, the decision rate was 85% and manual review 15%. |
Cost per request | Same deployment: $11.57 in week one, $14.76 in week three, $1.21 in week seven, while weekly volume went from 205 to 2,712. |
Accuracy | Freight: more than 95% browser-agent accuracy across 50+ vendor portals with no APIs. |
Hours | Freight: 20,000+ hours a year reclaimed. Billing: 150,000+. Tag validation: 8,000+. |
Two things transfer. First, the 64% in week one was the safety mechanism, so do not read a high manual-review rate at launch as failure. Second, cost per request rose before it fell, so judge it over weeks, not days.
How do reviews turn into changes?
The weekly review feeds the AOP. A reviewer's correction goes on a branch, runs against evals including a golden set, is reviewed and merges. The same loop moved the procurement rates above. Escalations carry context: a case that breaches a tolerance or lacks data becomes Needs Attention, and in freight, escalations arrived with the session recording attached. The procurement agent wrote its reasoning into SAP Ariba, which is what let reviewers check a decision without redoing it.
Read more in the procurement case study, the freight deployment, how the AOP changes and the self-learning system.
Review cadence: A practical starting point
- Daily: Review the exception queue, failed runs, and any action that affected a person, payment, or external commitment. Pause first if a control failure is unexplained.
- Weekly: Sample accepted outputs, inspect corrections and overrides, and check whether escalations reached the right owner. Use findings to update instructions or process controls, then test the changed workflow.
- Monthly: Revisit the role charter, access list, risk assessment, performance baseline, and owner assignment. Review whether the workflow still has a business purpose and whether any process or policy changed.
This cadence is a starting recommendation, not a legal standard. Higher-risk workflows may need more frequent checks.
Set explicit escalation rules
Route a case to a person when required data is missing, sources conflict, the request falls outside the role, confidence is not supported by evidence, or the next step could affect a person's rights, money, access, or legal position. The handoff should include the input, unresolved question, source references, and action the workflow did not take. A term such as Needs Attention can label cases that need review, but the label must correspond to a documented queue and owner. A status tag without a response time or escalation path does not provide oversight.
Keep the audit trail useful
For each run, retain the request or permitted input reference, the sources used, the output, actions attempted, approvals, exceptions, and final disposition according to the organization's retention policy. Limit access to the records and avoid collecting data that the workflow does not need. The NIST AI Risk Management Framework is a voluntary resource with four core functions: Govern, Map, Measure, and Manage. Use it to organize risk questions, not as proof of compliance or a certification. For broader category context, see the complete guide to AI employees, AI employee platforms compared, the state of AI employees in 2026, and the related AI analyst guide
