Honest limits of AI employees: hallucination, edge cases, empathy, project failure and cost. Which roles stay human and how to contain the risks.
An AI employee is software that owns a process and carries cases to a recorded result. That is useful, and it has limits that a buyer should hear plainly before signing anything. This page lists the main drawbacks, what the evidence says about each, which roles should stay human or human-led, and when you should not deploy one at all. Zamp builds AI employees, so read it with that in mind. We have tried to cite outside sources wherever a claim goes beyond our own published work.
Limitation | What goes wrong | Roles most affected | Mitigation |
|---|---|---|---|
Confident errors | The system states or posts something wrong without flagging doubt | Accounting, compliance review, reporting | Sampling, read-back checks, escalation on low confidence |
Novel cases | A new vendor, rule or edge case falls outside learned patterns | AP, procurement, onboarding | Human review of first-time scenarios, written thresholds |
Irreversible actions | A payment, deletion or public message cannot be undone | Treasury, records, communications | Approval gates before the action |
Emotional context | The system misses distress, tone or history | Support, HR, collections, account management | Keep a person on the conversation |
Weak scoping | The project targets a vague goal with no baseline | Any function | One workflow, one metric, one owner |
Fragile integrations | A screen change or access change stops the work | Teams on customized ERPs | Monitoring, API access where possible |
Customer preference | Customers want a person and say so | Customer service | A clear route to a human |
The first limitation is the one people notice first. Language models sometimes produce statements that sound right and are not. A 2025 paper from researchers at OpenAI and Georgia Tech, Why Language Models Hallucinate, argues that this persists partly because most benchmarks score a confident guess higher than an honest "I don't know." The authors describe models as optimized to be good test-takers. The paper is an analysis of causes and of how evaluations are graded. It does not give a rate for any particular product, so treat it as an explanation, not a measurement of your vendor.
For a back-office role, the practical point is that a wrong answer arrives looking identical to a right one. Zamp's own guide to finance automation accuracy and compliance makes a related point about vendor claims. Accuracy figures often measure different things, such as field extraction, coding, match rate or touchless rate, and a high match rate can hide wrong matches unless someone samples the matched population. A platform that escalates low-confidence cases protects you from its own errors. One that guesses looks faster and creates cleanup work.
AI employees do best on repetitive, rule-dense work. They do worse the first time something new shows up. Zamp's post on human-in-the-loop AI names the cases that still need a person: high financial or legal exposure, genuine ambiguity, irreversible actions and first-time scenarios. Its examples include approving a six-figure vendor payment against a duplicate invoice and closing a fraud case that should have stayed open. A mis-drafted marketing email is an annoyance. Those are different problems.
The same post makes the point that review is not free. Forcing a human to check low-risk, high-volume tasks adds no safety and brings back the bottleneck you set out to remove. So the limitation is not that AI cannot do the work. It is that someone has to decide, in writing, which cases it may close alone.
Many of the drawbacks of AI employees are about the project, not the model. Gartner has predicted that more than 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. This is reported in Outlook Business, which also quotes Gartner analyst Anushree Verma saying most agentic AI projects today are early-stage experiments or proofs of concept that are driven by hype and often misapplied. The same report says Gartner estimates only about 130 of thousands of agentic AI vendors are genuine, with many rebranding assistants, robotic process automation and chatbots. That is a prediction and an estimate, not a count of failures.
Zamp's post on the AI project failure rate reaches a similar place from the operations side. It notes that analyst estimates for enterprise AI failure often run between 70 and 85 percent, and it ties failures to unclear return targets, data that was never production-ready, skipped human review design, underestimated integration cost and no accountable owner. Note that those percentages are estimates quoted in that post, and definitions of failure vary.
An AI employee that works through other systems depends on those systems. Zamp's walkthrough of AP invoice exception handling is open about one trade-off. When an agent operates an ERP through the screen instead of an API, it can work where API access is locked down, but it is slower than a direct API call and sensitive to changes in the interface. A buyer with a heavily customized ERP should ask how the vendor handles that.
They can write text that reads as caring, and in one study evaluators rated chatbot replies as more empathetic than physician replies. They still struggle where empathy depends on a relationship: reading a distressed caller, knowing a person's history, and being accountable for the outcome. Keep a person on those conversations.
The evidence here is mixed, and it is worth reading closely. A 2023 study in JAMA Internal Medicine compared ChatGPT answers with physician answers to 195 patient questions posted on a public forum. Licensed health professionals preferred the chatbot answers in 78.6% of cases, and rated 45.1% of chatbot answers as empathetic or very empathetic against 4.6% of physician answers. That sounds like a win for the machine, but the authors list limits. The setting was an online forum, not a real care relationship. Chatbot answers averaged 211 words against 52 for physicians, and longer answers may score higher on empathy. The responses were not checked for accuracy. Some evaluators also had financial interests in AI.
So the fair reading is narrow. A model can produce warm, well-structured written replies. The study did not test whether it can notice that a customer on the phone is frightened, remember last quarter's dispute, or take responsibility for a bad outcome. Those are plain reasoning, not findings from the paper, and they are where we would keep people involved.
There is also evidence from the market. According to Tech.co, Klarna laid off around 700 employees in 2022 as it moved to AI, and by February 2024 the company claimed its AI assistant was handling 75% of customer chats. Tech.co reported that the company later began rehiring humans, with its CEO Sebastian Siemiatkowski saying it is critical that customers know there will always be a human if they want one. Klarna described the change as a dual-track approach that combines scalable AI with high-quality human support, not as a rejection of automation. It is one company's experience and the reasons were mixed, but it shows what happens when emotional and brand-sensitive contact is automated without a human option.
A role is not one task. Most jobs mix routine steps with judgment, and the split is where the decision lies. Based on Zamp's published guidance and the sources above, keep these human or human-led: roles that approve large or binding commitments, such as a controller signing off a payment above a set threshold or a lawyer accepting terms outside a template; roles that deal with people in distress, such as collections conversations about hardship, sensitive HR cases and complaint escalations; roles that own relationships over years, such as a key supplier manager or an enterprise account lead; and roles that must make a call in a new regulatory or fraud situation with no precedent.
In each of these, the AI employee can still gather the data, draft the response and flag the anomaly. Zamp's post on AI employees vs AI copilots draws the practical line: a copilot helps a person who still owns the process, while an AI employee owns the process and sends back the cases that need judgment. For personal, interactive work, such as drafting a customer email, that post suggests a copilot is often the better fit.
Approval gates. Put a human sign-off before payments above a threshold, binding terms and any action that cannot be undone. Zamp's guidance is that thresholds should be written down, covering dollar amounts, confidence scores and action types, not left to individual feelings about risk.
Scoped permissions. Give the AI employee only the access its process needs, and know what it is allowed to do. Zamp's post on tools that execute workflows describes real execution as taking actions inside other systems using permissions it has been given.
Audit trails. Every decision, automated or human-approved, should leave a record of what the system saw, what it decided and what it changed. The finance accuracy guide adds a harder test: can each action be reconstructed later, and can preparing, approving and posting sit with different identities so segregation of duties holds?
Read-back checks. After the AI employee posts a record, have it compare what it wrote to the source document, and have a person sample the results. This is our reasoning, not a published figure, but it follows from the sampling point above.
Escalation with context. When the system is unsure, it should route the case to a named person with the evidence attached. In the invoice walkthrough, a suspected duplicate is held for human review with the possible match shown side by side, and unresolved exceptions escalate on a timer with a tracked audit trail.
Start tight and widen. Zamp's human-in-the-loop post recommends starting with tighter oversight, measuring performance against real decisions, and loosening the checkpoint as the system earns a track record on that task. For a deeper definition of the category, see the complete guide to AI employees.
Wait, or skip it, if the process is not written down, because then there are no rules to follow and no thresholds to escalate on. Skip it if volume is low and setup effort will not pay back, or if the work is mostly personal conversation or negotiation. It is also a poor bet when nobody on the operations side can own the result, when there is no baseline metric to show whether it helped, or when your data is inconsistent and nobody has time to clean it. And if the only reason to buy is that a competitor did, do not.
Zamp's own material says the autonomous stage comes after a company has written down its process rules and set escalation thresholds. If those steps feel far away, a copilot or a process cleanup may serve you better this year.
