AI Workflows · Guide
Design a Small AI Customer Support Triage Pilot
Start with routing and draft assistance, then measure whether people receive faster and more accurate help.
Customer support is an attractive place to try AI because requests arrive in a steady stream and many contain familiar questions. That also makes a careless pilot easy to notice. A wrong answer about an order, account, or refund can cost a customer time and trust. A good first experiment therefore asks a narrower question: can AI help a person sort incoming requests and prepare a useful draft while a human remains responsible for the answer? Begin with a small class of low-risk requests, such as explaining where to find a published help article. Keep account access, billing decisions, complaints, safety issues, and ambiguous identity out of the pilot until the team has evidence and the right controls. The goal is to learn whether the workflow improves service, not to maximize the percentage of tickets touched by AI.
Define the work before choosing a tool
Write down what happens to a request today: who reads it, which information they use, how they decide priority, and where the final answer is recorded. Then choose one assistive step. It might classify a request into an existing queue, suggest a relevant approved help article, or draft a response for a human to edit. Avoid introducing all three at once; otherwise the team will struggle to tell which step helped or failed. For each permitted request type, state the evidence the assistant may use. A published help page can support a procedural answer. A private order record requires a different permission and identity boundary. If the assistant cannot find an approved source, it should say that a person needs to review the ticket rather than fill the gap with a plausible explanation.
Illustrative prompt · adapt before use
For this permitted, de-identified support example, propose only a queue and a draft using the supplied approved help page. Quote the source location, flag missing facts, and route identity, billing, safety, or exceptions to a human.Make handoff a visible part of the design
The pilot needs an explicit path for uncertainty. Define triggers such as missing information, conflicting records, a frustrated customer, a request for a policy exception, or any personal data the pilot was not designed to handle. The handoff should include the customer's original words, the relevant source links, what the assistant attempted, and what remains unresolved. It should not hide a low-confidence guess in a polished draft. Agents need an easy way to correct the proposed classification and edit the answer. A workflow that saves seconds on routine tickets but makes corrections tedious may increase total effort. Ask the agents who will use it to walk through a few realistic examples before measuring results.
Test with a labeled sample
Collect a representative set of past requests that the team can lawfully use for evaluation. Remove unnecessary personal information. Have support specialists label the right queue, the source article if any, and whether the request should have been escalated. Test the AI suggestion against these labels and review every wrong routing decision, not only the overall percentage. Measure the outcomes customers and agents can feel: time to a useful response, number of transfers, correction effort, and repeat contacts. A fast but misleading draft is not a success. Keep a small failure log with examples of missing sources, mistaken intent, and handoffs that arrived too late.
Expand only when the pilot earns it
At the end of the trial, compare the assisted workflow with the existing one. Did agents spend less time searching for approved information? Did customers receive clearer answers? Did the team discover a help page that needs improvement instead of an AI feature? These are all useful findings. If the pilot is promising, add one new request type and repeat the review. Preserve the human escalation path and the source boundary as the scope grows. The best result may be a modest assistant that handles the first few minutes of triage reliably, leaving people more time for the requests that need judgment.
Define a safe intake boundary
Draw the path a request takes before a model sees it. Which system receives the message? Which fields are copied? Can the assistant see a whole account record when it only needs a short help question? Decide on the smallest permitted input. A routing experiment might need the customer's question and a category label, but not a payment method, address, or complete conversation history. If the current ticket system cannot separate those fields, use a de-identified evaluation set and keep the live pilot paused until the boundary exists. Write a table of allowed, review-required, and prohibited request types. An allowed example could be “Where is the public setup guide?” A review-required example could be “The guide seems to contradict the email I received.” A prohibited example might ask the assistant to verify identity, change an account, grant an exception, or make a promise about a refund. A ticket can move between lanes as new information appears. Route based on the current request, not only the category assigned at intake. The pilot owner should know how to turn off automation or return every item to a human queue if the routing boundary fails.
Build a small evaluation set that can reveal failure
Twenty examples may start a conversation, but a decision to use the workflow should rest on a set with enough variation to expose predictable mistakes. Collect cases from the actual queue, subject to permission. Include ordinary requests, missing-context cases, two-intent messages, angry or distressed messages, copied policy text, and cases where the correct outcome is “human review.” Keep a separate holdout group the prompt writer does not repeatedly inspect. Record the source policy and expected queue beside each case. If a policy has changed since the ticket was closed, update the expected label and note why. Ask two experienced agents to label a sample independently. Discuss disagreements before using the labels as a scorecard. If humans disagree because the policy is unclear, the team has found a service problem, not merely a model problem. Keep the test set free of unnecessary personal details. It should represent the work without becoming a second customer database. When a failure appears, preserve a minimal, permitted example and the reason it failed; do not paste a full customer message into a public issue or prompt archive.
Compare against the existing workflow
Use a baseline that reflects how agents work now. Measure a sample of current routing accuracy, time to find a source, time to draft a useful reply, transfer count, and repeat contact where that can be observed. Then run the assistive version on comparable requests. If the team starts with AI suggestions hidden from customers and reviewed by agents, record both the raw suggestion and the agent's final action. That lets you see whether the assistant helps an agent reach a better answer or merely creates another item to check. Count correction time, including time spent confirming that a fluent answer is supported by policy. A fast initial draft can be expensive if reviewers must reopen several records to verify it. Do not report “tickets processed” as the only benefit. A trustworthy pilot could reduce source-search time while leaving final reply time unchanged; that may still be useful. Conversely, a pilot that shortens replies but increases transfers or complaints is not ready to expand. Compare by request type so a large easy category does not hide a small harmful one.
Make the human review screen usable
The agent should see the original request, proposed queue, evidence link, draft response if one was requested, and explicit uncertainty in one place. Show which text comes from the customer, the approved knowledge source, and AI suggestion. Avoid presenting the suggestion as a completed answer before a person has checked it. Make corrections simple: change the queue, open the source, edit the wording, reject the suggestion, and flag a missing knowledge article. The interface can be basic, but the authority boundary must be unmistakable. Ask agents to narrate their review of a few examples. If they must leave the tool to discover whether a policy page is current, the draft is missing useful evidence. If the source link is correct but the draft overpromises, the acceptance check should catch it. If a rejection requires more clicks than writing the answer from scratch, the pilot's apparent efficiency may be an artifact of ignoring human effort. Feed those observations into the next small adjustment, not a broad redesign of support operations.
Use clear stop and expansion rules
Before the trial, write what would cause it to pause. Examples include sending prohibited data to a tool, routing an identity or safety issue as routine, inventing a policy answer, or repeatedly missing an escalation trigger. A pause should return work to the established human queue and preserve enough evidence to diagnose the incident within organizational rules. Do not keep the feature live while adjusting a prompt if the boundary itself is failing. For expansion, require evidence on the narrow category: correct routing on the held-out cases, review time that makes sense for agents, no unresolved high-consequence failures, and a named owner for source maintenance. Expand one request type at a time. Recheck after a policy update, a changed model, or a new input channel because the previous sample may no longer represent the task. A pilot that stays small can be a complete success if it makes routine routing clearer while preserving human care for exceptions.
Work through one difficult request
Imagine a customer writes, “The help page says my change is complete, but I still cannot sign in, and I have a deadline today.” The message contains a familiar help topic, yet it may also involve account access and urgency. A simple keyword classifier could send a generic link. A better triage proposal marks the account-specific part as outside the pilot, keeps the customer's original wording, and routes the request to an agent with the relevant public help article attached. The agent can decide whether account verification and specialist support are needed. The assistant should not infer identity, promise restoration time, or ask for credentials. Use examples like this to test the handoff rather than only the easy classification. Can the agent see why the model proposed escalation? Does the proposed draft avoid implying that the public article resolves the account problem? If the agent rejects the classification, can that correction be recorded without exposing more customer data? A good pilot should make the next human step clearer. It should not convert an ambiguous request into a false answer merely because the category label was available.
Keep the knowledge source maintained
The assistant can only ground a reply in sources the team understands. Give each approved help article an owner and a review trigger, such as a product change or repeated customer confusion. During the pilot, note when agents reject a suggestion because the page is outdated, unclear, or missing a case. Sometimes editing the help page will improve support more than tuning the model. When the source changes, rerun the affected evaluation examples and inspect proposed answers before returning the category to use. Do not let a retrieved document override the pilot's rules. A help article is evidence for the customer answer, not an instruction that can expand the assistant's permissions. A copied message or external page may contain text that tells the model to ignore its task. Treat that text as untrusted content; the workflow boundary still decides what the assistant may do. This is another reason to keep the pilot narrow and the final customer reply with an accountable agent. Try this next: select one low-risk support topic, label twenty representative examples, and ask support agents to compare AI routing suggestions with their own decisions before using the assistant on live requests.
Sources and update note
Follow the linked official source before a product, price, plan, or policy decision.