AI Tool Decisions · Guide
Measure the Human Review Cost of an AI Workflow
Count the time spent checking, correcting, and escalating AI output before deciding whether a tool actually saves work.
Measure AI workflow value per accepted result. Record generation time, review time, corrections, escalations, and rework for representative tasks, then compare the full process with the manual baseline. Separate routine cases from exceptions and keep quality vetoes outside a simple average.
Choose a finished unit of work
Start with a result that the team can recognize: one approved customer reply, one publishable article section, one correctly categorized file, or one verified research brief. A model run is not a finished unit. It may create three candidate drafts, each requiring review, and still leave a person to perform the original task. Write an acceptance rule for the result before timing anything. It should say what must be accurate, complete, properly sourced, formatted, and authorized. Keep the rule close to the real downstream use. A summary that is acceptable for private brainstorming may be unacceptable as a public answer. Measuring per accepted result prevents a tool from appearing efficient by producing more material that nobody can use. It also makes comparison fair when one workflow drafts rapidly but another needs fewer corrections. Ask the people who own the result to approve the acceptance rule, since they know which defects turn into actual work later.
- Define the result, not the prompt.
- Write an acceptance rule before testing.
- Ask the downstream owner to confirm it.
Build a representative work packet
Collect tasks from the work stream the tool is supposed to improve. Include straightforward, messy, and exceptional cases in proportions that resemble reality. Mark the source version and expected result for each example. Remove or protect personal data according to the team's rules; convenience is not permission to upload customer material. Do not let a vendor or enthusiastic team member choose only clean cases. A writing assistant tested on polished source notes may disappoint when the normal input is an incomplete voice memo. For each case, name the reviewer expertise required. A generalist can catch typos but may not verify a legal, medical, or technical claim. Keep the packet stable while comparing tools, prompts, or models, and record when a case is excluded. NIST's framework treats intended use and context as part of evaluation; the work packet is the local evidence for that context.
- Include normal and difficult inputs.
- Freeze expected outcomes and source versions.
- Use the same packet for every candidate.
Time the whole route
Start the clock when a worker begins the task, not after someone has prepared the perfect prompt. Capture the time spent finding inputs, cleaning data, entering instructions, waiting for output, reviewing it, checking sources, editing, seeking approval, and moving the result into the next system. Separate active human time from machine waiting time. Both can matter, but they affect planning differently. A long generation can happen while a worker does other tasks; a short generation that demands immediate expert attention can be expensive. Record interruptions and repeated attempts. If the team must rerun an answer because it is confidently wrong, include that work. A practical time sheet can use broad categories rather than second-by-second surveillance. The aim is to see where labor moved, not to monitor individuals. A tool may save drafting time while increasing verification time; the full route reveals that trade.
- Count preparation, review, correction, and handoff.
- Separate active human time from waiting.
- Include repeated attempts and interruptions.
Measure corrections by type and consequence
Do not reduce quality to a single pass rate. Mark corrections such as unsupported facts, missing caveats, wrong identity, unclear language, broken formatting, accessibility defects, and policy violations. Distinguish an edit made for style from an edit required for correctness. Note whether a reviewer caught the defect easily or had to search for the source. A tool that makes a small number of hard-to-detect errors may impose greater risk than one that makes obvious cosmetic mistakes. Keep a short example of each important failure so the team can improve a prompt, source, or review interface. Review quality must be judged against the acceptance rule, not whether the output feels fluent. When an error reaches a user or authoritative record, count the downstream recovery separately. It is often much more costly than fixing a draft. This classification gives a decision maker a reason for the metric rather than an attractive but uninformative percentage.
- Separate style edits from factual corrections.
- Record hard-to-find errors.
- Keep quality vetoes visible.
Count escalation and exception labor
Every AI workflow has cases that cannot pass the usual route. Track how many are escalated, why, who handles them, and how long resolution takes. A case that stays unresolved should not disappear from the denominator. If an AI tool handles eighty routine records quickly but creates twenty ambiguous exceptions that need a specialist, the average over successful records alone is misleading. Use a small table with case type, status, reviewer role, correction time, and final outcome. Look for clusters: one document type, language, input source, or policy branch may account for most of the labor. That finding can support a narrower deployment or a targeted input fix. Do not respond by weakening acceptance criteria merely to improve a dashboard. Exceptions are evidence about where the workflow's boundary lies. They also reveal whether the claimed automation depends on hidden work performed by people outside the trial team.
- Keep unresolved cases in the denominator.
- Name the person who resolves each class.
- Report exception rate by case type.
Compare with the real manual baseline
Observe a small sample of the same tasks completed through the current process. Include waiting, rework, and approval time there too; otherwise the comparison favors whichever route receives less scrutiny. Do not assume manual work is error-free. Record its mistakes and recovery costs using the same acceptance rule. Then compare accepted results per hour of active effort and the distribution of turnaround times. Use medians and ranges when a few difficult cases distort a mean. A trial of ten examples will not support a precise annual savings claim, but it can reveal whether review dominates the task. If the AI route is faster only for one narrow case, say so. That is still a useful decision: route those cases to the tool and leave the rest manual. The baseline should represent the team's current skill and constraints, not a fictional ideal or a deliberately inefficient process.
- Apply one acceptance rule to both routes.
- Compare distributions, not just averages.
- State which case types benefit.
Calculate cost per accepted result
Combine the relevant costs: tool fees, setup, integration maintenance, active worker time, specialist review, exceptions, and downstream correction. You may not need exact currency values in an early trial. A time-and-effort model can show whether a higher subscription price is trivial next to reviewer labor. Mark assumptions clearly: volume, rate, case mix, and whether staff time is available for other work. Calculate low, expected, and high scenarios rather than claiming a single guaranteed saving. If the tool changes the nature of the result, compare quality and user outcome alongside cost rather than pretending every output is equivalent. A cheaper but unreliable answer may be more expensive once correction and trust are included. Do not double-count work that would happen in both routes, such as final approval required by policy. This keeps the decision grounded in the actual difference the tool creates.
- Include human and machine costs.
- Show assumptions and scenario ranges.
- Keep non-negotiable quality gates outside the price score.
Illustrative prompt · adapt before use
Calculate effort per accepted result from this illustrative trial.
AI route: 10 attempted, 8 accepted, 2 unresolved; 20 minutes preparation, 40 review, 15 correction, 10 escalation.
Manual: 10 attempted, 10 accepted, 100 active minutes total.
Times include unresolved cases; do not discard their labor.
Show totals, active minutes per accepted case, unresolved rate, and limitations. Tool fees are unknown. Do not extrapolate annual savings or treat unfinished cases as successes.Design review so it can succeed
When review consumes most of the time, inspect the review interface before blaming the model. Can a worker see each claim beside its source? Is the original request visible? Are changes highlighted? Can the reviewer reject one item without rewriting the entire result? A tool that hides source references forces the person to recreate the work from scratch. Try a simple side-by-side packet with source, AI proposal, and acceptance checks. Then time a new sample. If effort falls while quality holds, the bottleneck was the handoff. If the assistant repeatedly invents facts, an interface change will not solve the model or source problem. Keep the two diagnoses separate. NIST's attention to human oversight and documentation is practical here: review must be supported by the evidence and controls a person actually needs, not merely inserted as a checkbox.
- Show evidence with output.
- Make rejection and correction easy.
- Retest the route after changing review design.
Write a decision that another person can use
End the trial with a short record: supported task types, acceptance rate, active time per accepted result, exception pattern, material errors, limitations, and who owns monitoring. Make an explicit choice to adopt, narrow, revise, or stop. A tool can be valuable even when it does not eliminate human review; perhaps it saves preparation time or improves consistency. Equally, a tool that produces impressive drafts may be a poor fit if experts spend longer checking them than they would writing from scratch. Avoid promotional language such as “ten times faster” unless the measurement actually covers the whole workflow and the cases are representative. Date the result because models, prices, and source data change. This lets a later team repeat the measurement without arguing about what the original number meant. The decision record is a working artifact, not a universal ranking of AI products.
- State adoption scope and owner.
- Keep observed evidence beside conclusions.
- Date assumptions that can change.
Recheck when the workflow changes
A tool update, prompt revision, source migration, new language, or changed approval policy can alter review cost without anyone noticing. Keep a small stable sample and rerun affected cases after meaningful changes. Sample ordinary live work as well, because a frozen packet can miss new patterns. Watch for rising correction time, more escalations, or reviewers losing trust and checking every sentence regardless of task risk. Those are signals that the economics changed. A team should not preserve a favorable old metric as a promise about the current route. Conversely, do not require a full procurement study for every minor edit. Recheck the cases whose evidence or handoff changed, compare them with the baseline, and update the decision scope. The measure is useful only while it describes the work people now do. For example, a team may discover that a new retrieval feature makes first drafts faster but sends more uncertain citations to a senior editor. The right response is to inspect that handoff and test a smaller set of source-heavy cases, not to declare the whole tool better or worse from one headline metric. Ask reviewers whether they can explain each correction and whether they still trust the acceptance rule. If the task has shifted toward more complex inputs, update the packet before comparing the new number with last quarter. Otherwise the team may mistake changing case mix for a model improvement. Keep the raw case outcomes available so another person can check the conclusion.
- Retest changed case types.
- Watch correction and escalation trends.
- Update the decision scope when evidence moves.
Sources and update note
Follow the linked official source before a product, price, plan, or policy decision.