AI Workflows · Guide
Audit an AI Summary for the Context It Leaves Out
Check exclusions, qualifiers, time frames, and minority cases before a concise AI summary becomes a decision shortcut.
A reliable summary audit compares the summary with the source set and asks what was omitted, not only whether quoted facts are correct. Check scope, dates, exceptions, disagreement, and uncertainty. Restore the missing context or narrow the claim before the summary is used to decide or publish.
Define the summary’s job
A summary for finding a document is different from one used to approve a policy, brief a client, or publish a headline. Write the intended reader and next decision before asking AI to compress material. Decide whether the output should list facts, compare positions, surface open questions, or recommend action; do not blend those jobs without saying so. Give the model a bounded source set and date range. If the reader needs a conclusion, specify who has authority to draw it. This upfront boundary makes omissions easier to judge. A missing side example may be harmless in a navigation blurb but serious in a risk assessment. NIST's AI RMF asks teams to map intended use and impacts; a summary audit applies that idea at the document level. Review should focus on how the compressed text will be used, not on whether it sounds elegant.
- Name reader and decision.
- Define source scope and date range.
- Keep synthesis separate from approval.
Inventory sources before compression
List each source with its title, owner, date, version, and reason for inclusion. If one report replaces an older version, say so. If a source is a press release, team memo, research paper, or firsthand record, preserve that status. A summary can make unlike sources look equally authoritative simply by placing them in one paragraph. Keep a source manifest even when the final prose is short. Ask whether a relevant source is missing because it was unavailable, excluded by design, or overlooked. Do not let an assistant search its way into an unstated evidence set and then cite only what supports an attractive conclusion. The inventory lets a reviewer test completeness and identify what the summary cannot legitimately answer. It also helps when a source changes later and the team must know which statements need rechecking.
- Record version and source type.
- Mark excluded material.
- Link each source to its role in the summary.
Make a claim-to-source table
Extract the summary's material claims into a small table. For each claim, cite the exact source passage or data point, record the scope of the original statement, and note whether the summary adds interpretation. This is more useful than asking the model whether it is confident. A sentence can accurately repeat a number while hiding that it describes only one region or quarter. A table exposes those qualifiers. For a long summary, sample every claim that would change a decision and a few apparently routine ones. If the team cannot locate evidence for a consequential claim, remove it or flag it as unverified. Do not fill the gap with a plausible citation. The goal is an auditable trail that a second person can follow without reconstructing the entire prompt conversation.
- Check the exact passage.
- Separate source fact from editorial inference.
- Remove unsupported consequential claims.
Illustrative prompt · adapt before use
Audit this illustrative summary against the supplied source only.
Source S1: In a four-week pilot with 12 staff in one office, 9 reported faster drafting. Accuracy and total review time were not measured.
Summary: The tool makes all teams more productive without reducing accuracy.
Return: claim | supporting passage | missing qualifier | support status | narrower wording.
Use no outside facts. Preserve sample, timeframe, self-report limitation, and unmeasured outcomes. Say unsupported where appropriate.Look for missing denominators and time frames
Numbers lose meaning when a summary drops the group, period, baseline, or measurement method. A change from ten to fifteen may be a fifty percent rise, but readers need to know whether it covers a week, a year, or a tiny sample. Check every percentage, comparison, and trend for its denominator. If the source reports uncertainty or a range, keep it. Distinguish an observation from a forecast and a planned action from an accomplished one. AI summarizers are good at smoothing tables into sentences; that can also erase caveats that made the original analysis responsible. In a review, ask the simple question: what would a careful reader of the source know that a reader of this summary would not? Add the missing detail if it changes the interpretation. Otherwise, use a short link to the source rather than loading every sentence with numbers.
- Preserve group and period.
- Keep ranges and methods when material.
- Do not turn plans into outcomes.
Preserve disagreement and exceptions
A concise narrative often favors the most common view and pushes exceptions into silence. Review any material disagreement between sources, teams, or participants. An outlier may be noise, but it may also reveal a failure mode or affected group that the dominant pattern misses. Label which perspective each source represents and whether the evidence was collected in comparable ways. Avoid writing that all stakeholders agree because a majority of documents favor one option. Similarly, distinguish a policy exception from an implementation error. If the decision depends on the exception, give it its own sentence or a visible note. The summary should help a reader understand the shape of the evidence, including its limits. It should not force disparate observations into one confident-sounding conclusion.
- Surface dissent that changes action.
- Identify incomparable sources.
- Name exceptions without exaggerating them.
Check causal language
Summaries often convert sequence into cause: a metric improved after a tool arrived, so the tool improved it. The source may only report a correlation or an anecdote. Underline verbs such as caused, prevented, proved, and guaranteed, then trace them to the original evidence. If the source does not support that strength, use narrower wording such as coincided with or was associated with, and state what else could explain the result. Do not overcorrect every ordinary sentence into unreadable hedging. Focus on causal claims that affect spending, safety, staffing, or public trust. When the evidence does support a causal conclusion, retain the study design or explanation that justifies it. This keeps the summary useful without giving the model authority to strengthen the source.
- Flag strong causal verbs.
- Keep alternative explanations visible.
- Match wording to the evidence design.
Write a compact correction note
An audit should produce an improved summary, not an endless critique. Keep a short log of material changes: restored a date, separated proposal from decision, added an exception, removed an unsupported claim. Let a reviewer accept or reject each change against the source. If the audience needs only a short brief, place detailed evidence in a linked appendix or table. Avoid appending generic warnings that do not identify what is uncertain. An actionable note says exactly which source and claim need attention. Once corrections are made, ask a second reader to explain the decision they would take from the revised summary. If that differs from what the source set supports, revise the summary's emphasis or scope. A concise output is valuable when it compresses responsibly, not when it hides the cost of understanding.
- Record material corrections.
- Keep links to evidence.
- Test the reader’s interpretation.
Reaudit when sources move
A summary can age quickly when a source report is corrected, a policy changes, or new data covers a later period. Store the source manifest and revision date with the output. When a source changes, identify the claims tied to it and update only those portions, then check whether the overall conclusion still holds. Do not quietly overwrite a widely shared decision brief without noting that its basis changed. If the old summary led to action, tell the decision owner what changed. The review routine can remain light: inspect the affected passages and a few neighboring claims rather than rerunning an entire research program. A source-linked audit lets the team move quickly while preserving the distinction between what was known then and what is known now.
- Keep source and summary dates.
- Update affected claims first.
- Notify owners when changed context alters a decision.
Sources and update note
Follow the linked official source before a product, price, plan, or policy decision.