AI Workflows · Guide
What to Save When an AI Experiment Fails
A short, honest record of inputs, expectations, and observed output turns a failed pilot into a better next test.
AI experiments can fail in frustrating ways. A summary misses a condition, an image loses the intended subject, or a routing assistant sends the wrong request to the wrong queue. Teams often respond by tweaking a prompt immediately. That may help once, but it can also erase the evidence needed to understand why the result was wrong. A small failure record protects the learning. It does not need a new platform or a long postmortem. One page can preserve the task, the input, the expected behavior, the observed output, and what the team will test next. The record is most valuable when it is written before the next attempt changes the conditions.
Start with the job, not the model's mistake
State what a successful result would have helped someone do. “The assistant should identify the current owner of a request from the approved queue guide” is clearer than “the answer should be good.” Include the user or reader affected and the decision that depended on the output. This keeps the team focused on the work rather than on a single awkward sentence. Then record the test input. Use an approved, minimal example and remove unnecessary private information. If the failure occurred with real data, follow the team's data-handling rules before copying it into a note. Sometimes a description of the input pattern is enough.
Illustrative prompt · adapt before use
From this permitted failure example, separate observed output from possible causes. Return the job, expected result, actual result, conditions, and one bounded next test; do not claim a root cause yet.Capture the conditions that matter
Save the prompt or instructions, the relevant source set, the tool or model version when available, and the date. If a retrieval system was involved, note which documents were offered to the assistant. If a person edited the output, distinguish the raw result from the corrected one. Without these details, a later comparison may test a different workflow while pretending to repeat the same experiment. Avoid collecting every internal setting. Keep only the conditions that could plausibly explain the observed error or affect the next test. The record should help someone rerun the case, not become an archive of unrelated telemetry.
Describe what happened without a premature diagnosis
Write the expected result beside the actual result. Mark the point of divergence: missing source, wrong classification, invented detail, unclear handoff, weak visual composition, or another observable issue. Do not label “the model hallucinated” as a root cause when the source document was absent. Check the earlier steps first. Record the effect on the user or reviewer. Did the error change a decision, create extra review work, or merely make a draft less elegant? This helps choose whether the problem blocks the pilot or belongs in a later quality pass.
Choose one next hypothesis
The next test should change one meaningful condition when possible. Perhaps the source pack needs a clearer current-version marker, the prompt needs an explicit uncertainty path, or the task is too broad for the available data. Write the hypothesis before rerunning, then compare the new output with the old example and a few untouched cases. A fix that helps one example but harms the rest is not a reliable improvement. Share the record with the people who own the workflow. A failed experiment becomes useful when it changes the next decision: revise the source, narrow the task, add human review, or stop using AI for that step.
Keep an example the next person can actually use
An error label such as “hallucinated” is too broad to guide a repair. Save one permitted, representative input, the expected behavior, the actual output, and the specific difference. If the real input is sensitive, create a redacted or synthetic example that preserves the failure mechanism and record where the protected original can be found by authorized people. Include the source document or rule the system should have used. Another teammate should be able to run the example or inspect the evidence without guessing which version of the prompt or source was involved. Then choose a bounded next step. If the model cited a nonexistent page, test whether the retrieval source was missing or the answer instruction encouraged guessing. If the source was correct but a reviewer missed an altered number, revise the review check. If the task itself requires expertise the team does not have, stop the experiment rather than trying more prompt variations. The record is valuable because it separates observation from diagnosis and makes the next trial smaller and more informative. Try this next: take one recent failure and fill six lines—job, input, expected, observed, conditions, next hypothesis—before changing the prompt again.
Sources and update note
Follow the linked official source before a product, price, plan, or policy decision.