AI Workflows · Guide
Pilot AI Search in an Internal Knowledge Base
Test whether an answer can lead people back to the right approved source before offering AI search to a whole team.
An internal knowledge base often contains the answer to a question, yet finding it can still be hard. A new employee may use different words from the author of a policy page. A support agent may find three versions of a procedure and not know which one applies. AI search looks appealing because it can turn a question into a short answer. The first question for a pilot, however, is whether that answer helps people reach the right source and make a sound decision. A polished paragraph that hides a stale page is worse than a search result that openly asks for review. Start with one family of low-risk questions. A good candidate is a documented internal process with a named owner, a small set of current pages, and a clear way for a person to verify the answer. Do not begin with compensation, legal interpretations, access grants, customer account changes, or anything where a wrong answer could quietly cause a serious outcome. Keeping the scope small makes the test intelligible: the team can learn whether retrieval, answer drafting, or source maintenance is the actual bottleneck.
Define what an answer must do
Write the user job in ordinary language. For example: “A teammate needs to find the current steps for submitting a routine expense before the monthly cutoff.” The assistant's job is to identify the applicable source, summarize the steps without inventing exceptions, and show where the user can check them. It is not authorized to approve an expense or reinterpret policy. A useful pilot contract lists the question types it may answer, the documents it may draw from, and the point where it must say it cannot decide. Specify the expected output before selecting a search product. A strong answer names the relevant page, links to the exact section when possible, states the applicable date or version if available, and distinguishes a source statement from an inference. If two pages disagree, the assistant should show the conflict and direct the user to the owner rather than choose the newer-looking page by intuition. If it finds no support, a short “I could not locate an approved answer” is more useful than a plausible procedure.
Make the source set deliberate
An internal document folder is rarely one clean corpus. It may include drafts, meeting notes, archived pages, duplicates, and private records. For the pilot, create a source list with each item's owner, audience, last review date, and intended authority. A published procedure should outrank a meeting note that merely discussed changing it. A draft should not become authoritative because its title happens to match the query. If the organization has no owner or status marker for a high-use page, assign one before relying on an answer generated from it. Keep source access at the same level as the reader's permission. A retrieval system should not reveal a restricted document through a summary or citation to someone who cannot open that document. Test this as a direct behavior, not as a claim in a vendor page. Include a user who can access the page and one who cannot; submit the same question and compare the response. If the assistant cannot reliably preserve the access boundary, stop that source family from the pilot. A citation is not an acceptable permission check by itself.
Build a question set from real language
Ask the people who use the knowledge base what they actually type or ask a colleague. Collect a modest set of questions with permission, remove unnecessary personal information, and keep variations. Some people ask “How do I file travel?” while others ask about a receipt, deadline, or exception. Include easy direct matches, synonyms, ambiguous questions, questions with outdated terminology, and cases where the right response is “ask a person.” The sample should resemble the work rather than a marketing demo. For each question, have a knowledgeable owner mark the current source page, the key facts that must appear, prohibited assumptions, and whether a direct answer is appropriate. If two owners disagree, resolve the document or policy ambiguity first. A benchmark with uncertain labels cannot tell you whether the model was wrong. Keep a small holdout set outside the prompt-tuning loop so improvements on rehearsed examples do not masquerade as general reliability.
Score retrieval separately from the prose
The pilot needs to distinguish finding the right page from writing a fluent answer. First ask whether the correct source was retrieved within the results a user could reasonably inspect. Then ask whether the final answer faithfully represented it. A response may cite the right page but misstate a cutoff date. Another may answer correctly from memory but fail to cite a source. These failures need different repairs. Log the question, permitted source set, retrieved pages, answer, citation, reviewer judgment, and failure reason without retaining more sensitive text than needed. Useful categories are: correct and supported; source found but answer distorted; wrong source; stale source; access violation; unsupported answer; and appropriate abstention. Review high-impact failures individually. A simple average score can hide the one question type that matters most to users. Record the time it takes a person to verify the answer, too. If checking the citation takes longer than finding the original page, the interface is not yet improving the task.
Test abstention and disagreement on purpose
Write evaluation cases where the documents are silent, where two current pages conflict, and where the question is missing a key condition. The right behavior may be to show possible sources and ask for the missing detail. Do not pressure the assistant to answer every prompt. A system that confidently fills gaps will look productive until someone acts on a fabricated step. Track how often it declines appropriately and how often it declines a question that the source could have answered; both matter. One instructive test is a process whose rules changed recently. Give the pilot access to the current page and an archived page that uses the old terms. Ask in the old terminology. A trustworthy response should identify the current procedure, indicate that terminology changed if the source says so, and link to the current owner-approved page. If it blends the two procedures, the problem may be source metadata or ranking rather than the wording of the final prompt. Fix the authoritative source path before trying to hide the mixture with a longer instruction.
Put a person at the point of action
The first version should assist a human, not silently update a record. Show the source beside the proposed answer and make it easy to open. If a team member uses the answer to complete a task, the existing approval or authorization step still applies. The assistant should not acquire the ability to change permissions, submit forms, send messages, or edit policy documents as a side effect of search. A read-only pilot can still be valuable: it may reveal broken page titles, missing cross-links, and common questions no document answers well. Choose a small group of users and invite them to flag an answer as unsupported, outdated, unclear, or missing. Ask what they did next. Did they open the cited page? Did they ask a colleague? Did the answer save a search, or did it create another verification chore? These observations are richer than a count of questions answered. They also give the document owner an actionable maintenance list rather than a vague complaint that “AI search is inaccurate.”
Make the interface reveal uncertainty
A search result should visually separate the assistant's summary from the source excerpt or link. Show the document title, owner or status if available, and version information where it matters. Avoid a citation that merely points to the top of a hundred-page file. If exact anchors are unavailable, quote or identify the section title so the reader can find the evidence. Never show a confidence percentage without a tested meaning for that number. “Source not found” or “two pages disagree” is clearer than a spurious precision score. The handoff to a document owner should include the question, relevant page links, and the observed conflict. It should not attach the user's full private context by default. Feedback collection itself can become a new store of sensitive questions, so decide who can inspect it and how long the examples are needed for the pilot. When the source is updated, rerun the affected questions. A model change is not the only reason answers change; the corpus and access rules change too.
Compare with the current search experience
Run the same set of tasks through the existing knowledge search or navigation. Record whether users find the right page, how long it takes, and where they get stuck. Then compare the AI-assisted path on the same question types. Avoid declaring victory because an assistant generated a response faster than a user typed a query. The meaningful outcomes are supported answers, faster verification, fewer wrong turns, and a clearer route to the source owner. An ordinary search index with better titles may outperform AI on some categories. If AI search works well for short procedural questions but poorly for policy exceptions, keep those as separate lanes. The pilot may be useful without becoming the default for every internal question. If the biggest gain comes from cleaning up duplicate pages, do that source work and reassess. A search feature should not become a substitute for owning the knowledge base. It is a new way to encounter the quality of the underlying documents.
Use a concrete go or pause decision
Before the trial begins, name the conditions that would pause it: a restricted page leaking into an answer, a fabricated step presented as policy, repeated citation mismatch, or an owner-approved page being systematically outranked by a draft. Define who can remove a source, disable the pilot, and notify affected users. Do not keep a risky answer path live while you accumulate more examples simply to improve the score. Fix access and authority at their source, then rerun targeted tests. An expansion decision should be equally specific. The pilot's supported answer rate should be high on held-out questions for the chosen low-risk family; access tests and conflict cases should pass; users should be able to verify a source without unusual effort; and a person should own the corpus. The exact threshold depends on the task and its consequences. Publish the measured results and the unanswered question types to the team. That makes a narrow launch an honest improvement rather than a broad promise of “AI knowledge.”
Try one realistic task before buying scope
Imagine an employee asking whether a receipt can be submitted after a trip. The system finds a current expense page, an archived FAQ, and a team note proposing a new cutoff. The pilot should link to the current approved page, avoid treating the proposed change as policy, and ask for the trip date if the rule depends on it. If the user cannot access the current page, the assistant should not reveal the restricted answer. If the current page contradicts another approved page, the response should identify the conflict and direct the person to the policy owner. This example tests retrieval, document authority, access, and abstention in one small task. It also gives the team a concrete way to discuss failure. A summary that sounds reasonable but uses the proposal is not a partial success. A result that only says “ask finance” may be safe but not useful if the approved answer was available. The desired behavior is an answer a person can trace, challenge, and use within the ordinary process.
Keep a pilot record that supports revision
At the end of each review cycle, save a compact record: the source set and versions, question sample, scoring rubric, notable failures, changes made, and what was rerun. Do not retain full private documents in an informal spreadsheet simply because it is convenient. When a source owner changes a page, note the affected evaluation questions and retest them. When a tool or retrieval setting changes, compare on the held-out set before replacing the old configuration. The strongest reason to run an AI search pilot is not to announce that the knowledge base now answers everything. It is to learn whether a constrained question family can be answered with visible evidence and less effort for readers. If the answer is yes, expand one family at a time. If not, a clearer source inventory and a set of real unanswered questions are already useful outcomes.
Try this next
choose one low-risk question family and ask a document owner to label twenty representative questions with their current source page and expected answer boundary.
Sources and update note
Follow the linked official source before a product, price, plan, or policy decision.