Chatbot Knowledge Base: Build, Test, and Improve It

A chatbot knowledge base is a collection of approved business content that a chatbot searches to answer customer questions. It can include help articles, shipping and return policies, product guides, and vetted support answers. The chatbot retrieves relevant passages and uses them to draft a reply. Its answers depend on what those sources say and whether they are still accurate. It might include help articles, shipping and return policies, product guides, and support answers. The chatbot retrieves relevant passages, then uses them to draft a reply. Its answers depend on what those sources say and whether they are still accurate.

What is a chatbot knowledge base—and what is it not?

A chatbot knowledge base is the set of business information a bot can retrieve when answering customer questions. It is not simply an FAQ page, and it is not a guarantee that every answer will be right. The quality of the response depends on the content available and whether it supports the answer.

An FAQ page is one possible source, but a support chatbot can use a broader collection: policies, product documentation, troubleshooting instructions, and approved answers from the support team. A static FAQ presents information for a customer to browse. A knowledge-base chatbot searches across relevant content and turns what it finds into a conversational reply.

That difference matters when a customer asks a question in an unexpected way. An article may say, “Standard delivery usually takes four to six business days.” A shopper might ask, “Will my parcel get here by Friday?” The bot has to find the relevant delivery information and respond within what that information actually establishes. The source does not necessarily say whether a particular parcel will arrive by a specific day.

A grounded support bot uses business sources to inform its response. A general-purpose chatbot can also draw on its broad training, which may produce plausible information that does not match your current policies. For customer support, a polished answer is not enough: the reply needs to reflect what your business currently offers.

Think of the knowledge base as part of the support process, not a file you upload once and forget. If a return window changes, the source content needs to change. If old and new policies remain available together, retrieval may find the wrong one. A knowledge-base chatbot can only work with the information it can access.

What to put in a chatbot knowledge base

Start with approved information that customers need to make decisions or solve common problems. Useful sources include support pages, shipping and return policies, product guides, troubleshooting steps, and answers your team has already vetted. Organize the material by topic and keep the wording clear enough to retrieve and check.

Begin with questions your team repeatedly answers. For an online store, that might include delivery estimates, return eligibility, how to use a product, or how to contact support. For each topic, identify the policy or instruction that is authoritative. Use published help content where it is accurate, and bring in internal material only if it is approved for customer-facing answers.

Choose formats your platform accepts. These can include website pages, PDFs, Word files, plain text, and question-and-answer pairs. For example, a Q&A pair can store a common customer phrasing alongside the approved answer. Before relying on a document, check that it contains searchable text. A scanned page or complicated table may not be interpreted as intended.

Organize sources into useful topics and subtopics, such as shipping, returns, product use, and payments. Consistent headings and terms make the collection easier to maintain. Use customer language as well as internal terminology where it helps connect a question to the right topic. A customer may say “send it back,” while the policy calls the process a “return.”

Write short, specific passages that answer one question or explain one procedure at a time. Include useful variations in wording, but do not repeat the same policy in several places with different phrasing. Repetition can make maintenance harder and creates opportunities for conflicting answers. Include conditions and exceptions directly beside the rule they qualify.

For every source, record enough context for your team to recognize what it covers and who maintains it. A practical inventory might include the topic, the authoritative source, the person responsible, and whether the content is ready for customer use. This does not require elaborate knowledge management software. A simple, maintained list is better than an untracked folder of drafts.

Separate customer-facing facts from internal notes. A private note about how agents handle an exception should not become a customer promise unless the business has approved it. If the content includes changing details, such as current prices or stock, decide how those facts will stay current before making them available to the bot.

How a chatbot searches its knowledge base to answer questions

A retrieval-based chatbot searches business content for passages relevant to a question, supplies those passages to a language model, and asks it to draft a response from that context. Citations point back to supporting content so a person can inspect the basis for the reply. They help with checking, but a citation alone does not prove that the answer is correct.

Many systems represent content and questions in a form that helps find related meaning, rather than relying only on exact word matches. This is often explained through embeddings: a way of representing text so that related words and phrases can be found. A customer can ask about “sending something back,” while a policy uses the word “returns.”

The basic workflow has a few steps. First, the platform takes in business content and makes it searchable. Then a customer asks a question. The system retrieves passages that appear relevant, provides those passages as context to the model, and the model drafts a response. If the system includes citations, the answer can point to the passage or document that supports it.

For a support lead, the important question is not whether the platform uses a particular technical method. It is whether the right passage was retrieved, whether the reply stays within that passage, and whether the source is current. A bot can retrieve a relevant paragraph but still misread a condition, omit an exception, or combine it with outdated information.

Citations can help your team understand which source informed an answer, but they do not guarantee that every claim is correct. A reply can still be wrong if the source is stale, the passage is misread, or the citation does not support the full response. A citation to a shipping page, for example, does not support a delivery guarantee unless the page actually makes that promise.

Set an answer boundary as part of the workflow: when the retrieved content does not support a response, the chatbot should not fill the gap with a guess. This makes the boundary visible to the customer and actionable for the support team.

Worked example: answer a shipping question from approved sources

To test a shipping answer, start with the current, approved shipping-policy page and compare the bot’s reply with the exact policy wording. Then ask the same question in different ways and inspect the citations. Also test a question that requires account-specific information, such as an order’s current status, to check that the bot does not invent details.

Suppose the policy says: “Standard shipping usually takes four to six business days after dispatch. Delivery times may be longer during busy periods.” Treat that as the approved source for the test. The question “How long does standard shipping take?” should get an answer that preserves the range, starts the estimate after dispatch, and mentions the stated busy-period exception.

Now try natural variations: “When will my standard delivery arrive?” and “How long after it ships should I expect my order?” These phrasings test whether the system finds the same policy without relying on identical wording. They should not change what the policy permits the bot to say. If a reply says the parcel is guaranteed by a particular date, check whether the source actually supports that promise.

Inspect the citation to confirm it supports the answer. If the citation points to an old delivery page, or to a passage that describes a different shipping option, the answer needs attention even if it sounds reasonable.

Then ask, “Where is my order right now?” The policy alone cannot answer that account-specific question. The chatbot should not invent a tracking update or imply it has looked up an order. A suitable next step is to tell the customer it cannot provide that information from the available content and hand the conversation to the support team, collecting the details the business needs.

This single test checks several things: whether retrieval finds the relevant source, whether the reply preserves conditions, whether citations support the claims, and whether an unsupported question reaches a human. Repeat the test after updating the policy or changing the chatbot’s configuration. A good result on one phrasing does not establish that every related question will work.

What to do when the knowledge base cannot support an answer

When the content does not support an answer, the chatbot should say so plainly rather than make a plausible guess. Give the customer a useful next step, such as handing the conversation to a person. For sensitive, account-specific, or policy-dependent questions, a clear escalation path matters as much as the wording of the automated reply.

A refusal should be understandable, not cryptic. “I’m not sure from the information I have” tells the customer why the bot is stopping. Follow it with the next action the support process actually offers. If the team needs an order number or email address to investigate, collect those details before the conversation reaches a person.

Decide what the handoff needs to contain. Depending on the issue, that might be the customer’s question, the relevant product or order context they provide, and any details needed for follow-up. Ask only for information the team needs to continue the conversation. A human should be able to see what the customer already said and what information has already been collected.

Not every unanswered question is a content gap. Some questions need a person because they depend on an individual account, a judgment call, or an exception to a general rule. Others reveal a missing help article or an unclear policy. Label or otherwise review these cases so the team can distinguish “the bot should not answer this” from “we should add a reliable source.”

For a content gap, assign someone to confirm the correct answer and update the appropriate source. Then test the question again. A human answer can resolve the customer’s immediate issue, but unless the underlying approved content is improved, the same question may remain unanswered for the next customer.

Common knowledge-base mistakes that create wrong answers

Stale, conflicting, poorly extracted, or unapproved content can lead a chatbot toward the wrong response. Remove superseded policies and drafts, decide which source is authoritative for each topic, and check whether uploaded documents can be read properly. Assign responsibility for changing information so corrections do not depend on someone noticing a bad reply first.

A common problem is leaving old policies accessible after publishing replacements. A longer, older article can still look relevant to a search system. If a return period changed, remove or clearly retire the old answer from the chatbot’s available sources rather than assuming that the new one will always win.

Conflicts also occur when different sources describe the same rule differently. A product page might promise one delivery window while a shipping article gives another. Before launch, compare sources on topics likely to affect customer decisions, agree on the authoritative version, and bring other pages into line. A chatbot cannot make an unclear business policy consistent.

Check document extraction as well as the original file. Scanned PDFs may not provide searchable text. Tables can lose their relationships between headings and values. A document that looks clear to a person may become incomplete or confusing when processed. Review representative passages and test questions whose answers depend on a table or exception.

Treat content that changes frequently with extra care. Product availability and prices, for example, can become stale quickly. Decide how the source will be kept current and whether the chatbot should answer those questions at all. If updates do not reach the searchable content promptly, a once-correct response can become misleading.

Finally, do not upload every internal document just because it is available. Drafts, agent-only guidance, and old campaign pages can create avoidable risk. Make a deliberate choice about what the bot should use, check that each source is suitable for customer answers, and review the collection when policies or products change.

Measure quality, coverage, citations, and handoffs before launch

Build a test set from real support conversations, including common questions, awkward phrasings, questions the bot should refuse, and cases that need a person. For each test, record the expected answer and the source that supports it. Review response accuracy, source coverage, citation quality, and handoff usefulness, then rerun the tests after content or configuration changes.

A useful starting point is a set of thirty to fifty questions from your own inbox or helpdesk. Include recurring shipping and returns questions, product-use questions, complaints, and questions with incomplete details. Add cases where the correct behavior is not to answer, such as an order-status question without access to order information.

For each question, write down what a correct outcome looks like. That might be an answer that includes a policy condition, a request for missing context, or an escalation. This makes it easier for a reviewer to distinguish a genuinely correct response from a fluent answer that merely sounds plausible.

Check answer quality in separate ways:

• Factual accuracy: Does the reply match what the business currently does?
• Coverage: Did the bot find a source that answers the customer’s question?
• Source support: Do the sources available to the chatbot support the important claims in the reply?
• Completeness: Does the customer get the relevant steps, conditions, or next action?
• Handoff quality: When the bot cannot answer, does the conversation reach a person with useful context?

Keep these checks concrete. For a shipping question, compare the answer with the policy’s delivery range and conditions. For a return question, check whether eligibility requirements and exceptions are represented. For a handoff, review whether the team received the customer’s question and the information the business chose to collect.

Track unanswered questions and review samples of handed-off conversations. A high number of handoffs can mean the content is incomplete, the bot’s answer boundary is too cautious for the use case, or customers are asking questions that properly need human judgment. A low number is not automatically a good sign: the bot may be answering when it should stop. Read the conversations to understand what is happening.

When a human resolves a question that the chatbot could answer in future, update approved knowledge rather than relying on the agent’s reply alone. Check that the new content is correct, belongs in the customer-facing source collection, and does not conflict with existing material. Then rerun the relevant test. This turns support conversations into a maintenance loop instead of a growing pile of one-off fixes.

momo is one option for teams that want a support chatbot grounded in their own content. It can use a website crawl, PDFs, Word files, plain text, and Q&A pairs. It retrieves passages, drafts a response, and checks that draft against those sources; when confident, it sends the answer with citations, and when it is not, it tells the visitor and opens a ticket for the team. Human answers can be saved as approved knowledge. The helpdesk inbox is included on every plan, including Free.

If you are evaluating a tool, test it with the support content and questions your team actually handles. For a broader look at knowledge base software options, compare how each tool handles your sources and support workflow. You can try momo free and use the same checks: source quality, supported answers, citations, and what happens when the chatbot cannot answer.

Frequently asked questions

Can a chatbot knowledge base use PDFs and Word documents?

Yes, if the chatbot platform accepts those formats and can extract their contents. Check that PDFs contain searchable text and that tables or scanned pages have not lost important details during processing. After adding a document, ask questions whose answers depend on it and inspect the retrieved passage before relying on the chatbot’s replies.

Does a chatbot knowledge base need embeddings or RAG?

No particular technical approach is a requirement for every team. Embeddings can help a system find related meaning when a customer uses different wording from the source. Retrieval-augmented generation, or RAG, is a common way to supply retrieved business content to a language model as context. When choosing or testing a system, focus on whether it finds the right source and answers within what that source supports.

What should a support chatbot say when it cannot find an answer?

It should say that it is not sure or cannot answer from the information available, then offer a useful next step. If the question needs investigation, the chatbot should hand the conversation to a person and collect the details your team needs. It should not invent a policy, order update, or customer-specific fact to avoid an unanswered question.

How many questions should I use to test a customer-support chatbot?

A practical starting test set is thirty to fifty questions drawn from real support conversations. Include routine questions, wording variations, awkward cases, and examples where the chatbot should refuse or hand off. Keep the expected outcome beside each question and rerun the set after important content or configuration changes.

How often should a chatbot knowledge base be updated?

Update a source whenever the underlying policy, product guidance, or other customer-facing information changes. For information that changes frequently, plan how those updates will reach the chatbot’s searchable content and test that the new answer appears. Also review unanswered conversations and correct any stale or conflicting material that they reveal.