AI3 min read
LLM support bots that don't embarrass you: a practical checklist
On this page
A support bot that invents a refund policy is worse than no bot. Six rules we follow: answer from your own docs, set hard limits, always offer a human, protect user data, test like code and speak the user’s language.
Bots and automation are one of the four things our studio does, and support is where large language models pay off fastest: most questions repeat, users want an answer at 3 a.m., and the team wants fewer copy-paste replies. But a support bot that confidently invents a refund policy does more damage than no bot at all. This is the checklist we follow.
1. Answer from your own knowledge, not the model's
A general model knows nothing about your subscription prices, server locations or last week's release. Give the bot a curated knowledge base — FAQ, store descriptions, release notes, privacy policies — and have it retrieve the relevant passages before answering (retrieval-augmented generation). Two rules make the difference:
- the bot answers only from retrieved material and says so when nothing relevant was found;
- every answer can point to its source, so the team can check where a claim came from.
2. Write down what the bot must never do
Make the boundaries explicit in the instructions and enforce them in code:
- no promises about refunds, discounts or compensation;
- no changes to accounts or subscriptions without a separate, permission-checked action;
- no legal, medical or financial advice;
- no guessing about features, release dates or prices that are not in the knowledge base.
Warning
Instructions alone are not a security boundary. Anything that changes data — cancelling a subscription, deleting an account — must go through an action with its own checks, never through free text.
3. Always offer a human
A good bot knows when to stop. Hand the conversation to a person when the user asks for it, repeats the same question, sounds frustrated, or the topic is billing or data deletion. Pass along a short summary so the user does not have to explain everything again. Measure how often this happens: a falling escalation rate is good only if satisfaction does not fall with it.
4. Treat user data carefully
- Tell people they are talking to a bot.
- Never ask for passwords, card numbers or one-time codes — and filter them out if users paste them anyway.
- Mask emails and phone numbers in logs and keep transcripts only as long as you need them.
- Make sure the model provider and data retention match what your privacy policy promises.
5. Test it like code
Collect a set of real questions with the answers you expect — including tricky ones and attempts to make the bot misbehave. Run the set before every change to the prompt, the knowledge base or the model, and review failures by hand. Small changes in wording can shift behaviour more than you expect.
6. Speak the user's language
Our audience writes in English, Serbian and Russian. Answer in the language of the question, keep product names untranslated, and make sure the knowledge base exists in every language you support — otherwise the bot will translate on the fly and drift from the original meaning.
| Risk | What to do |
|---|---|
| Invented facts | Retrieval from your own docs, cite sources, refuse when unsure |
| Unauthorized actions | Separate actions with permission checks, never free text |
| Frustrated users | Easy handoff to a human with a conversation summary |
| Personal data leaks | Redact logs, never request secrets, short retention |
| Silent regressions | A test set of real questions run before every change |
The short version
A support bot is a product, not a prompt. Ground it in your own documentation, give it narrow powers, keep a human one step away, and test every change. Done this way, it answers the routine questions instantly and leaves the interesting ones to your team.
Want a bot like this for your product or your Telegram community? Tell us about it.