AI Support Bot Hallucination Audit Builder
Audit AI support bot answers for invented policies, inaccurate troubleshooting, unsafe guidance, missing escalations, and knowledge base gaps.
Prompt Template
You are a support QA lead auditing an AI customer support bot. Build a hallucination and accuracy audit for the bot below. Company/product: [what the support bot helps with] Bot scope: [billing, troubleshooting, account management, policy answers, product setup] Knowledge sources: [help center, policy docs, API docs, internal macros, product catalog] Sample bot conversations: [paste transcripts or summaries] Known risk areas: [refunds, legal claims, medical/financial advice, security, account access, pricing] Escalation rules: [when the bot should hand off to human support] Customer segments: [free, paid, enterprise, regulated, minors, international] Success metrics: [deflection, CSAT, escalation rate, resolution rate] Compliance or brand constraints: [tone, disclosures, prohibited promises] Create: 1. Audit rubric for factuality, policy accuracy, escalation, tone, and safety 2. Hallucination taxonomy with examples relevant to this product 3. Conversation review table with severity and recommended fix 4. Knowledge base gap analysis 5. Guardrail instructions or system prompt improvements 6. Escalation rule changes for high-risk intents 7. Regression test set of questions the bot must answer correctly 8. Monitoring dashboard metrics and sampling cadence 9. Remediation roadmap for the next 30 days Be strict: customer trust matters more than inflated deflection.
Example Output
AI Support Bot Audit — Billing and Account Help
Hallucination Taxonomy
| Type | Example | Severity | Fix |
|---|---|---:|---|
| Invented policy | Bot promises refunds after 60 days when policy says 30 | High | policy retrieval + refusal template |
| Unsafe account guidance | Bot suggests sharing login with teammate | Critical | security escalation and blocked response |
| Overconfident troubleshooting | Bot says data is permanently deleted without checking logs | Medium | add uncertainty and handoff trigger |
Sample Finding
Transcript #14: customer asked whether annual plans can be paused. Bot answered yes and gave steps that do not exist. Correct answer: annual plans can be downgraded at renewal only.
Recommended fix: add a verified billing policy snippet, update retrieval tags for "pause plan", and add a test question: "Can I pause my annual subscription for two months?"
Monitoring Cadence
Review 50 conversations weekly, oversampling billing, account access, and cancellation intents. Any critical hallucination triggers immediate rollback or bot handoff-only mode for that intent.
Tips for Best Results
- 💡Paste real conversation snippets; generic chatbot audits miss product-specific policy mistakes.
- 💡Oversample cancellation, billing, and account access questions because small errors there create outsized trust damage.
- 💡Treat 'I do not know, I will escalate' as a success for high-risk intents, not a failure.
Related Prompts
Customer Support Quality Assurance Scorecard
Build a comprehensive QA scorecard to evaluate support agent performance, ensure consistent service quality, and identify coaching opportunities.
Customer Complaint Response Template
Write a professional, empathetic customer complaint response that resolves issues and preserves the relationship.
Help Documentation Writer
Write clear, user-friendly help articles and documentation that reduce support tickets and improve self-service.