Enterprise RAG: 70% Support Ticket Reduction
A production RAG system over a 500-document internal knowledge base that reduced Tier 1 support tickets by 70%, using Claude 3.5 Sonnet, pgvector, and hybrid search — deployed in 6 weeks.
The Challenge
The Challenge: Support Team Drowning in Repetitive Tier 1 Questions
A Series B B2B SaaS company with 400+ enterprise customers was generating 2,800 support tickets per month. Analysis showed that 68% of those tickets — roughly 1,900 per month — were Tier 1 questions answerable from existing documentation: product guides, API references, integration tutorials, release notes, and FAQ articles. The problem was discoverability. The company had 500+ documentation articles spread across Notion, Confluence, a public docs site, and internal Google Docs — with no consistent taxonomy and poor search. Customers could not find answers themselves. Support agents were spending 60% of their time copy-pasting documentation links and writing variations of the same answers. The customer success team wanted a chatbot but had tried two off-the-shelf solutions that gave hallucinated answers using outdated content — destroying customer trust rather than building it. The requirement was clear: only answer from verified company documentation, cite sources, escalate to human when uncertain, and handle the full technical depth that enterprise API customers require.
The Solution
Solution: Agentic RAG with Source Citation and Human Escalation
I built a production RAG system that ingests and indexes the company's entire documentation corpus, uses hybrid search to retrieve precisely relevant content, and uses Claude 3.5 Sonnet to generate grounded, citation-backed answers — with a clear escalation path to human agents when the system is uncertain.
Documentation Ingestion & Indexing Pipeline
Built an ingestion pipeline that connects to Notion API, Confluence API, the public docs site (via sitemap crawler), and Google Drive to extract, clean, and chunk all documentation. Chunks are embedded using text-embedding-3-large and stored in pgvector with rich metadata: source system, document category, product area, last updated date, and confidence score for recency. The pipeline runs nightly to keep the index current with new documentation.
Hybrid Search & Reranking
Implemented hybrid search combining dense vector similarity (pgvector cosine) with BM25 keyword matching using reciprocal rank fusion. A Cohere reranker re-scores the top 20 candidates before passing the top 5 to Claude. This hybrid approach dramatically outperformed pure vector search for precise technical queries (specific error codes, API parameter names) while maintaining strong performance for semantic queries.
Claude 3.5 Sonnet Response Generation
Claude generates answers strictly grounded in retrieved chunks, with mandatory inline citations linking to source documents. The system prompt instructs Claude to explicitly state uncertainty when retrieved content is insufficient and to trigger the human escalation flow rather than speculate. Response confidence scoring routes low-confidence answers directly to the support queue with the full retrieval context pre-loaded for the agent.
Intercom Integration & Analytics
Deployed as an Intercom Fin integration, making the RAG chatbot available as the first response in every support conversation. Built analytics dashboards tracking resolution rate by question category, escalation rate, customer satisfaction scores, and documentation coverage gaps (questions the system could not answer) — feeding a continuous documentation improvement process.
Technology Stack
Tools & Technologies Used
“We had tried two chatbot vendors before this and both gave hallucinated answers that made our customers less confident in us. This system only answers from our own docs, cites sources, and gracefully escalates when it does not know — that is exactly the behaviour enterprise customers need.”
Related Services
Services Used in This Project
Have a similar project?
Let's discuss your requirements. Book a free 30-minute consultation.
Book Free Consultation