End-to-End RAG Customer Support System with Automated Evaluation and Knowledge Ingestion

Customer support teams often struggle to provide consistent, accurate answers when information is scattered across documents and knowledge bases. Traditional support relies on manual searches, while AI chatbots can generate plausible but incorrect answers, invent unsupported details or retrieve irrelevant information. The core challenge is enabling AI to respond using reliable, relevant knowledge rather than relying solely on its pretrained knowledge. Without effective retrieval, responses may lack context or contradict documented policies. Without systematic evaluation, teams have limited visibility into answer correctness, relevance and faithfulness to source material. These limitations can undermine customer trust, increase the risk of misleading responses and create additional work for support teams. A reliable solution must connect knowledge ingestion, retrieval, grounded response generation and answer-quality evaluation into a cohesive system.

Built a production-oriented Retrieval-Augmented Generation (RAG) customer support system in n8n, comprising four modular, interconnected workflows: document ingestion, a reusable RAG Query Engine, a customer-facing Production Chatbot, and an automated RAG Evaluation pipeline. The ingestion workflow processes knowledge-base documents from Google Drive, generates embeddings using Gemini, and stores them in Pinecone. The reusable query engine retrieves relevant knowledge to support grounded answer generation, while the customer-facing chatbot provides an interface for interacting with the system. Implemented an automated evaluation pipeline to assess answer correctness, relevance, and faithfulness to source material. The evaluation process incorporates queue-based processing, controlled execution, fallback evaluation, and persistent results in Google Sheets, supporting structured assessment and review of answer quality. The solution demonstrates production-oriented AI automation through modular architecture, reusable components, knowledge grounding, error handling, and systematic evaluation. It brings together retrieval, response generation, and quality assessment in an integrated system rather than treating the chatbot as a standalone AI interface.

Productivity Improvements: Automates knowledge ingestion, information retrieval, answer generation, and answer-quality evaluation. Quality Improvements: Establishes a structured process for generating grounded responses and assessing their quality. Accuracy Improvements: Uses relevant retrieved knowledge and evaluates answer correctness to help identify inaccurate responses. Knowledge Management Improvements: Converts documents from Google Drive into an embedded, searchable knowledge base stored in Pinecone. Operational Efficiencies: Separates ingestion, retrieval, chatbot interaction, and evaluation into four reusable, interconnected workflows. Onboarding Improvements: Provides a structured knowledge-retrieval foundation that could help new support staff access documented information more efficiently. Customer Experience Improvements: Supports more relevant, knowledge-grounded responses through a dedicated customer-facing chatbot. Risk Reduction: Evaluates answer correctness, relevance, and faithfulness to help identify potentially misleading or unsupported responses. AI Performance Improvements: Introduces systematic evaluation, persistent results, controlled processing, and fallback evaluation to support assessment and iterative improvement of answer quality.