RAG Engine with Hybrid Retrieval, Reranking & Evaluation

LLM applications cannot reliably answer questions from large private knowledge bases using model knowledge alone. Naive vector search can also miss relevant information, return noisy context, and provide no systematic way to determine whether retrieval or answer quality improves after a change. The goal was to build an end-to-end RAG system that could ingest document collections, retrieve relevant evidence using multiple retrieval strategies, rerank results for relevance, generate grounded answers, and quantitatively evaluate retrieval and response quality. The system also needed production-oriented APIs, asynchronous ingestion support, containerized deployment, observability, and a design that could evolve beyond a prototype.

Designed and implemented an end-to-end RAG platform using Python and FastAPI, deployed on AWS and containerized with Docker. The retrieval pipeline combines FAISS dense vector search with BM25 lexical retrieval and rank fusion, followed by cross-encoder reranking. The query pipeline supports query rewriting and multi-query expansion to improve retrieval coverage. Document ingestion uses configurable chunking and metadata-aware processing. Integrated Amazon Bedrock for LLM inference and implemented streaming chat APIs, document ingestion, job tracking, health monitoring, and knowledge-base management. Built an evaluation layer covering retrieval metrics including Recall, Precision and MRR, alongside answer-quality evaluation such as faithfulness, completeness and conciseness. The system was also optimized for practical deployment, including reducing reranker startup overhead by approximately 90%. I owned the complete lifecycle: requirements, architecture, implementation, retrieval experimentation, evaluation, API development, containerization, AWS deployment, performance optimization and production debugging. GitHub: https://github.com/bikash-singhal/rag-engine Live demo link in GitHub README.

Demonstrated an end-to-end, deployable RAG architecture rather than a proof-of-concept retrieval pipeline. Hybrid retrieval, reranking and quantitative evaluation provide a repeatable framework for measuring retrieval quality and identifying regressions as the system changes. Evaluation runs demonstrated 100% recall and 60% precision on the evaluated retrieval sample, providing concrete visibility into retrieval effectiveness and remaining noise. The reranking implementation was optimized to reduce startup overhead by approximately 90%, materially improving application startup characteristics. The deployment also validated the complete production path from document ingestion and indexing through retrieval, reranking, LLM generation, evaluation, containerization and AWS operation.