RAG engine built from scratch: custom HNSW, BM25, hybrid fusion and Self-RAG gates

RAG frameworks hide how retrieval actually works, which makes it hard to reason about quality, cost and failure modes. I wanted a transparent reference implementation where every retrieval step is readable, switchable and measurable per query.

Built a FastAPI service that calls the OpenAI API directly. Components written from scratch: a pure-Python HNSW graph index plus brute-force cosine search, an Okapi BM25 keyword index, and hybrid fusion by weighted ensemble or Reciprocal Rank Fusion. Optional stages can be toggled per query: multi-query expansion (3 variants), three Self-RAG gates (retrieval decision, relevance grading, hallucination check) and contextual compression that trims chunks to relevant sentences. A pipeline orchestrator records timing and metadata for every step and streams them to the UI as Server-Sent Events. Dual-layer memory: in-chat history and cross-session summaries. Personal open-source project, not a client engagement. The README lists known deficiencies: JSON persistence does not scale past tens of thousands of chunks, no authentication, and the HNSW is pedagogical.

Working application with a documented REST API, a single-page UI and a per-query pipeline trace showing the time taken by each step (the README example run completes in 6.2 s with Self-RAG, multi-query and compression enabled). Each retrieval strategy can be switched on or off and compared. The README states its limitations openly and no accuracy or adoption figures are claimed. Repository: github.com/muzammil5539/Agentic-AI-and-RAG-Projects (projects/beginner/rag-custom-engine).