GrantScope: evaluated RAG search over 200 federal grant PDFs with 90% recall@5 and cited answers

Federal grant documents are long, dense PDFs. Finding which grant fits a project means reading hundreds of pages, and generic LLM chatbots invent answers or cite the wrong document. The goal was a search tool that finds the right grant reliably, answers only from the source text with page citations, refuses when the answer isn't in the documents, and proves its quality with measurable evaluation.

Built and deployed an end-to-end RAG system over 200 federal grant PDFs (15K chunks). - Ingestion: section-aware chunking with metadata filters (beat fixed-size chunking on recall@5, 82% vs 63%). - Retrieval: hybrid pgvector semantic search + Postgres full-text search, merged with reciprocal rank fusion, then a cross-encoder reranker. - Generation: Llama 3.1 answers built only from word-for-word verified quotes, each with a PDF page citation; the system refuses when the documents don't contain the answer. - Evaluation: 93-question eval set measuring recall@5, MRR and faithfulness; CI rebuilds the index on every push and fails the build if recall drops. - Deployment: TypeScript Cloudflare Worker on Neon Postgres, publicly live at $0/month hosting cost.

Measured on a 93-question evaluation set: - Retrieval recall@5 raised from 63% (fixed-size chunks) to 82% (section-aware) to 90% (with cross-encoder reranker); hybrid search puts the right grant in the top 5 for every question. - 97% faithfulness: answers are grounded in word-for-word verified quotes with page citations. - 13 of 14 unanswerable questions correctly refused instead of hallucinated. - Runs in production at $0/month, with CI quality gates preventing regressions.