AI Deploy Network
GovernmentAI Evaluation FrameworkAI Case Study

GrantScope: evaluated RAG search over 200 federal grant PDFs with 90% recall@5 and cited answers

AI Deploy Network Approved

A RAG search engine over 200 federal grant PDFs that puts the right grant in the top 5 results for every test question, answers only with word-for-word verified quotes and page citations, and runs live at $0/month.

Mounika BharathaBy Mounika Bharatha
Chapter 01

The Business Challenge

The business situation the organisation faced before the project began.

Federal grant documents are long, dense PDFs. Finding which grant fits a project means reading hundreds of pages, and generic LLM chatbots invent answers or cite the wrong document. The goal was a search tool that finds the right grant reliably, answers only from the source text with page citations, refuses when the answer isn't in the documents, and proves its quality with measurable evaluation.

Chapter 02

Why AI Was the Right Solution

Why AI was the right approach — and what alternatives were considered.

Keyword search alone misses grants that describe the same need in different words, and reading every PDF manually does not scale to hundreds of documents. A general-purpose chatbot can understand questions, but it answers from memory and invents details. Retrieval-augmented generation combines the strengths of both: semantic understanding of the question, grounded in the actual source documents, with citations that make every answer verifiable.

Chapter 03

How the Solution Was Delivered

Discovery, design, development, testing and rollout — the journey, not the tooling.

  1. 1Defined the problem: find the right grant in long, dense PDFs, and never invent an answer
  2. 2Built a 93-question evaluation set first, including deliberately unanswerable questions, to measure retrieval and honesty
  3. 3Ingested 200 grant PDFs (15K chunks) and compared fixed-size against section-aware chunking on recall@5
  4. 4Added hybrid semantic + keyword search merged with reciprocal rank fusion, then a cross-encoder reranker, measuring each step
  5. 5Constrained answers to word-for-word verified quotes with PDF page citations, with refusal when the documents lack the answer
  6. 6Wired the evaluation into CI so the index is rebuilt on every push and the build fails if recall drops
  7. 7Deployed publicly as a serverless Cloudflare Worker on Neon Postgres at $0/month
Chapter 04

Key Technical & Architecture Decisions

Architecture, model selection, workflow and trade-offs.

Evaluation first: a 93-question gold set, including unanswerable questions, was built before tuning, so every design choice was decided by measurement rather than intuition.

Chunking: section-aware chunks that respect the documents' structure beat fixed-size chunks on recall@5 (82% vs 63%), the single biggest retrieval gain.

Hybrid retrieval: pgvector semantic search combined with Postgres full-text search via reciprocal rank fusion catches both paraphrased and exact-term queries; a cross-encoder reranker then lifted recall@5 from 82% to 90%.

Grounding over fluency: answers are restricted to word-for-word verified quotes with page citations, trading some fluency for trustworthiness and an explicit refusal path.

Cost: an open model (Llama 3.1) on a serverless Cloudflare Worker with Neon Postgres keeps hosting at $0/month.

Chapter 05

Challenges & How They Were Solved

Obstacles hit along the way and how they were overcome.

Fixed-size chunking split sections mid-thought and buried the right grant (63% recall@5). Switching to section-aware chunking, with metadata filters, raised it to 82%.

Semantic search alone missed exact program names and terms, while keyword search missed paraphrases. Merging both with reciprocal rank fusion, then reranking with a cross-encoder, reached 90%.

The model could produce fluent answers that were not actually supported by the text. Requiring word-for-word quote verification with page citations, and allowing the system to refuse, brought faithfulness to 97% and correct refusals to 13 of 14.

Quality could silently regress as the index changed, so the evaluation now runs in CI and fails the build if recall drops.

Chapter 06

User Adoption & Change Management

How users responded — training, change management and feedback loops.

GrantScope is a publicly accessible live demo. Trust was designed in rather than trained in: every answer shows the exact quote and the PDF page it came from, so a reader can verify it immediately, and the system says so plainly when the documents do not contain an answer instead of guessing.

Business Impact

Business Outcomes & Impact

The measurable outcomes from the underlying AI Deployment — with the story behind them.

Hours Automated
0
Cost Savings
Currency not specified
Revenue Impact
Currency not specified
Measurable Business Outcome

Recall@5 improved from 63% to 90% (+27 points) on a 93-question eval set; 97% answer faithfulness; 13 of 14 unanswerable questions correctly refused; $0/month hosting cost.

Business Outcome Categories

Quality ImprovementKnowledge ManagementAI Performance Improvement

The story behind the numbers

The numbers translate directly into trust. With hybrid search and reranking, the correct grant appears in the top five results for every question in the evaluation set, so a person researching funding does not need to read hundreds of pages to find a fit.

Faithfulness of 97%, with every claim backed by a verified quote and a page citation, means answers can be checked in seconds rather than taken on faith. Refusing 13 of 14 unanswerable questions matters just as much: in grant work, a confident wrong answer about eligibility or deadlines is worse than no answer.

Because quality is enforced in CI, improvements cannot silently regress, and the $0/month serverless deployment shows that a reliable RAG system does not need expensive infrastructure.

Reflection

Lessons Learned

What surprised the team, what worked well, and what would be done differently.

  • Document structure matters: section-aware chunking beat fixed-size by 19 points of recall@5
  • Hybrid search plus a reranker beats either semantic or keyword search alone
  • Citations and refusals are what make a RAG system trustworthy
  • Evaluation in CI prevents silent regressions
  • A production-quality RAG system can run at $0/month
Looking Ahead

Future Opportunities

Where this solution could go next.

Add eligibility and deadline filters so results can be narrowed to grants a project actually qualifies for.
Extend to state and foundation grants and keep the corpus updated automatically as new notices are published.
An agentic workflow that drafts an application outline from the selected grant's requirements, with citations.
Alerts when a newly published grant matches a saved project profile.

For Peers

Professional Reflections

Guidance for another builder tackling a similar problem.

Build the evaluation set before you touch prompts or models, and include questions that should be refused. Without it, you cannot tell whether a change helped.

Retrieval quality usually matters more than the generator: chunking strategy and hybrid search moved results far more than prompt changes.

Make honesty measurable. Faithfulness and correct refusals deserve the same attention as recall.

Put quality gates in CI so improvements stay improvements.