The Business Challenge
The business situation the organisation faced before the project began.
Federal grant documents are long, dense PDFs. Finding which grant fits a project means reading hundreds of pages, and generic LLM chatbots invent answers or cite the wrong document. The goal was a search tool that finds the right grant reliably, answers only from the source text with page citations, refuses when the answer isn't in the documents, and proves its quality with measurable evaluation.
Why AI Was the Right Solution
Why AI was the right approach — and what alternatives were considered.
Keyword search alone misses grants that describe the same need in different words, and reading every PDF manually does not scale to hundreds of documents. A general-purpose chatbot can understand questions, but it answers from memory and invents details. Retrieval-augmented generation combines the strengths of both: semantic understanding of the question, grounded in the actual source documents, with citations that make every answer verifiable.
How the Solution Was Delivered
Discovery, design, development, testing and rollout — the journey, not the tooling.
- 1Defined the problem: find the right grant in long, dense PDFs, and never invent an answer
- 2Built a 93-question evaluation set first, including deliberately unanswerable questions, to measure retrieval and honesty
- 3Ingested 200 grant PDFs (15K chunks) and compared fixed-size against section-aware chunking on recall@5
- 4Added hybrid semantic + keyword search merged with reciprocal rank fusion, then a cross-encoder reranker, measuring each step
- 5Constrained answers to word-for-word verified quotes with PDF page citations, with refusal when the documents lack the answer
- 6Wired the evaluation into CI so the index is rebuilt on every push and the build fails if recall drops
- 7Deployed publicly as a serverless Cloudflare Worker on Neon Postgres at $0/month
Key Technical & Architecture Decisions
Architecture, model selection, workflow and trade-offs.
Evaluation first: a 93-question gold set, including unanswerable questions, was built before tuning, so every design choice was decided by measurement rather than intuition.
Chunking: section-aware chunks that respect the documents' structure beat fixed-size chunks on recall@5 (82% vs 63%), the single biggest retrieval gain.
Hybrid retrieval: pgvector semantic search combined with Postgres full-text search via reciprocal rank fusion catches both paraphrased and exact-term queries; a cross-encoder reranker then lifted recall@5 from 82% to 90%.
Grounding over fluency: answers are restricted to word-for-word verified quotes with page citations, trading some fluency for trustworthiness and an explicit refusal path.
Cost: an open model (Llama 3.1) on a serverless Cloudflare Worker with Neon Postgres keeps hosting at $0/month.
Challenges & How They Were Solved
Obstacles hit along the way and how they were overcome.
Fixed-size chunking split sections mid-thought and buried the right grant (63% recall@5). Switching to section-aware chunking, with metadata filters, raised it to 82%.
Semantic search alone missed exact program names and terms, while keyword search missed paraphrases. Merging both with reciprocal rank fusion, then reranking with a cross-encoder, reached 90%.
The model could produce fluent answers that were not actually supported by the text. Requiring word-for-word quote verification with page citations, and allowing the system to refuse, brought faithfulness to 97% and correct refusals to 13 of 14.
Quality could silently regress as the index changed, so the evaluation now runs in CI and fails the build if recall drops.
User Adoption & Change Management
How users responded — training, change management and feedback loops.
GrantScope is a publicly accessible live demo. Trust was designed in rather than trained in: every answer shows the exact quote and the PDF page it came from, so a reader can verify it immediately, and the system says so plainly when the documents do not contain an answer instead of guessing.
Business Outcomes & Impact
The measurable outcomes from the underlying AI Deployment — with the story behind them.
Recall@5 improved from 63% to 90% (+27 points) on a 93-question eval set; 97% answer faithfulness; 13 of 14 unanswerable questions correctly refused; $0/month hosting cost.
Business Outcome Categories
The story behind the numbers
The numbers translate directly into trust. With hybrid search and reranking, the correct grant appears in the top five results for every question in the evaluation set, so a person researching funding does not need to read hundreds of pages to find a fit.
Faithfulness of 97%, with every claim backed by a verified quote and a page citation, means answers can be checked in seconds rather than taken on faith. Refusing 13 of 14 unanswerable questions matters just as much: in grant work, a confident wrong answer about eligibility or deadlines is worse than no answer.
Because quality is enforced in CI, improvements cannot silently regress, and the $0/month serverless deployment shows that a reliable RAG system does not need expensive infrastructure.
Lessons Learned
What surprised the team, what worked well, and what would be done differently.
- Document structure matters: section-aware chunking beat fixed-size by 19 points of recall@5
- Hybrid search plus a reranker beats either semantic or keyword search alone
- Citations and refusals are what make a RAG system trustworthy
- Evaluation in CI prevents silent regressions
- A production-quality RAG system can run at $0/month
Future Opportunities
Where this solution could go next.
Add eligibility and deadline filters so results can be narrowed to grants a project actually qualifies for.
Extend to state and foundation grants and keep the corpus updated automatically as new notices are published.
An agentic workflow that drafts an application outline from the selected grant's requirements, with citations.
Alerts when a newly published grant matches a saved project profile.
Professional Reflections
Guidance for another builder tackling a similar problem.
Build the evaluation set before you touch prompts or models, and include questions that should be refused. Without it, you cannot tell whether a change helped.
Retrieval quality usually matters more than the generator: chunking strategy and hybrid search moved results far more than prompt changes.
Make honesty measurable. Faithfulness and correct refusals deserve the same attention as recall.
Put quality gates in CI so improvements stay improvements.