Zap: LangGraph multi-agent email and calendar assistant for EmailZap

EmailZap needed an assistant that could answer questions about a user's email and act on it (drafting replies, organising mail, scheduling) across Gmail and Google Calendar, at consumer scale and consumer margins. The constraints were real: 150K+ email updates a day, sensitive personal data that could not be sent to third-party LLMs as-is, provider models that change behaviour without notice, and an LLM bill that had to stay well under a dollar per active user per month.

Email is unstructured, personal and endlessly varied. The questions users ask ("find my boarding pass", "what did Acme say about the invoice", "set up the call this email asks for") need language understanding, retrieval across threads, and judgement about what counts as needing attention. A rules engine breaks on the long tail within days, and off-the-shelf filters sort mail but cannot answer questions or draft a reply in the user's voice.

Headcount is not an option for a consumer product. Nobody pays for a human to read their inbox inside a consumer subscription.

We were deliberate about where AI does not belong. About 70% of mail is filtered out with deterministic rules before any model sees it, because a newsletter does not need an LLM to be recognised as a newsletter. In the rebuild the split is explicit: the model proposes, the harness decides. Anything the system can know for itself (IDs, whether the user is a recipient, whether a later reply exists, whether a quote really appears in the source) is computed in code, and the model is used only for judgement: what matters, what to say, and which tool to call next. Safety rules like "never send without an explicit instruction" live in code, not in prompts.

Single agent before multi-agent. v1 used a LangGraph multi-agent design. For v2 the lesson was to start with one agent and good tools, and add agents only when measured need demands it, since every extra agent adds cost and new failure modes.

Design from hardest to change to easiest. Output and evidence contract first (UI, evals and feedback all depend on it), then tool contracts, context layer, harness loop, and the framework last. A framework is just an engine for the loop; you cannot compare frameworks fairly until you have tools to plug in and tasks to grade them on.

The model proposes, the harness decides. The model is treated like an untrusted browser client. It works with short handles issued by the harness instead of real IDs, so it cannot invent or cross-reference records. It quotes, and the harness locates the quote in the source and records verified evidence. Answer sentences carry claim markers tied to verified claims. Result fields are split into model-authored proposals and harness-authored facts, so the model cannot say "sent" or "I checked your replies" when that did not happen.

No write tools for the model. Send, archive and calendar changes only come from a validated result shown on a confirm card, and the harness executes after the user taps. An injected email cannot talk its way into a send.

Core plus plugins. Gmail and Calendar are integrations that bring their own tools under one contract, so adding a CRM is a plugin, not a core change.

No model calls inside tools. Search returns provider metadata, not generated summaries, which keeps latency, cost and tracing honest.

Ledger vs trace. Full run traces go to Langfuse; a thin database ledger keeps what must never be lost: approvals, what the card showed, send receipts, memory changes.

Greenfield architecture, reused infrastructure. Existing drafts, search and send are reused behind adapters, after checking in code what they actually do.

Build vs buy on file safety: a managed malware scanner, and never a public scanning service that shares uploaded files.

Provider models change without notice. We saw a 40% drop in response length and 28% higher latency from unannounced updates. Fix: a golden eval set and automated weekly drift detection, so changes are caught in days with evidence.

The same model disagreed with itself on routing. Three-run majority voting resolved 92.5% of routing flips in evaluation.

Inference cost scaled with inbox size. Fixed with provider changes, prompt caching, batching, and a rules-based pre-filter that keeps about 70% of mail away from the LLM, for an 86% cost cut at the same accuracy.

Sensitive data. Personal email cannot go to third-party models as-is. PII/SPII is pseudonymised before every call, which also carried us through CASA Tier 2.

Prompt injection and untrusted content. Emails can contain instructions aimed at the agent, links, and attachments of any type. Rules now live in code: never fetch a URL from an email, scan and parse files only in a sandboxed pipeline, mark sender-controlled text as untrusted, and give the model no write tools.

Real mail is messier than invented examples. Requests addressed to someone other than the inbox owner, conditional actions ("if you didn't grant this access..."), deadlines that have passed, and a matching phrase from last year's flight. Designing from 30 real production emails turned each into a rule and a test fixture.

A spec that looks finished is not. After two review rounds, making the contracts executable still found 70 gaps. One let an approval card appear before the result was validated, so a draft with an invented address could reach the user. Another, found only by mutation testing, would have misreported a sent email on a double tap. All were fixed in code and in the spec before any model was involved.

Reuse claims were wrong in the details. "Reuse search" hid an AI query-builder call; "reuse drafts" came with constraints (one live draft per thread, replies only). Checking the code before recording the decision kept these out of production.

For an agent that touches someone's inbox, adoption is trust. v1 earned it through usage: weekly unique users rose about 3.7x, and weekly sends of memory-enabled drafts rose about 4.4x. People send drafts they trust, and drafts grounded in their own history got sent.

v1 also showed where trust is fragile: an agent that sounds confident but cannot show its evidence, or that might act on the wrong email. The rebuild designs trust in from the start:

Prepare by default, act only when asked. Reading proceeds, drafts and actions are shown, and anything that commits (send, archive, calendar change) needs a tap on a card bound to exactly what was shown. No auto-send; the hard target is zero sends without an explicit instruction.

Answers carry evidence. Claims link to verified quotes from the user's own mail, so users can check an answer instead of taking it on faith.

Memory is visible. Low-risk preferences save automatically but are shown; facts about people, sensitive topics, or anything that changes Zap's behaviour need confirmation. Old facts expire, so "my flight is tomorrow" never becomes a permanent belief.

Ask, don't guess. Mail that looks meant for someone else surfaces as "Is this yours?", and the answer shapes how similar mail is treated.

Feedback is several separate signals rather than one "Zap score": explicit feedback, opens, draft edits and sends, provider receipts, and human-reviewed eval cases, each with a known limitation.

Inside the team, the change was in how we build. Every open question became a numbered decision with an example and an owner, and the rebuild runs on written briefs and approval gates, so engineers and coding agents work from the same record instead of chat history.

The rebuild is mid-flight. Phase A (executable contracts, no model) is done. Next:

Phase B: a human-reviewed eval set covering search, answers, drafts and calendar cases, loaded into Langfuse datasets with score configs.

Phase C: bake-offs scored against the same contracts and evals. Runtime first (own loop vs Pydantic AI vs LangGraph vs an agent SDK), then model, then the method for judging importance. The contracts make this a fair comparison instead of a vibe check.

Phase D: one real slice end to end, with real Gmail and Calendar adapters, the chosen runtime, prompts tuned against evals, and a single surface.

After that: a CRM integration as the first plugin beyond Google; typed or voice approval (never the model interpreting words as approval); more proactive jobs from the same core; and applying the same contract-first method to the next agent the company builds.

Define the agent before you build it. Write down the goal, what it may read, prepare and change, how you will know it worked, and how fast it must be. Most agent bugs are unclear authority, not bad prompts.

Look at real data first. Production traces are the fastest source of hard, realistic examples. But production labels are not ground truth: our labels agreed with the old classifier on 26 of 27 cases, and that is agreement, not accuracy.

Pick the framework last. Output contracts, tools and context are expensive to change. The framework is cheap to swap if the layers under it are clean. Teams that start with the framework end up designing around it.

Put safety in code. A prompt that says "never auto-send" can be talked around by one malicious email. The harness decides; the model proposes.

Make the spec executable before making it smart. Scripted golden runs in the model's slot, fake providers with hard-case fixtures, and mutation testing found far more than two rounds of human review. Every gap found this way costs minutes. Found after a model is plugged in, it looks like a flaky model and costs days.

Treat confusion as a design signal. When a clear explanation was hard to give, the concept was usually wrong: a vague name, two ideas in one field, a missing boundary.

Measure cost and drift from day one. Consumer AI economics are unforgiving, and providers change models under you.

Parallel coding agents work well once one lead has built the core and the slices touch disjoint files. Give each shared file one owner and keep decisions in one written table.