CAVAS AI Assistant — Human-in-the-Loop AI Workflow System

Manual opportunity intake and review becomes inconsistent and time-consuming when information arrives from multiple sources and must be evaluated against business rules before action is taken. The challenge was to create a system that could collect, structure, analyse and route incoming opportunities while preserving human control over important decisions and maintaining a clear audit trail.

The workflow had two different types of decisions.

Some decisions were deterministic: explicit exclusions, required fields, workflow states, confirmation rules, and other business constraints. These were better handled in application logic.

Other decisions required interpretation of unstructured text, such as understanding what an opportunity was really asking for, comparing it with the target profile, and distinguishing a strong match from a superficially similar one. That is where AI added value.

The final design therefore used a hybrid approach rather than asking an LLM to control the entire process. Deterministic logic handles hard constraints and state transitions; AI supports interpretation, analysis, and structured recommendations; the human operator retains final control over important actions.

The most important architectural decision was to separate AI reasoning from workflow authority.

AI can analyse, classify, summarise, and prepare recommendations, but application logic controls workflow states and execution. Important actions require explicit human confirmation.

The system uses persistent state so the workflow does not depend on conversational memory. This makes interrupted or multi-step processes easier to resume, inspect, and audit.

A deterministic quality filter runs before semantic-fit classification. Clear mismatches are rejected using explicit rules, while AI is reserved for cases that benefit from interpretation. If semantic classification fails, the workflow uses a safe fallback rather than silently rejecting the item or proceeding automatically.

Structured outputs were preferred over free-form LLM responses wherever downstream logic depends on the result.

A local-first testing strategy was used to control development risk and cost. Automated tests cover workflow logic, guards, state transitions, failure cases, and integration boundaries before changes are allowed into the main workflow.

One challenge was source variability. Different sources expose different fields, formats, quality levels, and failure modes. The workflow was designed around normalized intake and explicit provider boundaries so one weak source would not define the whole system.

A second challenge was relevance quality. Simple keyword filtering produced false positives, while unconstrained AI classification would have introduced unnecessary cost and unpredictability. The solution was a layered pipeline: deterministic quality filtering first, semantic-fit evaluation second, and existing ranking logic afterward.

Another challenge was preventing AI output from becoming workflow authority. Explicit states, confirmation boundaries, fail-safe fallbacks, and audit history were introduced so the system could assist decisions without silently executing them.

Operator usability also required iteration. Controlled trials exposed missing next-step guidance, weak rejected-item visibility, and drafts that could be too generic. These issues were addressed with clearer operator guidance, rejected-item previews, stronger fit guards, and tighter draft-readiness checks.

Finally, live sources are inherently unstable. The system was therefore designed to distinguish code reliability from live-source availability rather than treating a temporary lack of candidates as a software failure.

This was an owner-operated system rather than a client-wide organisational rollout, so adoption was handled through controlled trials instead of formal employee training.

The workflow was introduced incrementally. Individual stages were first validated locally, then exercised through guided end-to-end trials using the Telegram operator interface.

The operator remained in control at review and confirmation points. This made it possible to observe where guidance was unclear, where drafts were not ready for use, and where the system needed stronger safeguards before broader operation.

Feedback from those trials was converted directly into workflow improvements, including clearer next-step instructions, rejected-item visibility, stronger exclusion guards, and safer handoff behaviour.

The next opportunity is to generalise the same architecture beyond opportunity discovery into broader business workflows.

The core pattern — structured intake, deterministic rules, AI-assisted analysis, explicit workflow states, human approval, audit history, and controlled handoff — can be applied to lead qualification, service routing, internal review, operational triage, and other decision-support workflows.

Future versions can add more approved data sources and integrations, stronger evaluation datasets, richer operator dashboards, and reusable workflow modules for different business processes.

Another important direction is deeper measurement: once the system is used in a real client workflow, outcome tracking can be added for review time, throughput, error reduction, conversion, and other business-specific KPIs.

The strongest lesson was that reliability comes from workflow design, not from the language model alone.

AI works best when its role is clearly bounded. Hard business rules, state transitions, confirmation requirements, and execution logic should remain deterministic wherever possible. AI should be used where interpretation genuinely adds value.

A second lesson was to test the operator experience, not only the code. A technically correct workflow can still fail if the next action is unclear, rejected items disappear from view, or generated output looks complete when it is not ready for use.

The project also reinforced the value of incremental hardening. Controlled trials repeatedly exposed small operational weaknesses that unit tests alone would not reveal. Fixing those issues before expanding the scope produced a much stronger system than adding more features would have.