AI Deploy Network
Software & SaaSConversational AIAI Case Study

Generic AI Assistant Framework – Secure Enterprise AI Assistant

AI Deploy Network Approved

Developed a configurable enterprise AI assistant framework that enables users to query organizational knowledge through natural language. The solution combines document ingestion, vector-based retrieval, authentication and access-controlled responses to provide a reusable foundation for enterprise AI applications.

MayankBy Mayank
Chapter 01

The Business Challenge

The business situation the organisation faced before the project began.

Organizations often store critical business information across documents and internal knowledge sources, making it difficult for employees to find accurate and relevant information efficiently. Traditional chatbots lack the ability to provide context-aware responses while enforcing granular access permissions. Additionally, integrating AI assistants with external business tools and maintaining consistent behavior across different LLM providers introduces architectural complexity. The challenge was to develop a reusable enterprise AI assistant framework that could securely retrieve organizational knowledge, respect user-level access permissions, support configurable assistants, and provide an extensible foundation for integrating external tools and services.

Chapter 02

Why AI Was the Right Solution

Why AI was the right approach — and what alternatives were considered.

Traditional keyword-based search is often insufficient when employees need answers expressed in natural language or when relevant information is distributed across multiple documents.

An LLM-powered retrieval approach allows users to ask questions conversationally while grounding responses in relevant organisational information. Combining retrieval with an LLM provides contextual answers without relying exclusively on the model's internal knowledge.

The decision to use a retrieval-augmented approach also supports updating the knowledge base as documents change, without requiring model fine-tuning for every content update. Access-control filtering provides an additional layer for enterprise use cases where different users have different information permissions.

Chapter 03

How the Solution Was Delivered

Discovery, design, development, testing and rollout — the journey, not the tooling.

  1. 1Defined the framework architecture around configurable assistants and reusable backend services.
  2. 2Established the document ingestion pipeline covering source discovery, access metadata capture, parsing, chunking and embedding generation.
  3. 3Implemented PostgreSQL with pgvector for persistent document and vector storage.
  4. 4Developed semantic retrieval with tenant, assistant and access-label filtering.
  5. 5Implemented a provider-agnostic authentication layer using JWT/OIDC validation and a common principal context.
  6. 6Integrated Keycloak as the local development identity provider.
  7. 7Built the FastAPI backend with configuration-driven assistant definitions and Pydantic-based validation.
  8. 8Integrated the LLM orchestration layer through a model-provider abstraction.
  9. 9Validated authentication, ingestion, retrieval and backend functionality through automated tests.
  10. 10Established a React and Vite-based embeddable assistant interface for integration with applications.
Chapter 04

Key Technical & Architecture Decisions

Architecture, model selection, workflow and trade-offs.

The framework was designed using modular components to separate document ingestion, retrieval, authentication, authorization and LLM orchestration.

FastAPI was selected for the asynchronous backend API, with Pydantic v2 for structured configuration and data validation. PostgreSQL with pgvector provides persistent storage and vector similarity search.

A model-provider abstraction was introduced to reduce dependency on a single LLM provider and make model integrations interchangeable. Google ADK and Gemini were used in the model integration layer.

A dedicated PrincipalContext represents the authenticated user's identity, clearance and namespaced access labels. Retrieval applies tenant, assistant and access-label constraints to prevent unrestricted document access.

Keycloak was used as the local development identity provider, while the authentication design supports integration with an external organisational identity provider in the futur

Chapter 05

Challenges & How They Were Solved

Obstacles hit along the way and how they were overcome.

One important engineering challenge was ensuring that document permissions remained consistent throughout the ingestion and retrieval lifecycle. Access metadata needed to be captured before parsing and chunking so that every generated chunk retained the appropriate access labels.

This was addressed by capturing namespaced access labels during the ingestion process and propagating them to every chunk before storage.

Another challenge was separating authentication from authorization. Rather than embedding permission logic throughout individual API endpoints, the framework introduced a common PrincipalContext and authorization policy layer.

The system also needed to remain independent of a particular LLM provider. A common ModelProvider interface was introduced to isolate provider-specific implementation details from the orchestration layer.

Automated tests were used to validate key backend, authentication, ingestion and retrieval workflows.

Chapter 06

User Adoption & Change Management

How users responded — training, change management and feedback loops.

The initial implementation focused on establishing and validating the technical foundation rather than conducting an organisation-wide rollout.

The assistant architecture was designed to support configurable use cases and future integration into existing applications through an embeddable frontend widget and backend APIs.

For a future enterprise rollout, the adoption approach would include controlled user onboarding, integration with the organisation's identity provider, permission validation, feedback collection and monitoring of retrieval quality.

No organisation-wide adoption or end-user impact metrics have been established at this stage.

Business Impact

Business Outcomes & Impact

The measurable outcomes from the underlying AI Deployment — with the story behind them.

Hours Automated
0
Cost Savings
Currency not specified
Revenue Impact
Currency not specified
Measurable Business Outcome

Established a reusable enterprise AI assistant framework with end-to-end document ingestion, semantic retrieval, and permission-aware access controls. Implemented tenant, assistant, and role-based filtering, enabling secure knowledge retrieval. Validated core backend functionality, authentication, ingestion, and retrieval workflows through automated tests. Quantitative business impact and production performance benchmarks have not yet been established.

Business Outcome Categories

Risk ReductionKnowledge ManagementAI Performance ImprovementQuality Improvement

The story behind the numbers

The implementation established a reusable technical foundation for enterprise AI assistants rather than a single-purpose chatbot. Organisational documents can be processed through a structured ingestion pipeline and made searchable through semantic retrieval.

A key outcome was the implementation of permission-aware retrieval using tenant, assistant and access-label filtering. This helps ensure that retrieved knowledge is constrained by the requesting user's authorised context.

The framework also separates authentication, authorization, ingestion, retrieval and model orchestration into distinct components, making the system easier to extend and maintain.

Core backend, authentication, ingestion and retrieval workflows were validated through automated testing. Quantitative business benefits, production usage metrics and cost savings have not yet been established.

Reflection

Lessons Learned

What surprised the team, what worked well, and what would be done differently.

  • Access-control metadata should be captured before document parsing and chunking.
  • Every document chunk must retain its original authorization context.
  • Authentication and authorization should be separate architectural responsibilities.
  • Retrieval filters must enforce permissions before information reaches the LLM.
  • Model-provider abstraction reduces dependency on individual LLM vendors.
  • Automated testing is essential for validating security-sensitive retrieval workflows.
  • Enterprise AI architecture should be designed around extensibility, maintainability and security from the beginning.
Looking Ahead

Future Opportunities

Where this solution could go next.

The framework can be extended with MCP-based tool integrations, enabling assistants to move beyond knowledge retrieval and interact with external business systems.

Further opportunities include advanced RAG techniques such as reranking, query rewriting, hybrid search, contextual chunking and parent-child retrieval to improve retrieval relevance.

Additional enhancements could include systematic LLM evaluation, retrieval quality benchmarks, expanded observability through Langfuse and OpenTelemetry, and integration with enterprise identity providers such as Microsoft Entra ID.

The longer-term objective is to evolve the framework into a reusable platform for secure, configurable and tool-enabled enterprise AI assistants.

For Peers

Professional Reflections

Guidance for another builder tackling a similar problem.

One of the most important lessons from this implementation was that enterprise AI requires more than connecting an LLM to a vector database. Identity, permissions, data ingestion and retrieval boundaries need to be considered from the beginning.

Capturing access metadata before document chunking was an important architectural decision because permissions must remain consistent across every derived chunk.

Separating model integration from business logic also makes the framework easier to maintain and extend. I would continue prioritising modular architecture, fail-closed authorization and automated testing when developing similar enterprise AI systems.