The Business Challenge
The business situation the organisation faced before the project began.
Managing real-time IoT telemetry alongside vast volumes of technical documentation presents a dual operational challenge: processing high-throughput time-series data while enabling non-technical operators to query complex station parameters instantly.
Relying on public cloud LLM APIs introduced critical bottlenecks—high operational token costs, strict data privacy concerns regarding infrastructure logs, and unacceptable response latencies (often exceeding 13 seconds per RAG query). The organization required a resilient, self-hosted architecture that could serve intelligent document responses combined with real-time sensor context, while maintaining strict data isolation and enterprise-grade performance.
Why AI Was the Right Solution
Why AI was the right approach — and what alternatives were considered.
Static rule engines and traditional database lookups could not bridge the gap between unstructured technical manuals and dynamic, real-time sensor streams. A localized RAG system was essential to interpret complex natural language queries, extract exact context from vector embeddings, and reason over continuous environmental data in a secure, self-contained environment.
How the Solution Was Delivered
Discovery, design, development, testing and rollout — the journey, not the tooling.
- 1Requirements Discovery & Data Schema Audit: Mapped IoT time-series data pipelines and indexed technical documentation for semantic retrieval.
- 2Core Infrastructure Setup: Configured on-premise server environment with Docker containerization, TimescaleDB, and local vector indexing via Qdrant.
- 3Local LLM & RAG Engine Prototyping: Benchmarked Qwen-7B inference using llama.cpp and implemented Cross-Encoder reranking for context accuracy.
- 4Backend & API Latency Optimization: Restructured FastAPI processing pipelines and integrated smart caching to drastically compress end-to-end response times.
- 5Full-Stack Integration & Rollout: Connected Golang Fiber backend with Next.js dashboard and WebSockets for real-time alerting and interactive AI chat.
Key Technical & Architecture Decisions
Architecture, model selection, workflow and trade-offs.
Selected an on-premise RAG approach over fine-tuning or cloud LLM APIs to maintain strict data sovereignty and eliminate recurring API costs. Architected the LLM core using Qwen-7B compiled with llama.cpp, leveraging local hardware for low-latency inference. Chosen Qdrant as the vector database for its high-performance similarity search, and integrated a Cross-Encoder reranking layer to solve classic vector-search precision limits. For backend throughput, decoupled real-time data ingestion (Golang Gin & TimescaleDB) from AI processing (FastAPI), ensuring heavy AI queries never bottleneck live IoT telemetry.
Challenges & How They Were Solved
Obstacles hit along the way and how they were overcome.
High Latency Bottleneck:
Initial RAG implementation suffered from high response times (~13.5s). Solved by re-architecting the FastAPI pipeline, optimizing vector payload indexing in Qdrant, and fine-tuning local llama.cpp execution parameters—drastically cutting latency down to 1.2s.
Retrieval Relevance Noise:
Standard cosine similarity search occasionally returned loosely related technical documentation. Solved by implementing a two-stage retrieval pipeline: Qdrant fetches candidate vectors, and a Cross-Encoder reranker re-scores top results for peak precision before prompting the LLM.
User Adoption & Change Management
How users responded — training, change management and feedback loops.
Rolled out the system in a phased approach alongside the existing IoT monitoring dashboard. Field operators initially used the AI assistant in shadow mode to compare generated documentation answers against manual lookup procedures. Feedback loops were established to log unanswered or inaccurate queries, allowing continuous refinement of document vector chunking strategies. Trust was rapidly built once operators verified sub-second response times and precise context accuracy during live environmental events.
Business Outcomes & Impact
The measurable outcomes recorded on this AI Case Study — with the story behind them.
AI System Optimization: Dramatically improved the system's end-to-end RAG response pipeline latency from an unstable 13.5 seconds down to a production-grade 1.2 seconds, achieving an 11x performance speedup. Local Inference Efficiency: Optimized on-premise LLM execution using llama.cpp to achieve a rapid inference speed of 0.04s per request with the Qwen-7B model. Search Precision & Accuracy: Enhanced information retrieval precision by implementing a two-stage retrieval pipeline using Qdrant vector database search combined with Cross-Encoder semantic reranking. Data Privacy Compliance: Eliminated data leak risks and reliance on external cloud APIs by deploying a 100% locally-hosted AI processing architecture integrated into a live IoT analytics monitoring dashboard.
Business Outcome Categories
The story behind the numbers
The deployment transformed the organization's technical operations across three key dimensions:
- Massive Speed Efficiency: End-to-end RAG query latency was reduced from 13.5 seconds down to a crisp 1.2 seconds—an 11x performance increase. Raw local LLM inference achieved a blazing 0.04s execution time.
- 100% Data Sovereignty & Zero API Overhead: By hosting the entire inference engine on-premise, the company eliminated recurring cloud API expenditures while guaranteeing complete security for sensitive infrastructure logs.
- Enhanced Decision-Making: Field personnel can now query complex operational manuals and receive context-aware answers instantly alongside real-time IoT sensor telemetry, drastically cutting downtime and incident response times.
Future Opportunities
Where this solution could go next.
Multi-Agent Orchestration:
Expanding the single RAG pipeline into specialized multi-agent workflows (e.g., dedicated agents for IoT predictive maintenance and automated security/SOC threat explanation via tools like n8n).
Predictive Analytics Integration:
Integrating real-time time-series forecasting models directly into the AI conversational context for automated water level and environmental anomaly alerts.
Professional Reflections
Guidance for another builder tackling a similar problem.
- On-Premise AI is Production-Ready:
Building a fully localized RAG architecture proved that organizations do not need to compromise on data privacy or run up massive cloud API bills to deploy powerful AI capabilities. With open-weight models like Qwen-7B and runtime engines like llama.cpp, enterprise-grade performance can be achieved on dedicated hardware.
- Optimization is Mandatory for User Experience:
A raw RAG pipeline that takes 13+ seconds to respond is functionally unusable in real-time operational environments. Focusing heavily on pipeline efficiency—reducing latency down to 1.2s through vector indexing optimization and smart caching—was the single most critical factor in driving user adoption.
- Reranking Solves Retrieval Noise:
Vector search alone is rarely enough for high-stakes technical documentation. Adding a Cross-Encoder reranking stage dramatically increased response precision, proving that retrieval architecture is often more important than raw model size.
- Decouple AI from Core Data Ingestion:
Separating high-throughput data pipelines (Golang/TimescaleDB) from AI orchestration (FastAPI) ensured that intensive vector searches and LLM inferences never impacted real-time IoT monitoring or live dashboard telemetry.