The Business Challenge
The business situation the organisation faced before the project began.
Developing a conversational AI assistant that could maintain contextual conversations, remember information across sessions, and answer questions using external documents rather than relying entirely on an LLM's pretrained knowledge.
The system needed to combine conversational memory, document retrieval, and local LLM inference while operating under hardware constraints, including 8 GB RAM and no dedicated NVIDIA GPU.
Additional challenges included maintaining relevant conversation context, reducing irrelevant document retrieval, managing persistent storage, and optimizing response performance without depending on paid cloud AI APIs.
Why AI Was the Right Solution
Why AI was the right approach — and what alternatives were considered.
The project required more than a conventional rule-based Discord bot. A traditional command-and-response system would have limited the assistant to predefined interactions and would not have supported flexible natural-language conversations or document-based question answering.
A language model provided the flexibility to interpret user queries and generate contextual responses. However, relying exclusively on pretrained model knowledge would limit the assistant's ability to answer questions about external documents and retain information across sessions.
Combining local LLM inference with Retrieval-Augmented Generation and persistent memory provided a practical approach. The assistant could use retrieved document context when answering knowledge-based questions while maintaining conversational continuity through stored history and memories.
Local inference was also an intentional choice to explore privacy-conscious AI development, reduce dependence on paid API services and work within the available hardware resources.
How the Solution Was Delivered
Discovery, design, development, testing and rollout — the journey, not the tooling.
- 1Problem identification and project planning.
- 2Designed the application architecture around local LLM inference and conversational interaction.
- 3Integrated the Discord interface with a Python-based message processing system.
- 4Implemented local LLM inference and tested different model configurations against available hardware.
- 5Developed conversational history and persistent memory functionality using SQLite.
- 6Built the document ingestion pipeline, including text extraction, chunking and metadata handling.
- 7Integrated embedding generation and persistent vector storage for document retrieval.
- 8Implemented semantic search and experimented with MMR to improve retrieval diversity.
- 9Integrated retrieved document context into the question-answering workflow.
- 10Tested conversation handling, memory recall, document retrieval and slash commands.
- 11Identified performance bottlenecks, debugged retrieval issues and refined the implementation through iterative testing.
Key Technical & Architecture Decisions
Architecture, model selection, workflow and trade-offs.
One of the primary decisions was to use local LLM inference rather than depending on hosted AI APIs. This allowed experimentation without per-request API costs and provided greater control over model execution.
Qwen 2.5:3B was selected as the primary conversational model through Ollama, balancing response capability with the limitations of an 8 GB RAM system without a dedicated NVIDIA GPU.
A Retrieval-Augmented Generation approach was chosen instead of fine-tuning. The goal was to allow the assistant to retrieve information from external documents without modifying model weights whenever new content was introduced.
EmbeddingGemma was used to generate document embeddings, while ChromaDB provided persistent vector storage and semantic retrieval.
SQLite was selected for conversational history and persistent memory because it offered lightweight, local storage without requiring a separate database server.
MMR retrieval was introduced to explore a balance between query relevance and diversity among retrieved document chunks. The implementation used a candidate retrieval stage followed by selection of the most relevant and diverse results.
The overall architecture prioritised modularity, local execution and manageable resource consumption over a more complex cloud-based infrastructure.
Challenges & How They Were Solved
Obstacles hit along the way and how they were overcome.
One of the biggest challenges was running an LLM-based application on hardware with only 8 GB RAM and no dedicated NVIDIA GPU. Model selection and resource consumption became important considerations. I experimented with different model sizes before settling on Qwen 2.5:3B as the primary model.
Another challenge involved document retrieval. Initial retrieval tests highlighted the importance of chunking, metadata extraction and selecting relevant context. I developed a document-processing pipeline with chapter, page and section metadata to make retrieval more structured.
I also encountered retrieval and query-validation issues while integrating ChromaDB. These were investigated through isolated test scripts, debugging collection configurations and validating retrieval outputs.
Basic similarity retrieval also raised questions about redundant context. I explored Maximum Marginal Relevance (MMR) to retrieve a more diverse set of relevant chunks.
Performance was another consideration. Embedding generation and model inference required careful testing because local hardware limitations affected execution time.
Rather than treating these issues as isolated bugs, I used them to improve my understanding of the complete pipeline, from document processing and embedding generation to retrieval and final response generation.
User Adoption & Change Management
How users responded — training, change management and feedback loops.
As an independently developed project, JustAnotherBot did not involve a formal organisational rollout or external customer onboarding.
Testing and validation were carried out through direct interactions with the bot in a Discord environment. I used different prompts and scenarios to examine conversational behaviour, memory persistence, document retrieval and command handling.
The testing process helped identify inconsistent responses, retrieval issues and areas where the assistant required further refinement.
Different personality configurations were also implemented to explore how configurable response behaviour could improve the interaction experience.
Future iterations can include structured feedback from external users, usability testing and a more systematic evaluation of response quality.
Business Outcomes & Impact
The measurable outcomes from the underlying AI Deployment — with the story behind them.
Developed a persistent document retrieval system with approximately 671 indexed chunks during testing. Successfully implemented and tested semantic search, document chunk retrieval, and MMR-based retrieval using ChromaDB and EmbeddingGemma. Integrated persistent conversational memory through SQLite and local LLM inference through Ollama. The implementation demonstrates document knowledge retrieval, contextual conversation handling, and locally executed AI workflows on an 8 GB RAM system without a dedicated GPU. These are technical project outcomes rather than quantified business savings.
Business Outcome Categories
The story behind the numbers
The project resulted in a functional, locally hosted AI assistant capable of handling conversational interactions, maintaining persistent information and answering questions using indexed documents.
It demonstrated how conversational AI and document-based knowledge retrieval can be combined into one application without relying on paid cloud inference services.
The implementation also established a reusable foundation for experimenting with more advanced AI workflows, memory strategies and retrieval techniques.
From an engineering perspective, the project provided practical experience in integrating multiple AI components, evaluating retrieval behaviour, debugging system-level issues and working within hardware constraints.
The outcomes are based on project development and testing. No external organisational productivity gains, customer adoption figures or financial savings have been claimed.
Lessons Learned
What surprised the team, what worked well, and what would be done differently.
- Understanding the complete data flow is essential when building multi-component AI applications.
- Hardware constraints should influence model selection and architecture decisions from the beginning.
- Effective document chunking and metadata extraction are important for reliable retrieval.
- MMR provides an alternative retrieval strategy when relevance and diversity both matter.
- Persistent memory requires deliberate decisions about storage, retrieval and context management.
- Testing individual components makes debugging complex AI pipelines significantly easier.
- AI-generated responses should be evaluated rather than assumed to be correct.
- Building incrementally makes experimentation and troubleshooting more manageable.
- Future improvements should be guided by observed limitations and measurable evaluation results.
Future Opportunities
Where this solution could go next.
There are several opportunities to extend JustAnotherBot beyond its current implementation.
The next stage would involve integrating LangGraph to introduce structured, stateful workflows and more advanced orchestration of AI operations.
Model Context Protocol (MCP) is another planned area of exploration, particularly for standardising how the assistant connects to external tools and data sources.
Further improvements could include more robust retrieval evaluation, automated testing, response-quality assessment and better handling of ambiguous queries.
I also plan to explore improved memory management, more efficient document processing, asynchronous execution and performance optimization.
Ultimately, the goal is to evolve the project into a more capable AI assistant with modular workflows, reliable tool integration and measurable evaluation benchmarks.
Professional Reflections
Guidance for another builder tackling a similar problem.
Building JustAnotherBot taught me that developing a useful AI application involves much more than selecting a language model and connecting it to an interface.
One of my biggest takeaways was the importance of understanding the entire workflow. Document quality, chunking strategies, embedding generation, retrieval configuration and context construction all influence the final response.
I also learned that hardware constraints should influence architecture decisions from the beginning. Choosing a model that fits the available resources is often more practical than pursuing the largest available model.
Another important lesson was the value of isolated testing. Breaking the system into smaller components made it easier to identify whether an issue originated in document processing, embeddings, vector retrieval, memory management or model inference.
Most importantly, I learned to approach AI development iteratively: implement, test, observe, debug and improve. A working prototype is only the starting point; reliability and evaluation are equally important.