JustAnotherBot: Local LLM-Powered Discord Assistant with RAG and Persistent Memory
A locally hosted AI-powered Discord assistant built with Python and Ollama, featuring LLM-based conversations, persistent memory, document-based RAG, semantic search, and customizable personality modes.
Business Challenge
Developing a conversational AI assistant that could maintain contextual conversations, remember information across sessions, and answer questions using external documents rather than relying entirely on an LLM's pretrained knowledge. The system needed to combine conversational memory, document retrieval, and local LLM inference while operating under hardware constraints, including 8 GB RAM and no dedicated NVIDIA GPU. Additional challenges included maintaining relevant conversation context, reducing irrelevant document retrieval, managing persistent storage, and optimizing response performance without depending on paid cloud AI APIs.
Solution Delivered
Designed and developed JustAnotherBot, a locally hosted AI-powered Discord assistant using Python, discord.py, and Ollama. The implementation includes: 1. LLM integration- Integrated Qwen 2.5 (3B) through Ollama for locally executed conversational AI. 2. Conversational memory- Implemented conversation history and persistent long-term memory using SQLite. 3. RAG pipeline- Developed document ingestion, text extraction, chunking, embedding generation, and semantic retrieval. 4. Vector database- Integrated ChromaDB with EmbeddingGemma to store and retrieve document embeddings. 5. Retrieval optimization- Implemented Maximum Marginal Relevance (MMR) retrieval to improve the diversity and relevance of retrieved document chunks. 6. Discord integration- Built message handling, slash commands, personality switching, memory management, and document-based question answering. 7. Resource optimization- Designed the system to operate locally on hardware with limited RAM and no dedicated GPU. The architecture combines conversational AI, persistent storage, vector search, and document-grounded responses within a single Discord application.
Outcomes Achieved
Successfully developed and tested an end-to-end local AI assistant integrating conversational LLM inference, persistent memory, and Retrieval-Augmented Generation. Technical outcomes include: - Indexed approximately 671 document chunks in the persistent ChromaDB collection during development and testing. - Implemented semantic retrieval using EmbeddingGemma embeddings. - Successfully tested document retrieval across chapter-based and page-based content. - Implemented MMR retrieval to improve retrieval diversity and reduce redundant context. - Integrated SQLite-based persistent conversation and memory storage. - Established a fully local inference and retrieval workflow without requiring paid cloud LLM APIs. The project demonstrates the feasibility of building a multi-component Generative AI application on resource-constrained consumer hardware. These results are development and testing outcomes, not measured commercial business improvements.
Measurable Business Outcome
Developed a persistent document retrieval system with approximately 671 indexed chunks during testing. Successfully implemented and tested semantic search, document chunk retrieval, and MMR-based retrieval using ChromaDB and EmbeddingGemma. Integrated persistent conversational memory through SQLite and local LLM inference through Ollama. The implementation demonstrates document knowledge retrieval, contextual conversation handling, and locally executed AI workflows on an 8 GB RAM system without a dedicated GPU. These are technical project outcomes rather than quantified business savings.