Informatica :: A private chatbot on my own hardware
Informatica started as a "talk to your documents" project. My wife works with a lot of legal texts and legislation, and she wanted to search them in plain language instead of scrolling through PDFs. That became my excuse to learn Retrieval Augmented Generation (RAG) and to upgrade my home server setup.
Over time it grew into something more useful: a general-purpose chat assistant that runs entirely on my own machines. Ask it to write a shell script, explain a concept or draft an email and it just answers. Ask it about a document you've uploaded and it pulls in the relevant parts of that document. No data leaves the house.
Access is private: you sign in with Google, and I approve who gets in.
The hardware
"Kottos" is a second-hand Dell Optiplex with a 6-core Intel Core i5 8500, 16GB of DDR4 RAM and a 512GB SSD, bought for just over 100 bucks. It started with a tiny Nvidia T600 (4GB), which I have since replaced with a low-profile Nvidia T1000 with 8GB of VRAM. That doubles the memory, so a 9-billion-parameter model now fits entirely on the GPU, together with a 32k-token context.
Kottos is the control plane of my k3s Kubernetes cluster, with a few more machines joined as workers. Next to it sits "Briareos", my gaming desktop with an RTX 4090 (24GB). It joins the cluster as a worker that is allowed to come and go: when it's on, Informatica uses a much bigger model there; when I'm gaming or it's switched off, everything quietly falls back to Kottos.
The models
Everything runs on Ollama. On Kottos it's Qwen 3.5 9B (4-bit quantised), kept loaded in VRAM all the time. When Briareos is up it switches to Qwen 3.8 27B, which is noticeably smarter and, on a 4090, also faster. Informatica only uses Briareos when the big model is already loaded there, so you never sit waiting for a cold start. If Briareos disappears in the middle of an answer, the answer restarts on Kottos.
The Kottos model is shared: my home cluster's AI agent and a few small jobs use the same loaded copy, so nothing gets swapped in and out of the GPU.
A chatbot first, documents second
Every message goes to the model as a normal conversation: system prompt, previous turns and your question, in a single call. Documents only come into play when they're actually relevant. Your question is turned into an embedding and compared with your library in the Weaviate vector database. Only excerpts that are close enough (by cosine distance) are handed to the model, together with the file they came from. An off-topic question gets no document context at all and simply gets a normal answer. If the automatic choice guesses wrong, you can switch document use to "Always" or "Never".
How documents get in
You upload PDFs and the text is extracted with PyMuPDF. It's cut into overlapping chunks of about 1000 characters, so a sentence is never lost on a boundary. Each chunk is embedded with nomic-embed-text and stored in Weaviate, tagged with its file name and its owner. Everyone's library is private: searching, listing and deleting only ever see your own documents.
Answers stream in token by token. On modest hardware that makes a big difference in how fast it feels. The interface is built with Streamlit, the model calls go through LangChain, and users and access requests live in Cassandra. I get an email when someone new asks for access.
You can find the source code here, and try it here (Google sign-in, access on request).
