Planning out a setup and would love to hear from anyone who’s actually built this rather than just theorized about it. The goal is a fully private AI system, no cloud APIs, nothing leaving our own network, that answers questions using retrieval augmented generation (RAG) against a library of documents we feed it ourselves. Not fine-tuning, just indexing our own reference material and having the model answer from that with citations back to the source. The wrinkle is scale and access. About 50 people spread out across the country need to be able to query this thing, all connecting back to one central setup at a single physical location. So it’s not a single person running Ollama on their desktop, it needs to hold up as shared infrastructure with real concurrent usage and some redundancy if a machine goes down. all 50 people will not be using it at the same time. I would guess we may have a handful of people per day. After watching NetworkChuck on YouTube, I think a small cluster of Mac Mini Pros (M4 Pro chips, 48GB RAM each) running a 30 to 35B class open-weight model, a few active nodes plus one standby for failover, with VPN access for remote users and a load balancer routing requests. Considered pooling the Mac Minis together for one big model but landed on independent nodes instead since it’s simpler and this isn’t really a “need a massive model” situation, it’s a “need reliable concurrent access” situation. Questions for anyone who’s actually done this at a similar scale: Did independent nodes with a load balancer actually hold up in practice, or did you run into issues I’m not anticipating? Any regrets on model size versus RAM tradeoffs? Went with 48GB per machine since 64GB wasn’t available at this price and chip tier. How’d you handle document ingestion as your library grew over time, anything better than just re-indexing periodically? Anything about VPN or remote access for a distributed team that bit you later? Not looking for a cloud API recommendation, the whole point is keeping this fully private and internal. Just trying to learn from anyone who’s actually built and run something like this instead of only speculating about it. submitted by /u/rogo725
Originally posted by u/rogo725 on r/ArtificialInteligence