Two years ago, "run it yourself" meant settling for a model that was clearly worse than the cloud. That's no longer true. Open-weight models from OpenAI, Google, NVIDIA, Mistral and others now handle summarization, drafting, extraction, search and tool-calling well enough for most production work, and the hardware to run them has collapsed from a data-center rack to something that fits under a desk.
When private AI is the right call
- The data can't leave. Privileged legal material, PHI, controlled unclassified information, trade secrets, M&A work. A private model removes the third party entirely, which is a far simpler story for clients, auditors and opposing counsel.
- The volume is high and steady. Document review, transcription, classification, overnight batch summarization and always-on agents burn tokens around the clock. At high utilization, owned hardware is cheaper per token than any API, and the bill doesn't move when usage spikes.
- You need control. The model version you validated stays the model you run. No silent updates, no terms-of-service changes, no outage on someone else's status page, and it keeps working on an air-gapped network.
When it isn't
I'll tell you if the cloud is the better answer. Low or bursty volume, work that needs the very best frontier reasoning, or no one to own the system after go-live all point toward an enterprise plan with strong controls instead. Many companies end up hybrid: private models for the sensitive and high-volume work, frontier models for everything else, one gateway in front of both. See Claude and ChatGPT Enterprise rollouts for that side.
Picking the model
Models are chosen by task, license and origin, not by leaderboard rank. A short version of what I recommend as of this review:
| Task | Strong open-weight choices | Notes |
|---|---|---|
| General assistant, RAG over company documents | gpt-oss-120b, Nemotron 3 Super, Gemma 4 31B | US-origin, permissive licenses; gpt-oss-120b fits in about 65GB |
| Coding and agents | DeepSeek V4 Flash, GLM-5.3-Flash, Mistral Medium 3.5 | Check origin rules before choosing a Chinese-origin model |
| Very long documents | Models with 256K to 1M context, served on NVIDIA hardware | Reading the prompt dominates the time, and NVIDIA is much faster at it |
| OCR and scanned documents | olmOCR 2, PaddleOCR-VL | Small models that run alongside the main one |
| Search: embeddings and reranking | Qwen3-Embedding and Reranker, BGE-M3 | The quiet half of every good RAG system |
| Transcription | Whisper large-v3-turbo, NVIDIA Parakeet and Canary | Depositions, calls, dictation, clinical encounters |
Origin and license matter to regulated clients. The FY2026 defense authorization act bars DoD contractors from using DeepSeek, and some boards simply won't accept Chinese-origin weights. Several popular "open" models also carry custom licenses with revenue thresholds or attribution clauses. I check both before anything is shortlisted, then run a bake-off on your own documents, because published benchmarks rarely predict performance on a firm's real files.
Sizing the hardware
For generation, speed is mostly set by memory bandwidth; for reading long prompts, by raw compute. Agents and document work read far more than they write, which is why the same model can feel very different on two machines with similar specs.
| Tier | Typical hardware | What it runs well | Good for |
|---|---|---|---|
| Workstation pilot | One NVIDIA DGX Spark, one AMD Strix Halo 128GB system, or a Mac Studio M5 Ultra | gpt-oss-120b class models at roughly 35 to 56 tokens per second for one user | A proof of concept, a practice group, a single always-on agent |
| Department | Two linked DGX Sparks, a 4-node Mac Studio cluster over Thunderbolt 5, or one server with four professional GPUs | 200B to 400B parameter models, or a mid-size model shared by dozens of users | A department, a few hundred documents an hour, shared agents |
| Enterprise | A multi-GPU server or cluster, for example 8x NVIDIA RTX PRO 6000 Blackwell on PCIe Gen5, served by vLLM or SGLang | Frontier-class open models with high concurrency, plus OCR, embeddings and transcription side by side | Hundreds of users, firm-wide document pipelines, many agents |
A few things I've learned the hard way and check on every design:
- Clustering small boxes adds memory, not speed. Linking machines lets a bigger model fit, but each request usually gets slower, and Thunderbolt 5 has no switches, so Mac clusters top out around four machines.
- Old GPU servers can be a false economy. A 2019-era 10-GPU chassis looks cheap refurbished, but PCIe Gen3 links and an unsupported mix of modern cards can make an eight-GPU model slower than a four-GPU one. Sometimes the right answer is several smaller model copies instead of one big one.
- Memory prices moved a lot in 2026. The same box can cost very different amounts from one quarter to the next, so every quote I give is dated.
What a private deployment includes
- Serving. vLLM or SGLang for many concurrent users, llama.cpp or MLX for single boxes, behind an OpenAI-compatible endpoint so any approved tool can use it.
- Gateway and identity. Single sign-on, per-team keys and budgets, model routing and a complete request log. The same gateway can send non-sensitive work to a frontier model.
- Permission-aware retrieval. Documents are indexed with their access rules, and every search is filtered by who is asking. An associate never sees a matter they're walled off from just because the AI could.
- Evaluation. A test set built from your own documents and questions, run before go-live and after every model change, with results you can show a client or a regulator.
- Operations. Monitoring, patching, model updates, backups and a runbook, run by me as an ongoing service or handed to your team.
From the lab: a single 128GB AMD Strix Halo workstation runs a 35B-parameter mixture-of-experts model for a whole fleet of internal agents and automations, at about 54 tokens per second generation and about 700 tokens per second prompt processing at 11K to 25K tokens of context, with no per-token cost. Getting there took a bake-off: the Vulkan build of llama.cpp beat the HIP and ROCm builds, and two of the candidate models failed the smoke test outright.
A published example worth knowing
In late 2025, Cliniques Universitaires Saint-Luc in Brussels published how it runs a private assistant for clinicians: Qwen3-235B served by vLLM on eight NVIDIA H200 GPUs, LiteLLM for access control, and retrieval over its own clinical data. Over five months, 1,028 staff held almost 15,000 conversations, mostly retrieval, summarization and note drafting. The authors also note that a capable setup is possible for a fraction of their hardware budget. It's a good reference for what a governed private deployment looks like in a regulated environment. (PLOS Digital Health)
Frequently asked questions
Is a local LLM as good as ChatGPT or Claude?
For the hardest reasoning, frontier models are still ahead. For most business work, such as summarizing, drafting from templates, extracting fields, searching your documents and running agents with tools, today's best open-weight models are good enough, and the gap keeps closing. A bake-off on your own documents settles it.
Does running AI on-prem make us HIPAA compliant?
No tool makes you compliant on its own. Keeping PHI on your own hardware removes the third-party processor from the picture, which simplifies things a lot, but you still need access controls, audit logs, encryption, backups and policies that cover the AI system like any other system holding PHI.
What hardware do we need for a private LLM?
It depends on the model, the number of people using it at once and how long the documents are. A pilot for one team can run on a single workstation-class box. A firm-wide deployment usually needs a multi-GPU server. I size from your real workload, not the other way around.
How much does private AI cost?
A workstation pilot is a desktop-computer purchase plus setup. A department system is in the range of a high-end server. An enterprise cluster is a capital project. The break-even against cloud APIs depends almost entirely on how busy the hardware stays, and I model that before you buy anything.
Can private AI and cloud AI work together?
Yes, and that's usually the best design. A gateway in front of both decides where each request goes, so sensitive prompts stay on your hardware and everything else can use a frontier model.
Do you set this up outside Arizona?
Yes. I'm in Scottsdale and work on site across the Phoenix area, and I deploy and support systems nationally.