Every private AI project eventually reaches the same question: what do we actually need to buy? The honest answer depends on three things more than any spec sheet: how many people or agents use it at once, how long the prompts are, and how big a model the work really needs.
Sizing calculator
Find your starting tier
- Serve it with
- Example models
- Watch for
A starting point, not a quote. Real sizing depends on context length, response length, peak versus average load and the rules your data falls under.
Two numbers that decide everything
Generation speed (tokens per second while the model writes) is set mostly by memory bandwidth. Every token means reading the model's active weights from memory, so a box with twice the bandwidth writes roughly twice as fast.
Prompt processing (tokens per second while the model reads) is set mostly by compute. This is where NVIDIA's tensor cores pull ahead, often by a wide margin.
Which one matters depends on the work. A person chatting reads short prompts and watches the answer stream, so generation speed is what they feel. Document review, RAG and agents are the opposite: one published measurement of a coding-agent session found about 170 tokens read for every token written. For that work, prompt processing speed dominates.
Two more rules of thumb:
- Mixture-of-experts models are the sweet spot. A model like gpt-oss-120b holds 117 billion parameters in memory but only uses about 5 billion per token, so it fits in 128GB and still runs fast. Dense models of similar size crawl on the same hardware.
- Memory decides what fits; bandwidth decides how fast. At 4-bit quantization a model needs roughly 0.55 to 0.6 bytes per parameter, plus 10 to 30% for context. Plan for the context length you'll really use, not the demo.
Tier 1: one box on a desk
For a pilot, a practice group, a single always-on agent or one power user. All three run a 120B-class model comfortably.
| NVIDIA DGX Spark | AMD Strix Halo (Ryzen AI Max+ 395) | Apple Mac Studio M5 Ultra | |
|---|---|---|---|
| Memory | 128GB unified | 128GB unified | 96GB to 512GB unified |
| Bandwidth | 273 GB/s | 256 GB/s | 1.2 TB/s |
| gpt-oss-120b, generating | 35 to 55 tok/s | 47 to 56 tok/s | not yet measured cleanly |
| gpt-oss-120b, reading prompts | 1,000 to 2,400 tok/s | 730 to 1,000 tok/s | strong, behind NVIDIA |
| Power | 240W supply | about 120 to 140W at the wall | up to 300W |
| Price, Nov 2026 | $6,950 (64GB model $4,999) | $3,999.99 | $7,299 (96GB), $11,299 (256GB), $22,799 (512GB) |
| Best engines | vLLM, SGLang, TensorRT-LLM, llama.cpp | llama.cpp (Vulkan or ROCm), Ollama, LM Studio | MLX, Ollama, LM Studio |
| Best for | Agents and long documents; the same software stack as NVIDIA servers | Best value per gigabyte; single-user chat and automations | The largest models in one box; very fast generation |
DGX Spark is the one I'd pick when the pilot is meant to grow. It reads long prompts 1.5 to 2.5 times faster than Strix Halo, and everything you build on it (CUDA, vLLM, the container images) moves straight to a GPU server later. Dense models are its weak spot: a 70B dense model runs at about 3 tokens per second.
Strix Halo boxes from Framework, HP, Minisforum, GMKtec, Beelink and others are the best value per gigabyte. Use llama.cpp: the Vulkan build usually generates fastest and ROCm usually reads prompts faster, while vLLM support on this chip is still experimental. A refresh with up to 192GB (Ryzen AI Max+ PRO 495) starts shipping in November 2026.
Mac Studio M5 Ultra shipped in September 2026 with about 1.2 TB/s of bandwidth, roughly four times the others. MacStories measured a 125B mixture-of-experts model at 54 to 75 tokens per second generating and over 2,700 tokens per second reading on a 256GB unit, about 70% faster generation than the M3 Ultra. With 512GB (shipping late October) it holds models nothing else on a desk can. Long prompts are still slower than on NVIDIA, and macOS is a weaker server platform: no out-of-band management, fewer enterprise tools.
From the lab: one 128GB Strix Halo workstation runs a 35B-parameter mixture-of-experts model for a fleet of internal agents and automations at about 54 tokens per second generating and about 700 tokens per second reading at 11K to 25K tokens of context. llama.cpp's Vulkan build beat the HIP and ROCm builds on this hardware.
Tier 2: two boxes, or four Macs
When one box can't hold the model you need, you can link several. The catch, every time: linking machines adds memory, not speed. Each request has to cross the link, so a single user usually gets the same or lower speed, in exchange for a bigger model.
| Setup | Memory | What it runs | Measured |
|---|---|---|---|
| 2x DGX Spark over ConnectX-7 (200 Gb/s) | 256GB | 235B to 400B mixture-of-experts models | Qwen3 235B about 11.7 tok/s (TensorRT-LLM); Qwen3.5-397B about 14 tok/s (llama.cpp) |
| 2x Strix Halo over USB4 or 10GbE | 256GB | Large models, slowly | Clustering four Strix Halo boards made gpt-oss-120b slower: 24 tok/s vs 33 on one board |
| 4x Mac Studio over Thunderbolt 5 with RDMA | up to 2TB | 700B to 1T+ parameter models | Apple, 4x M5 Ultra: an 816B model at 36.7 tok/s. Jeff Geerling, 4x M3 Ultra: DeepSeek V3.1 at 32.5 tok/s, Kimi K2 at 25 to 30 |
Dual DGX Spark is the most capable two-box setup for teams. With vLLM's tensor parallelism, the pair runs 1.71 times faster than one box for a single user and holds about 1.5 times at 32 concurrent users. For many users on a model that fits in one box, it's usually better to run two independent copies instead.
Two Strix Halos are better sold as two separate servers than as one cluster. The network link, not the chip, becomes the limit.
A Mac Studio cluster is the only way to put a trillion-parameter model on a desk. Apple added RDMA over Thunderbolt 5 in macOS 26.2, and MLX and exo use it to split a model across machines with very low latency. Three caveats:
- Four Macs is the proven ceiling. There are no Thunderbolt 5 switches, so every Mac cables directly to every other. Five Macs can be fully meshed, but that ties up four of each machine's five Thunderbolt ports, scaling is already well below linear at four, and Apple's and the community's published RDMA results stop at four. A five-node build is an experiment, not a design I'd sell. If you need more, you've outgrown the desk.
- It's built for one user at a time. These setups are fast for a single conversation, not for 30 concurrent ones.
- Long prompts are the weak spot. In one comparison, a 32,000-token prompt took 44 to 50 seconds to process on an M5 Ultra and about 17 seconds on two DGX Sparks.
Tier 3: GPU servers
Once more than a few dozen people or agents depend on the system, you need a GPU server and a serving engine built for concurrency (vLLM or SGLang). This is where most mid-market and enterprise deployments land.
| GPU | Memory | Bandwidth | Power | Street price |
|---|---|---|---|---|
| NVIDIA RTX PRO 5000 Blackwell | 48GB or 72GB | 1.34 TB/s | 300W | $8,000 (48GB), about $9,000 (72GB) |
| NVIDIA RTX PRO 6000 Blackwell | 96GB | about 1.8 TB/s | 300 to 600W | $16,500 |
| NVIDIA H200 / B200 (8-GPU NVLink systems) | 141GB / 180GB+ each | 4.8 TB/s+ | 700W+ each | about $45,000 per H200; roughly $300K to $500K for an 8-GPU system |
The budget route: a refurbished Dell DSS 8440
The DSS 8440 is a 4U server that takes up to ten double-wide GPUs, and refurbished units are cheap. It's tempting to fill one with RTX PRO 5000 cards. Know what you're buying:
- It's a 2019 platform. Second-generation Xeon Scalable means PCIe Gen3: a quarter of the bandwidth between GPUs that a modern server gives you. Splitting one model across many GPUs depends on exactly that bandwidth.
- It was qualified for passive data-center cards (V100, T4, Quadro RTX). Actively cooled Blackwell workstation cards are an unsupported configuration: check power cabling, airflow and firmware before buying ten.
- Ten GPUs divide badly. vLLM splits a model across a number of GPUs that must divide the model's attention heads, so ten usually means eight for the model and two spare, or five pairs.
It can still be a sensible build: run several independent copies of a mid-size model on pairs or quads of GPUs, and give the spares to embeddings, OCR and transcription. That keeps traffic off the slow links and serves a lot of users. Just don't expect it to run one giant model fast.
The current route: Gen5 servers
A modern 4U server such as the Dell PowerEdge XE7745 takes eight RTX PRO 6000 Blackwell cards (96GB each, 768GB total) on PCIe Gen5, with 2x AMD EPYC processors and room for fast networking. Even here, without NVLink, more GPUs per copy isn't always faster: one test measured a 235B model at 58 tokens per second on four GPUs and 37 on eight. On the other hand, a single RTX PRO 6000 serving a mid-size model to many users at once has been measured above 3,000 tokens per second in total, so a well-configured eight-GPU box serves hundreds of people.
For the largest open models at high concurrency, 8-GPU NVLink systems (H200 or B200 class) are still the answer. They're a data-center purchase: plan for 10kW+ per server and proper cooling.
What I recommend most often
| Situation | Starting point |
|---|---|
| Pilot for one team, document or agent work | One DGX Spark |
| Pilot, chat and drafting, tight budget | One Strix Halo 128GB system |
| One power user who needs the biggest open models | Mac Studio M5 Ultra 512GB, or four of them |
| A department of 15 to 60 people | One server with 2 to 4 RTX PRO 6000 cards, vLLM, behind a gateway |
| Firm-wide, 60 to 250+ people | One or two 8-GPU Gen5 servers, several model copies, Kubernetes |
| Largest models at high concurrency | 8-GPU NVLink systems in a data center or colocation |
Every one of these goes behind an AI gateway with single sign-on and logging, and every one gets an evaluation run on your own documents before anyone relies on it. See Private and on-prem AI for the rest of the build, and Open models by task for what to run on it.
Sources
Speeds above are published measurements unless marked as ours. Main sources: NVIDIA on DGX Spark performance, llama.cpp community benchmarks, the Strix Halo benchmark guide, Apple Machine Learning Research on MLX distributed inference, MacStories' M5 Ultra review, Jeff Geerling's Mac Studio RDMA and Framework cluster tests, CloudRift's RTX PRO 6000 benchmarks, ServeTheHome and Dell's XE7745 specifications. Prices are November 2026, except the 72GB RTX PRO 5000 and the 8-GPU system range, which are October 2026 street prices. All of them moved sharply during the 2026 memory shortage.
Frequently asked questions
DGX Spark or Strix Halo?
For one person chatting, they generate at similar speeds and Strix Halo costs about 40% less. For agents, RAG and long documents, the DGX Spark reads prompts 1.5 to 2.5 times faster and runs the same NVIDIA software as production servers, which usually makes it the better pilot.
Is the Mac Studio M5 Ultra good for local AI?
Yes, especially for one user who needs a very large model: up to 512GB of memory and about 1.2 TB/s of bandwidth make it the fastest desktop at generating. It's weaker at reading long prompts than NVIDIA hardware, and macOS lacks the remote management most IT teams expect from a server.
Can I cluster five Mac Studios over Thunderbolt 5?
Physically yes, but four is the proven limit. Thunderbolt 5 has no switches, so every Mac must connect directly to every other one, which uses four of each machine's five ports at five nodes. Speed gains already flatten out by four machines, and published RDMA results stop there. Beyond four, a GPU server is the better investment.
How many users can one local AI server handle?
A single desktop box is comfortable for one to a few people. A server with two to four professional GPUs running vLLM handles dozens of concurrent users on a mid-size model. An eight-GPU server running several copies of a model can serve hundreds, depending on prompt length and how bursty the usage is.
Is a refurbished multi-GPU server a good deal?
It can be, for running several independent copies of a mid-size model. Older servers use PCIe Gen3, which slows down splitting one large model across many GPUs, and modern workstation GPUs may be unsupported in them. Check power, cooling and the vendor's qualified GPU list before buying.