Local LLM Hardware Guide 2026: Servers, Workstations & GPUs

The case for local inference has gotten stronger every quarter for three years running. In 2026, four arguments close most deals:

• Cost predictability. A single in-house inference server amortizes in 6–18 months versus per-token cloud billing — and after that, every additional token is essentially free.
• Data privacy and compliance. HIPAA, attorney-client privilege, financial-services data, ITAR-controlled documents, and GDPR-sensitive PII never leave your network. No third-party retention policies, no shifting terms of service.
• Latency and offline operation. Time-to-first-token under 100 ms is realistic on local hardware. Round trips to a cloud API rarely beat 300–500 ms. Air-gapped deployments work at all.
• Model freedom. Switch between Llama, Qwen, DeepSeek, Gemma, Mistral, and specialized models (coding, embeddings, vision, ASR) without re-architecting integrations. Fine-tune on your own data without sending it anywhere.

The 2026 generation of open-weight models has closed most of the gap with GPT-4-class proprietary systems on the tasks that matter for business automation. A quick map of what is available right now:

General-purpose chat and reasoning

• Llama 4 Scout — 17B active / 109B total parameters (MoE, 16 experts), 10M-token context window. Best-in-class for long-document RAG and multi-file code review. Fits on a single 80 GB H100 at Q4.
• Llama 4 Maverick — 17B active / 400B total (MoE, 128 experts), 1M context. Strong instruction following, tool use, and multimodal reasoning.
• Qwen 3.5 — released February 2026; family ranges from 0.8B / 2B / 4B / 9B dense models for edge use up to a 397B-A17B flagship MoE, with 201-language coverage and a native 262K context window across every size.
• DeepSeek V4 Pro — released April 2026; a 1.6-trillion-parameter MoE with ~49B active per token, 1M-token context window, and Compressed Sparse Attention for cheap long-context prefill. One of the strongest open-source models on hard reasoning and long-context benchmarks.
• DeepSeek V4 Flash — a 284B-parameter MoE with 13B active per token. Smaller and cheaper to serve than V4 Pro, designed for high-throughput production workloads.
• Gemma 4 — released April 2026 under Apache 2.0; four sizes including Effective 2B and 4B variants for on-device, a 26B Mixture-of-Experts model, and a 31B dense flagship. Widely used for compliant on-device and edge deployments.
• Mistral Medium 3.5 — released April 29, 2026; a 128B dense model with a 256K context window, unifying chat, reasoning, and code in a single set of weights. European data residency.

Specialized models worth knowing

• Qwen3-Coder (30B-A3B and 480B-A35B MoE variants) and DeepSeek-Coder V4 — code completion and refactoring quality comparable to commercial copilots; the obvious choice when your codebase is under NDA.
• Kimi K2.6 — Moonshot AI's reasoning and coding flagship; 1T total / 32B active MoE, Modified MIT license, 262K context. Among the strongest open models on real-world software-engineering benchmarks (SWE-Bench Verified).
• BGE-M3, Nomic Embed v2, Qwen3 Embedding — embedding models for RAG; cheap to run, often on CPU.
• Whisper Large v3 Turbo — OpenAI's open ASR model, real-time transcription on a single mid-range GPU.
• Qwen 3 VL, Llama 4 (natively multimodal), InternVL 3 — vision-language models for document understanding, OCR, screenshot reasoning.
• Stable Diffusion XL Turbo, FLUX.1 — image generation that pairs naturally with the same GPU server.

Hardware is easy to justify when you have a concrete workload in mind. (For broader background on AI server deployments, see our overview of how to use an AI server for your business and the companion piece on supporting your AI stack with hardware upgrades.) These eight use cases produce measurable ROI for our customers in 2026:

1. RAG over internal documents

Connect a small-to-mid model (Llama 4 Scout, Qwen 3.5 32B, Gemma 4 27B) to SharePoint, Confluence, Google Drive, ticketing systems, or document repositories via an embedding model and a vector store. Employees ask in natural language, get cited answers from internal sources. Typical impact: 40–70% reduction in time spent searching for information. Especially valuable for legal, medical, insurance, and engineering teams whose documents cannot leave the building.

2. Customer support copilot

Classify incoming tickets, route them, draft replies, surface relevant knowledge-base articles. Even a 7B–13B model runs this workload comfortably. Typical impact: first-response time cut in half, agent throughput up 30–50%, senior agents freed from tier-1 work.

3. On-prem code assistant

Replace GitHub Copilot for teams that cannot send proprietary source code to a third party — defense contractors, fintech, healthcare software, anyone with NDAs that forbid uploading code. Qwen 3 Coder, DeepSeek Coder V4, and Kimi K2.6 are the strong open options. Runs comfortably on a single workstation per small team, or one rack server for an entire engineering org.

4. Contract and document review

Extract clauses, identify deviations from a template, flag risk language, summarize amendments. Insurance, legal, procurement, and compliance teams reclaim hours per document. Long-context models (Llama 4 Scout with 10M tokens) load entire contract packages at once.

5. Voice → text → action pipelines

Whisper Large v3 Turbo transcribes calls or meetings. A 13B–32B LLM then produces summaries, action items, SOAP notes, or CRM updates. Healthcare scribe, call-center QA, sales-call analysis. Real-time transcription runs on a single mid-range GPU.

6. Email and lead triage

Score incoming leads, classify support email, summarize long threads, generate first-pass drafts. A 7B model on a workstation can process tens of thousands of messages a day with no per-token bill.

7. Data extraction and structuring

Pull structured records out of invoices, lab reports, PDFs, faxes, scans. Pair a vision-language model (Qwen 3 VL, Llama 4 Vision) with a text LLM for verification. Replaces dedicated OCR + rules engines that take months to maintain.

8. Internal analytics and BI assistant

Natural-language SQL, dashboard explanations, anomaly summaries. Works against your warehouse without exposing schema or data to an external API. Pairs well with embeddings of your data dictionary.

The single most useful skill for picking LLM hardware is converting "I want to run model X" into "I need this much GPU memory." Three numbers drive it:

1. Parameter count — the headline number in the model name (8B, 27B, 70B, 235B, 671B). For Mixture-of-Experts models like Llama 4 or DeepSeek V4, the total parameter count must fit in memory, even though only the smaller active subset runs per token.
2. Quantization — the precision at which weights are stored. FP16 is the native size; Q8 cuts it in half; Q4_K_M cuts it by roughly 4× with minimal quality loss for most tasks. Going below Q4 starts to degrade reasoning noticeably.
3. Context length / KV cache — every token of context consumes additional memory. At 32k context you need an extra few GB on top of the model; at 128k or 1M, substantially more.

A practical sizing table for the models most teams ask about, including safety margin for KV cache at a typical 16k–32k context window:

Numbers are rounded for planning. Add ~10–20% headroom for KV cache, batching, and inference-engine overhead (vLLM, TGI, llama.cpp, TensorRT-LLM). For Mixture-of-Experts models, the total parameter count must fit in memory even though only the smaller active subset runs per token. Long-context windows (256k+) significantly increase KV cache and may require more memory than the model weights themselves.

Inference vs fine-tuning — different hardware

Inference (running a trained model) is memory-bound: you need enough VRAM to hold weights + KV cache, and FLOPS to push tokens out fast enough. Fine-tuning (teaching a model new behavior on your own data) needs roughly 3–4× more memory than inference of the same model, because optimizer states and gradients have to live alongside the weights. If your roadmap includes fine-tuning a 70B model, plan for 4–8× H100/H200 rather than a single GPU. For LoRA / QLoRA fine-tuning of smaller models, a single high-VRAM workstation card (RTX 6000 Ada or A6000) is enough.

Below are five build tiers we configure most often for local-LLM workloads, with concrete platforms from our refurbished server and workstation catalogs. Every platform is built to order — pick CPU, RAM, storage, GPU, and warranty — and ships with a 1- to 5-year warranty.

Tier 1 — Entry workstation (single user, ≤9B models, embeddings, Whisper)

What it runs: Gemma 4 E2B/E4B, Qwen 3.5 2B/4B/9B, embedding models, Whisper transcription, image generation. Comfortable for one developer or a small RAG demo.

Platform: Lenovo ThinkStation P5 or Dell Precision 3650 with a single 16–24 GB GPU (RTX A4000 / A4500 / 4090).

Typical config: 1× Xeon W or i9, 64–128 GB DDR5, 1× 1 TB NVMe, 1× GPU 16–24 GB VRAM.

Tier 2 — Pro workstation (30B–31B models at Q4, small team)

What it runs: Gemma 4 31B Dense, Qwen3-Coder 30B-A3B (MoE), Qwen 3.5 35B-A3B at Q4 with room for context. Suitable for a code-assistant server for a small engineering team, an internal RAG service, or a single-tenant analytics assistant.

Platform: Dell Precision 7875 (AMD Threadripper Pro, up to 2 TB DDR5 RDIMM, multiple double-width GPUs) — starting around $5,500 refurbished, configurable up to Threadripper Pro 7985WX.

Typical config: Threadripper Pro 7945WX or 7985WX, 256–512 GB DDR5 RDIMM, 2× 2 TB NVMe RAID, 1× RTX 6000 Ada 48 GB or A6000 48 GB.

Alternative: HP Z8 G4 dual-Xeon, up to 1.5 TB DDR4, supports 3× double-width GPUs — strong value when you want multiple mid-range cards instead of one premium card.

Tier 3 — Single-GPU inference server (departmental, 100B-class MoE at Q4)

What it runs: Llama 4 Scout (Q4 on 80 GB card), Qwen 3.5 35B-A3B MoE, Mistral Medium 3.5 (Q4 on 80 GB card), Qwen3-Coder 30B-A3B at full precision — with serving for tens of concurrent users via vLLM or TGI. The sweet spot for a "company-wide LLM" without going to multi-GPU racks.

Platform: Dell PowerEdge R760 (Intel Xeon Scalable, up to 8 TB DDR5, GPU-enabled risers) or Dell PowerEdge R7625 (AMD EPYC Genoa, current generation).

Typical config: 2× Xeon Gold / EPYC, 512 GB – 1 TB DDR5, 2× 3.84 TB NVMe RAID, 1× A100 80 GB or 1× H100 80 GB for 100B-class MoE; 1× L40S 48 GB for dense models up to 32–40B. Redundant power, iDRAC / iLO out-of-band management, rack-ready. (For a deep look at how we approach a custom 1U server build, see our buyer's guide to the Dell PowerEdge R640.)

Tier 4 — Multi-GPU server (large MoE at Q4, dense 70B–128B at FP16, batch serving)

What it runs: Llama 4 Maverick at Q4 production scale, Qwen 3.5 397B-A17B at Q4, DeepSeek V4 Flash at Q4–Q8, Mistral Medium 3.5 at FP16, dense 70B models with hundreds of concurrent requests, image generation at speed.

Platform: Dell PowerEdge C4140 — a purpose-built 1U server explicitly engineered for AI, deep learning, and HPC, configurable with up to 4× double-width GPUs in a single chassis. Also: Dell PowerEdge R7525 (2U, AMD EPYC, 24-bay SFF options) for workloads that need GPU + heavy local storage for vector stores.

Typical config: 2× EPYC or Xeon, 1–2 TB RAM, NVLink-bridged GPU pairs, 4× A100 80 GB or 4× H100 80 GB for 400B-class MoE; 4× L40S 48 GB for high-throughput dense 70B–128B serving; NVMe RAID for hot data.

Tier 5 — Fine-tuning and training rig (8× H100 / H200, DeepSeek V4 Pro serving)

What it runs: LoRA / QLoRA fine-tuning of 70B–400B MoE models, full fine-tuning of mid-sized dense models (up to ~70B with ZeRO-3 / FSDP), serving 1T+ parameter MoE giants like DeepSeek V4 Pro, training proprietary models from scratch on focused corpora.

Platform: 8-GPU SXM platforms (HGX H100, HGX H200 class) — built to order via our quote desk. Refurbished and certified pre-owned H100/H200 inventory is available at a fraction of new-list pricing, with the same warranty options as our standard server lineup.

Typical config: 8× H100 80 GB SXM (640 GB pooled VRAM) or 8× H200 141 GB SXM (1.13 TB pooled VRAM) with NVLink, 2 TB DDR5, 30+ TB NVMe scratch, 200/400 GbE for multi-node scaling.

You do not always need a GPU. Modern Intel Xeon (Sapphire Rapids / Emerald Rapids with AMX instructions) and AMD EPYC Genoa / Bergamo can run 7B–13B models at usable speeds (8–25 tokens/second) with plenty of RAM and the right inference engine (llama.cpp, IPEX-LLM).

For workloads like background batch summarization, classification, or low-volume support drafting, a CPU-only server is dramatically cheaper than a GPU build. Platforms such as the PowerEdge R760 or R7625 with 256–512 GB RAM are the natural choice.

This is where most decisions actually get made. A realistic 12-month scenario for a mid-sized team running a coding assistant, a RAG service, and a support copilot — call it 50 million input tokens and 10 million output tokens per day across the organization:

Even on the conservative end, a Tier 3 server pays for itself in 3–6 months at this workload, and a Tier 4 server in 8–14 months. After that, every additional use case — a new RAG corpus, a new copilot, a new agent — runs at zero marginal cost. Cloud billing keeps climbing. For real-world numbers on what refurbished enterprise hardware saves over a multi-year horizon, see our case study on cost savings achieved through refurbished IT equipment.

For regulated industries, cost is often the secondary argument — compliance is the primary one. A local LLM means:

• Healthcare (HIPAA, HITECH): Protected Health Information never leaves your network. No Business Associate Agreement to negotiate with a model vendor, no audit-trail gaps when an API provider changes terms.
• Legal: Attorney-client privilege is preserved. Drafts, memos, discovery materials, and client communications stay inside the firm's perimeter.
• Financial services (SOX, GLBA, PCI-DSS scope): Material non-public information, customer financial data, and trade strategies stay on-prem. No exposure to cloud-vendor breaches.
• Defense and government (ITAR, CUI): Controlled technical data and CUI requirements are dramatically easier to satisfy when inference runs in your existing classified or controlled environment.
• EU operations (GDPR): No cross-border transfer concerns, no Schrems-II-style legal risk, full data minimization control.

The most-overlooked variable in LLM hardware planning is the room the server is going into. A four-GPU AI server draws 2,000–3,500 W under load, runs at server-room noise levels (60–80 dB), and dissipates that wattage as heat into the room. Practical guidance:

• Office tower or under-desk workstation: single GPU, up to ~1,000 W total, acoustically tolerable. Tier 1 and most Tier 2 builds fit here.
• Server closet or dedicated room with AC: single rackmount GPU server, redundant power, requires real cooling and a 20–30 A circuit. Tier 3 builds.
• Server room or colocation cabinet: multi-GPU 1U/2U, 208 V power, structured cooling. Tier 4 builds and above.
• Form factor matters: Threadripper Pro towers (Precision 7875, HP Z8) are office-friendly. C4140 and R7625 belong in a rack.

We help spec power, rack PDU, and airflow when scoping a quote — tell us where it is going and we will flag anything that does not fit.

If you have been following our coverage of the 2026 memory market — What Every IT Buyer Needs to Know Right Now and the earlier DDR4 vs DDR5 pricing analysis — the connection here is direct. The same AI buildout that produced Llama 4, Qwen 3.5, and DeepSeek V4 also caused Samsung, SK Hynix, and Micron to reallocate manufacturing capacity from standard DDR5 toward HBM for AI accelerators. DRAM contract prices are up 90–95% quarter-over-quarter. NAND/SSD is up 246% year-over-year.

For anyone building a local LLM server, this matters a lot — these systems are RAM-hungry by design. A Tier 3 inference server typically wants 512 GB to 1 TB of DDR5; a Tier 4 multi-GPU build often goes to 2 TB. At current spot pricing, RAM alone on a new-build server can equal the cost of an entire refurbished platform.

Refurbished enterprise hardware sidesteps the shortage entirely. The memory is already installed, tested, and warrantied. You are not bidding against hyperscalers for scarce DIMMs or sitting in a 6–12 month backorder queue.

The market for AI hardware has created a clear arbitrage opportunity: enterprise GPUs (A100, H100, L40S, A6000, RTX 6000 Ada) lose 50–70% of their list price within 18–24 months of deployment as hyperscalers refresh fleets. The silicon itself is built for tens of years of continuous operation. We test, recertify, and warranty refurbished servers, workstations, and GPUs so that you get the same hardware at a fraction of the price. (If you have lingering doubts about used enterprise gear, our deep dive into the 5 misconceptions about refurbished computers and the top 5 benefits of refurbished servers for the data center address every objection we have ever heard.)

Browse our current inventory: refurbished servers, refurbished workstations, refurbished GPUs. For enterprise GPUs not listed publicly (A100 / H100 / H200 / L40S / A6000 / RTX 6000 Ada), contact our quote desk — we keep inventory and typically respond within one business day.

Hardware is half the story; the inference engine you run on it determines real-world tokens per second. A quick orientation:

• Ollama / LM Studio — easiest path to a working setup, GGUF-based, great for workstation-class single-user deployments.
• llama.cpp — the engine under Ollama and LM Studio; runs anywhere, including CPU-only, with GGUF quantizations.
• vLLM — the standard for high-throughput multi-user serving on NVIDIA GPUs; PagedAttention dramatically improves throughput.
• Hugging Face TGI — production serving with batching, streaming, and tool integration.
• TensorRT-LLM — peak performance on NVIDIA H100/H200 when you are willing to do the build work.
• SGLang — strong for agentic and tool-using workloads with structured output.

For most business teams, vLLM behind a small FastAPI gateway, with Ollama for developer-machine prototyping, covers 95% of needs.

Can I run Llama 4 on a single GPU?

Llama 4 Scout (109B total MoE, 17B active) fits on a single 80 GB H100 or A100 at Q4 without offload, or on 2× 48 GB cards (A6000 / L40S / RTX 6000 Ada). Llama 4 Maverick (400B total MoE) needs at least 4× H100 80 GB or 4× A100 80 GB for Q4 inference. For smaller workloads, dense models in the 30B class run comfortably on a single 24–48 GB card.

What is the cheapest server that runs a 100B-class model locally?

A refurbished workstation like the Dell Precision 7875 with a single 80 GB GPU (H100 or A100) will run Llama 4 Scout (109B MoE) or Mistral Medium 3.5 (128B dense) at Q4 with full responsiveness for one to a few users. Total build is typically in the $15,000–$25,000 range depending on the GPU. For dense models up to ~32B, a 48 GB card (RTX 6000 Ada or A6000) in the same workstation works and brings the total well under $12,000. For multi-user serving, step up to a rackmount Tier 3 server.

Do I need an NVIDIA GPU, or can I use AMD?

NVIDIA is the practical default in 2026 because the inference-engine ecosystem (vLLM, TGI, TensorRT-LLM, FlashAttention) is most mature on CUDA. AMD MI300X / MI325X hardware is excellent and we can quote it, but expect more engineering work to get the same throughput. For most teams, NVIDIA is the lower-friction choice.

Is refurbished enterprise hardware reliable for 24/7 inference?

Yes. Enterprise servers and GPUs are designed for continuous duty over many years. Every system we ship is load-tested, recertified, and covered by a warranty (1–5 years). Failure rates on properly refurbished enterprise hardware are indistinguishable from new for the first several years of additional service.

Can I start with a workstation and move to a rack server later?

Absolutely, and we recommend it for most teams. A Tier 2 Precision 7875 or HP Z8 G4 with a 48 GB GPU is a great way to prove out a workload, build the integration, and measure real usage. Once you know what you need at scale, the same models and inference code move directly to a Tier 3 or Tier 4 server.

How much VRAM do I need to run DeepSeek V4?

DeepSeek V4 Pro (1.6T MoE, ~49B active) requires approximately 800 GB at Q4_K_M and 1.6 TB at Q8 — typically an 8× H200 141 GB platform. The smaller DeepSeek V4 Flash (284B MoE, 13B active) needs ~150 GB at Q4 and fits on 2× A100 80 GB or 4× L40S 48 GB.

Does running LLMs locally help with HIPAA compliance?

Yes. On-premise LLM inference keeps Protected Health Information inside your network, eliminating the need for Business Associate Agreements with model vendors and removing the audit-trail gaps that arise when a third-party API changes terms or retention policies. This is one of the most common reasons healthcare customers move from cloud APIs to local servers.

How do you handle GPUs that are not listed on the website?

Enterprise GPUs (A100, H100, H200, L40S, RTX 6000 Ada, A6000, MI300X) are sold through our quote desk because pricing and availability move quickly. We typically respond within one business day with current stock and lead time.

What about cooling and noise in an office?

Workstations (Precision 7875, HP Z8, ThinkStation P5) are designed for office acoustic levels and standard 120 V power. Rackmount AI servers are not — plan on a server closet or colocation. We will flag this on every quote.

Can you help with deployment, not just hardware?

Our focus is the hardware, but we maintain a short list of partners for inference-stack deployment (vLLM, RAG pipelines, fine-tuning) and we are happy to make introductions when scoping a system.

If you are upgrading from older Tesla V100, A100 40GB, or first-generation H100 hardware — or retiring enterprise servers and workstations of any kind — we are actively buying across the board. That hardware has real value in today's market, especially GPUs. Get in touch and we will quote a fair price, handle NIST 800-88 data sanitization, and arrange pickup.

• What Every IT Buyer Needs to Know Right Now — the 2026 memory and storage shortage, why it is structural, and how to plan procurement around it.
• How to Use an AI Server for Your Business — the broader business case for on-prem AI infrastructure, from data preparation to model serving.
• Support Your AI Stack With These Hardware Upgrades — targeted upgrades (networking, RAM, OS) for teams already running AI workloads.
• Guide to Buying a Custom Dell PowerEdge R640 1U NVMe Server — how to spec a 1U rackmount inference platform component by component.
• Case Study: Cost Savings Through Refurbished IT Equipment — real-world numbers on what refurbished enterprise hardware saves over a multi-year horizon.
• Top 5 Benefits of Choosing Refurbished Servers — the operational, financial, and sustainability case for refurbished data-center hardware.
• PC Prices in 2026: Why Buying Now Beats Waiting — the macro picture on workstation pricing heading into 2026 and beyond.
• Why CAD Professionals Need Purpose-Built Workstations — how to think about workstation specs for compute-heavy work (with direct parallels to LLM workloads).

Sources

• Meta AI, Llama 4 Scout and Maverick model cards, 2026
• Alibaba Cloud, Qwen 3.5 technical report, 2026
• DeepSeek, DeepSeek V4 Pro and Flash release notes, 2026
• Google DeepMind, Gemma 4 model documentation, 2026
• Hugging Face, open LLM leaderboards and inference benchmarks, Q1–Q2 2026
• vLLM, TGI, and TensorRT-LLM official documentation
• TrendForce, DRAM & NAND Price Outlook, Q1 2026