null
Server & Workstation RAM at Wholesale Volume Pre-tested Ships in 1–2 business days
Best Local AI Models 2026 by VRAM, and What to Run Them On

Best Local AI Models 2026 by VRAM, and What to Run Them On

Posted by Konstantin Protasov, PCSP on Nov 4th 2026

The best local AI model in November 2026 is decided by a number you already know: how much memory your graphics card has. On 16 GB the answer changed this year. The strongest models that fit are mixtures of experts, which read only a few billion parameters per token, and the 2026 releases from Qwen and Google keep a growing cache on one layer in four or six, so long context got cheap. The tools hide both: Ollama gives a card under 24 GiB a 4k context, and only llama.cpp can park a model’s experts in system RAM.

We sell used workstations with 8 and 16 GB NVIDIA cards, so read this with that in mind. Model facts come from each model card and config.json on Hugging Face, file sizes from the publishers’ file lists, all read November 1, 2026; memory and speed figures are our arithmetic, ceilings rather than benchmarks. Build tiers and cost of ownership are in our local LLM hardware guide, the apps in LM Studio vs Ollama.

The short version:

  • 8 GB: Qwen3.5-9B (February 2026, Apache 2.0, image input built in) is a 5.68 GB Q4_K_M file plus a 0.92 GB vision projector, 7.7 GB with 32k of context.
  • 16 GB: a mixture of experts beats a dense model. OpenAI’s gpt-oss-20b (12.11 GB, 3.6B active parameters) holds its full 131k context in 15.35 GB; the best dense model, Qwen3.8-27B, fits only at 3-bit.
  • Context is now a per-model number. Qwen3.6-35B-A3B costs 20 KiB of cache per token, Mistral’s Devstral Small 2 160 KiB: at 128k, 2.68 against 21.47 GB.
  • System RAM is part of the budget, in llama.cpp. Its --n-cpu-moe and default --fit keep a model’s experts in RAM; Ollama has no such option.
  • gpt-oss-120b needs 68.2 GB at 128k (a 63.39 GB file plus 4.83 GB of cache): an 80 GB GPU, a 128 GB Mac or DGX Spark, or a 16 GB card plus 128 GB of RAM.
  • Prices as of November 4, 2026: a Lenovo P520 with a Quadro P4000 8 GB is $785.93; an HP Z6 G4 with a Quadro RTX 5000 16 GB $1,973.98; our HP Z8 G4 configurator with two Quadro P5000s (32 GB) and 192 GB of RAM $3,114.85.
  • When used is the wrong answer: a Mac Studio with 128 GB at 614 GB/s is $5,399 and holds models no card we sell can; for occasional use an API lists gpt-oss-120b at $0.60 per million output tokens.

What we do not sell:

  • No card with 24 GB or more in depth: one GeForce RTX 3090 24 GB build, one RTX A4500 20 GB and one RTX A6000 48 GB configurator option, a single unit each.
  • No Ada or Blackwell cards, no GeForce RTX 40 or 50 series.
  • No datacenter GPUs on the shelf: those are configured and sourced to order, and our GPU servers are sold by quote.
  • No Apple Silicon Macs.

How Much Memory a Local AI Model Needs: File, Cache, Headroom

The weights are the file, not the parameter count. Qwen3.8-27B has 27.78 billion parameters; half a byte each would be 13.89 GB. The four Q4_K_M files on Hugging Face are 16.46 GB (unsloth), 16.81 (lmstudio-community), 17.44 (bartowski) and 18.97 GB (ggml-org): 4.7 to 5.5 bits per weight, because the “M” mixes keep some tensors at higher precision. Q4_K_M is a label; the file list is the number. Conversely, unsloth’s gpt-oss-20b Q4_K_M is 11.62 GB, only 4% below its 12.11 GB Q8_0, because OpenAI released the experts already in 4-bit MXFP4.

The cache grows with context, and the 2026 models changed how fast. Bytes per token = 2 × layers that keep a cache × KV heads × head dimension × 2 bytes at f16, the formula our LM Studio and inference-node articles use. Older models cache every layer. Qwen3.5, 3.6 and 3.8 run three Gated DeltaNet layers for each attention layer (the “Hidden Layout” on the 3.6 and 3.8 cards, layer_types in Qwen3.5’s config), and DeltaNet keeps a fixed-size state instead; llama.cpp builds a hybrid memory for these architectures. Gemma 4 and gpt-oss mix full layers with sliding-window layers, whose cache llama.cpp caps at the window plus one batch (llama-kv-cache-iswa.cpp). Both are per slot, and llama-server defaults to four; set -np 1.

A worked example. Qwen3.6-35B-A3B’s card gives 40 layers as 10 × (3 DeltaNet + 1 attention), so 10 layers cache, and its config.json gives 2 KV heads of dimension 256: 2 × 10 × 2 × 256 × 2 = 20,480 bytes, 20 KiB per token, 2.68 GB at 128k. Devstral Small 2, a classic design, caches 40 layers × 8 heads × 128: 160 KiB per token, 21.47 GB at 128k. A user’s log in llama.cpp discussion #31177 (October 26, 2026) puts Qwen3.6 at “about 10,880 bytes” per token with an 8-bit cache, which is our 20 KiB at q8_0’s 34 bytes per 32 values.

A mixture of experts is stored whole and read in part. gpt-oss-20b is “21B parameters with 3.6B active”. All 21 billion must sit in VRAM or system RAM; the 3.6 billion read per token set the speed. So it is quick on a slow card, and its experts can live in system RAM.

Headroom. Leave 0.5 to 1.5 GB for compute buffers and the CUDA context; llama.cpp’s automatic fitting keeps a 1 GiB margin per device.

Model File (publisher) Cache per token, f16 Total at 32k Total at 128k One 16 GB card?
Qwen3.5-9BQ4_K_M 5.68 GB (unsloth)32 KiB6.80 GB10.03 GBYes, any context
Gemma 4 12BQ4_K_M 7.12 GB (unsloth)16 KiB + 0.50 GB window8.16 GB9.77 GBYes
gpt-oss-20bMXFP4 12.11 GB (ggml-org)24 KiB12.93 GB15.35 GBYes; full context is tight
Gemma 4 26B-A4BUD-IQ4_XS 13.60 GB (unsloth)20 KiB + 0.32 GB window14.58 GB16.60 GBTo about 32k
Qwen3.8-27BUD-IQ3_XXS 10.93 GB (unsloth)64 KiB13.23 GB19.68 GBTo about 32k
Qwen3.8-27BQ4_K_M 16.81 GB (lmstudio-community)64 KiB19.11 GB25.55 GBNo: two cards
Qwen3.6-35B-A3BUD-IQ4_XS 17.73 GB (unsloth)20 KiB18.46 GB20.48 GBOnly with experts in RAM
Devstral Small 2 24BQ4_K_M 14.33 GB (unsloth)160 KiB19.70 GB35.81 GB4k–8k at most (15.67 GB at 8k)
Gemma 4 31BQ4_K_M 18.32 GB (unsloth)80 KiB + 1.26 GB window22.27 GB30.32 GBNo

Hugging Face file lists and each model’s config.json, read November 1, 2026. Our arithmetic: f16 cache plus the fixed DeltaNet state (0.05–0.16 GB, one slot) or capped sliding-window cache; projectors, compute buffers and CUDA context excluded. GB = 109 bytes; 128k = 131,072 tokens, gpt-oss-20b’s maximum. A planning ceiling, not a measurement.

Dumbbell chart of model file plus f16 cache for one slot at 32k and 128k tokens: Qwen3.5-9B 6.8 to 10.0 GB, Gemma 4 12B 8.2 to 9.8 GB and gpt-oss-20b 12.9 to 15.35 GB stay under 16 GB; Qwen3.8-27B at 3-bit reaches 19.7 GB, Qwen3.6-35B-A3B 20.5 GB and Devstral Small 2 24B 35.8 GB at 128k.

File plus f16 cache, one slot, at 32k and 128k tokens. Gemma 4 12B barely grows with context; Devstral Small 2, which caches every layer, outgrows two cards. Our arithmetic from Hugging Face files and config.json, November 1, 2026.

The Best Local AI Models by VRAM: November 2026 Shortlist

Every model here is Apache 2.0 unless the row says otherwise.

Memory Job Model, released File to download (publisher) Fits at, our arithmetic
Any, CPU is fineEmbeddings for search over your documentsQwen3-Embedding-0.6B, June 2025Q8_0 0.64 GB (Qwen)Trivial
8 GBChat, reasoning, image inQwen3.5-9B, February 2026Q4_K_M 5.68 GB + 0.92 GB vision (unsloth)7.7 GB at 32k with vision
8 GBChat, image and audio inGemma 4 E4B, April 2026Q4_K_M 4.98 GB + 0.99 GB (unsloth)128K native; cache small, not computed
12 GBChat, image and audio inGemma 4 12B, June 2026Q4_K_M 7.12 GB + 0.18 GB (unsloth)8.3 GB at 32k with vision
16 GBReasoning, tool usegpt-oss-20b, August 2025 (Apache 2.0 + OpenAI usage policy)MXFP4 12.11 GB (ggml-org)12.9 GB at 32k; 15.35 GB at its full 131k
16 GBWriting, image inGemma 4 26B-A4B, April 2026UD-IQ4_XS 13.60 GB (unsloth)14.6 GB at 32k, text only
16 GBBest dense quality that fitsQwen3.8-27B, August 2026UD-IQ3_XXS 10.93 GB (unsloth)13.2 GB at 32k
16 GB + 64 GB RAMCoding, agentsQwen3.6-35B-A3B, April 2026UD-Q4_K_M 22.13 GB (unsloth), experts partly in RAM, llama.cpp onlyCache 2.68 GB at 128k
2 × 16 GB or 24 GBCoding, long context, image inQwen3.8-27BQ4_K_M 16.81 GB (lmstudio-community)21.3 GB at 64k; 25.6 GB at 128k
24 GBWriting, image inGemma 4 31B, April 2026Q4_K_M 18.32 GB + 1.20 GB (unsloth)22.1 GB at 16k with vision
80 GB, or 16 GB + 128 GB RAMReasoning, tool usegpt-oss-120b, August 2025 (Apache 2.0 + usage policy)MXFP4 63.39 GB (ggml-org)68.2 GB at 128k
96–128 GBBiggest that runs locallyQwen3.5-122B-A10B, February 2026; Mistral Small 4, March 2026MXFP4_MOE 74.66 / 71.81 GB (unsloth)About 76.4 GB at 64k (Qwen)

Model cards and file lists read November 1, 2026; release month from each repository’s first public commit. “Fits at” = file + f16 cache + fixed state for one slot (-np 1; llama-server defaults to four), plus the projector where noted; add 0.5–1.5 GB headroom. Our arithmetic, a planning ceiling.

Licences that are not plain Apache 2.0

Meta’s Llama 4 (released April 5, 2025; Scout is 109B total) needs a separate licence above 700 million monthly active users, and its use policy withholds rights to the multimodal models from people and companies based in the EU. Mistral Medium 3.5’s modified MIT licence excludes companies with more than $20 million of monthly revenue. Qwen licenses per model: Qwen3.5, Qwen3.6 and Qwen3.8-27B are Apache 2.0, but Qwen3.8-Flash-Next’s community licence asks model-as-a-service and AI coding or office-assistant businesses for a separate licence, internal use excepted.

Names that mislead

NVIDIA’s Nemotron 3.5 Bolt-74B-A9B-24GB is a 22.02 GB file plus a 1.73 GB projector, and its own GGUF card (October 15, 2026) names RTX cards with 32 GB or more and gives 28 and 32 GB recipes. Mistral Small 4 has 119 billion parameters. Gemma 4 E4B is “4.5B effective (8B with embeddings)” and downloads at 4.98 GB. Ollama’s deepseek-r1 tags from 1.5b to 70b are distilled Qwen and Llama models; R1 itself is the 671b tag, 404 GB. Too big for this list: DeepSeek V4.1-Flash (552B backbone), Kimi K3 (2.8T), GLM-5.3-Flash (320B) and Mistral Large 4 (about 1T).

Best Local AI Models for 8 GB and 12 GB Cards

Qwen3.5-9B is the 8 GB default. It reads images and has a 262,144-token native context. Its config.json marks 24 of 32 layers as linear attention, so only 8 keep a cache: 32 KiB per token. The 5.68 GB file (unsloth) runs 32k in 6.80 GB, or 7.72 GB with the vision projector; on an 8 GB card with images, 16k (7.19 GB) leaves headroom. The 4B, at 2.74 GB, is the step down.

Gemma 4 E4B adds audio. Google’s April 2026 small model takes text, images and audio with a 128K context. At 4.98 GB plus a 0.99 GB projector it is barely smaller on disk than Qwen3.5-9B: its 8 billion parameters include per-layer embeddings. We did not compute its cache (18 of its 42 layers reuse another layer’s, which breaks the formula); it is small either way.

12 GB: Gemma 4 12B. Added in June 2026, the 11.95B “Unified” model takes text, images and audio and has a 256K context; its 8 global layers carry one KV head each, 16 KiB per token. The 7.12 GB Q4_K_M runs 32k in 8.3 GB with vision; Google’s own QAT Q4_0 build is 6.98 GB.

Ollama run qwen3.5: there is no “small” tag

Ollama’s qwen3.5 tag list (November 1, 2026) has no tag called small. The small sizes are qwen3.5:0.8b (1.2 GB), :2b (2.7 GB), :4b (3.3 GB) and :9b (6.6 GB, also latest). Each includes the vision projector, and the 2b default is not its 4-bit build (2b-q4_K_M is 1.9 GB). All list 256K, but Ollama’s context-length doc gives machines with under 24 GiB of VRAM 4k by default and wants “at least 64000 tokens” for agents and coding tools. On 8 GB, start it with OLLAMA_CONTEXT_LENGTH=16384 ollama serve and check that ollama ps still reads 100% GPU.

Two 8 GB cards and a mixture of experts. The #31177 user runs Qwen3.6-35B-A3B at 3-bit on two 8 GB AMD cards (Vulkan, five layers’ experts in RAM) and reports “about 38.5 tok/s for one user”: a community measurement, command posted, not our cards.

Our 8 and 11 GB machines, as of November 4, 2026. The Lenovo P520 with a Xeon W-2133, 32 GB, a 1 TB NVMe drive and a Quadro P4000 8 GB is $785.93, 110 in a pool shared with Xeon W-2135 builds with 64 or 128 GB of RAM, for experts in memory, at $1,047.87 and $1,495.87. In our workstation configurators a P4000 adds $159.99, a GeForce RTX 3070 8 GB $274.99 (Dell T7920 and T7820 and Lenovo P620 only) and an RTX A1000 8 GB $399.99. The A1000 is Ampere and draws 50 W, but its 192 GB/s is below the P4000’s 243 GB/s (our Quadro generations table): for Qwen3.5-9B, a ceiling of about 34 against 43 tokens a second. For 11 GB, an HP Z4 G4 with a Xeon W-2145, 64 GB and a GeForce RTX 2080 Ti 11 GB is $1,643.89, 51 in one pool; it holds Gemma 4 12B at 32k with room to spare, but not gpt-oss-20b’s 12.11 GB.

Best LLM to Run Locally on 16 GB: a Mixture of Experts

gpt-oss-20b is the reasoning pick. OpenAI’s August 2025 model runs “within 16GB of memory”, per its card. The native release on ggml-org is a 12.11 GB MXFP4 file. Half of its 24 layers use a 128-token sliding window, so the cache is 24 KiB per token and the full 131,072-token context totals 15.35 GB. That is tight; 64k, at 13.74 GB, is the comfortable setting.

Gemma 4 26B-A4B for writing. Google’s April 2026 mixture of experts has 25.2 billion parameters, 3.8 billion active. unsloth’s UD-IQ4_XS is 13.60 GB and runs 32k in 14.58 GB, text only; its 1.19 GB vision projector does not fit beside it. For images on 16 GB, Gemma 4 12B at Q8_0 (12.67 GB) runs 32k with vision in 13.9 GB.

Qwen3.8-27B is the best dense model, at a cost. Qwen’s current open model (August 2026; image and video input; thinking on by default) needs 3-bit to fit: UD-IQ3_XXS is 10.93 GB, 13.23 GB at 32k. Its 4-bit UD-IQ4_XS (14.25 GB) fits only at 8k.

Speed: the only reproducible numbers are for Llama 2. llama.cpp’s CUDA scoreboard (discussion #15013, updated October 16, 2026) runs llama-bench on Llama 2 7B Q4_0 with every layer on one GPU. Users in the thread posted a desktop Quadro P5000 at 48.61 tokens a second of generation (build b7565, December 29, 2025) and a laptop Quadro RTX 5000 at 74.76 (May 25, 2026); there is no desktop RTX 5000 result. None of the 234 comments has a 2026 model on either card, so we give the ceiling: datasheet bandwidth divided by bytes read per token. The P5000 run landed at 65% of its ceiling.

File read per token Quadro P5000, 288 GB/s Quadro RTX 5000, 448 GB/s
Llama 2 7B Q4_0, 3.82 GB75 ceiling; 48.61 measured117 ceiling; 74.76 measured on a laptop
Qwen3.5-9B Q4_K_M, 5.68 GB5179
Gemma 4 12B Q4_K_M, 7.12 GB4063
Qwen3.8-27B UD-IQ3_XXS, 10.93 GB2641
Qwen3.8-27B UD-IQ4_XS, 14.25 GB2031

Tokens per second of generation. Ceilings = bandwidth from NVIDIA’s P5000 and Quadro RTX 5000 datasheets ÷ GB read per token (the whole file for a dense model): our arithmetic, not benchmarks. Measured: user posts in llama.cpp discussion #15013, tg128, flash attention off, different builds and hosts; the RTX 5000 run is a laptop. Read November 1, 2026.

A mixture of experts reads only its active weights, around 2 GB per token for the 3–4B-active models here, against all 11–16 GB for the dense 27B. That is why gpt-oss-20b and Gemma 4 26B-A4B are the better 16 GB picks, even on a P5000.

Paired bars of generation-speed ceilings, bandwidth divided by gigabytes read per token: Quadro P5000 at 288 GB/s from 75 tokens a second on Llama 2 7B down to 20 on Qwen3.8-27B UD-IQ4_XS, Quadro RTX 5000 at 448 GB/s from 117 down to 31, with the one measured desktop P5000 run at 48.61.

Ceilings from NVIDIA’s bandwidth figures and each file’s size, not benchmarks. The one desktop P5000 measurement, 48.61 tokens a second on Llama 2 7B, reached 65% of its ceiling; expect real runs well below these bars.

Our 16 GB machines, as of November 4, 2026. The lowest-priced is the HP Z6 G4 with two Xeon Gold 6136, 64 GB and a Quadro RTX 5000 16 GB at $1,973.98, 26 in stock. A Dell Precision T5820 with one Xeon W-2145 and the same card is $2,158.90, 72 in stock. The RTX 5000 is Turing: a laptop one posted about 1.5× the P5000’s result, and it is supported by vLLM and CUDA 13 builds (see below). The Pascal P5000 is the cheaper 16 GB: $349.99 as a loose card (nine in stock, 90-day parts warranty), or $344.99 as an option in our HP Z8 G4, Dell T7920 and T7820 and Lenovo P620 configurators, from a shared pool of 31. A Z8 G4 with two Xeon Gold 6140, 64 GB of DDR4-2666, a 1 TB NVMe drive and a P5000 comes to $1,745.94.

Check the context you actually got. llama.cpp’s automatic fitting shrinks context to make a model fit — its own example logs “context size reduced from 40960 to 4096” — and Ollama starts every card under 24 GiB at 4k. Read the load log, or set it yourself: -c 32768 in llama.cpp, OLLAMA_CONTEXT_LENGTH in Ollama. Qwen’s card for Qwen3.6 asks for “at least 128K tokens to preserve thinking capabilities”.

Build a 16 GB workstation with room for experts in RAM

HP Z8 G4 with two Xeon Gold 6140, a Quadro P5000 16 GB, a 1 TB NVMe drive and 192 GB of DDR4-2666 as twelve 16 GB modules, six per processor — $2,769.86 as of November 4, 2026. Add a second P5000 for 32 GB. One-year warranty covering parts, labor and shipping, extendable up to five years.

Configure an HP Z8 G4 Browse HP Z8 G4

Bigger Models on a 16 GB Card: Experts in System RAM

llama.cpp’s options (master, November 1, 2026) include --cpu-moe, to “keep all Mixture of Experts (MoE) weights in the CPU”, and --n-cpu-moe N for the first N layers; attention, shared weights and the cache stay on the card. With --fit, now on by default, it decides itself: the fit-params README shows it moving whole layers’ expert tensors to the CPU for an F16 Qwen3-30B-A3B needing 61,807 MiB on an RTX 4090. LM Studio accepts the same arguments.

Ollama has no such switch: the request for one, issue #11772, opened August 7, 2025, was still open on November 4, 2026; when VRAM runs out, Ollama moves whole layers to the CPU. For a model bigger than your card, use llama.cpp.

Speed follows the active bytes and the RAM. With experts in RAM, each generated token reads roughly its active weights from system memory; prompts instead copy the experts in use over PCIe (ggml-cuda.cu). One Xeon Scalable socket with all six DDR4-2666 channels filled peaks at 128 GB/s, per our LM Studio article’s arithmetic from Intel’s specifications. Divided by bytes per token, that is a ceiling of about 70 tokens a second for Qwen3.6-35B-A3B (about 1.8 GB active), about 47 for gpt-oss-120b (about 2.7 GB) and 7.6 for the dense Qwen3.8-27B at 16.81 GB. Users report less. In llama.cpp discussion #30333 (October 11, 2026), a CPU-only dual-Xeon box with sixteen DDR4-3200 channels, measured at about 158 GB/s (39% of paper), ran the dense Qwen3.8-27B at about 5.6–5.9 tokens a second and Qwen3.8-Flash-Next, a mixture of experts reading 5.70 GB per token at 2-bit, at about 11.3, with effective bandwidth around 40–55% of peak. On a second machine the same author saw default memory-mapping collapse prefill about tenfold, an 84 GB model in 128 GB of RAM; loading with -lm none fixed it.

A second socket adds bandwidth only if each thread reads its own socket’s memory; llama.cpp’s --numa options only “attempt” that.

Fill the channels. The build in the box above puts six of its twelve modules on each processor. Our Z8 G4 configurator’s 128 GB option, eight modules, is $511.96 less but leaves two of six channels per socket empty.

gpt-oss-120b memory requirements

OpenAI’s card gives gpt-oss-120b “117B parameters with 5.1B active” and says MXFP4 lets it “run on a single 80GB GPU”. The arithmetic, from files and config.json:

  • Weights: the ggml-org MXFP4 file is 63.39 GB; Ollama’s gpt-oss:120b is 65 GB, and unsloth’s Q4_K_M build 62.77 GB.
  • Cache: 36 layers, 18 of them full attention, 8 KV heads of dimension 64: 36 KiB per token, 1.21 GB at 32k and 4.83 GB at its 131,072 maximum. The 18 sliding-window layers add almost nothing.
  • Total: 64.6 GB at 32k, 68.2 GB at 128k, plus headroom.

That fits one 80 GB GPU, a 96 or 128 GB Mac, raising its GPU memory limit if needed (below), a 128 GB DGX Spark, or, in llama.cpp, a 16 GB card with 128 GB or more of system RAM. A 48 GB card still needs its experts in RAM, and a 64 GB machine is too small.

32 and 48 GB: Two 16 GB Cards, or One Big One

Two cards hold what one cannot. llama.cpp’s default split is by layer, “split layers and KV across GPUs (pipelined)”, so only activations cross PCIe and no bridge is needed. That matters because NVIDIA’s P5000 datasheet lists no NVLink. The Quadro RTX 5000 has it, but its datasheet adds that scaling to 32 GB over NVLink works only if the application supports it. Ollama loads a model on one card if it fits and spreads it otherwise, and two 16 GB cards move its default context to 32k. Capacity doubles, speed does not: with the default split, one user’s token passes through the cards in turn.

What 32 GB buys. Qwen3.8-27B at Q4_K_M (16.81 GB) with 128k of context in 25.6 GB, the best dense model at long context; Qwen3.6-35B-A3B at UD-Q4_K_M (22.13 GB) entirely on the cards at 128k in 24.9 GB, our pick for coding; or Gemma 4 31B (April 2026, 30.7B dense) with vision at 32k in 23.5 GB.

On a single 24 GB card the same models fit with less context: Qwen3.8-27B at 64k in 21.3 GB, Qwen3.6-35B-A3B at UD-IQ4_XS (17.73 GB) at 128k in 20.5 GB, Gemma 4 31B with vision at 16k in 22.1 GB. We have one 24 GB machine, a Lenovo P620 with a GeForce RTX 3090 at $5,827.36, and no 24 GB card on its own.

Our two-card builds, as of November 4, 2026. The Z8 G4 configurator takes a P5000 as primary and as secondary card: two P5000s, two Xeon Gold 6140, 192 GB and a 1 TB NVMe drive come to $3,114.85. Pre-built, a Dell Precision T7920 with two Xeon Gold 6148, 512 GB and two P5000s is $6,170.61, 15 in stock.

48 GB and up. On 48 GB, Qwen3.8-27B at Q8_0 (29.05 GB) runs its whole 262,144-token native context in 46.4 GB. The 120B-class mixtures, Qwen3.5-122B-A10B (74.66 GB at MXFP4_MOE, about 76.4 GB with 64k) and Mistral Small 4 (71.81 GB), need 96 to 128 GB. Our one RTX A6000 48 GB is a single configurator option at $4,499.00; in the Z8 G4 with two Gold 6140, 128 GB and 1 TB it comes to $6,411.91.

Embeddings, Vision and Speech: the Small Models That Make Local AI Useful

Search over your own documents needs an embedding model beside the chat model; these are small enough to share its card or run on the CPU.

  • Qwen3-Embedding (June 2025): the 0.6B is a 0.64 GB Q8_0 file from Qwen and qwen3-embedding:0.6b in Ollama (639 MB, 32K context); the 8B is 4.7 GB with 40K.
  • EmbeddingGemma 2 (October 2026, Apache 2.0, not gated): 740M parameters, text, images, video and audio in one 768-dimension space, 8,192 tokens of input. The 2025 EmbeddingGemma-300m stays under Google’s gated Gemma licence.
  • Older options: bge-m3 (1.2 GB, 8K, MIT) and nomic-embed-text:v1.5 (274 MB, 2K).

Vision is built in now. Qwen3.5, 3.6 and 3.8 and every Gemma 4 size take images natively, so a separate vision model is no longer needed. On Hugging Face the projector is a separate mmproj file you must download (0.18 to 1.20 GB above); Ollama’s tags include it, which is why qwen3.5:9b is 6.6 GB against a 5.68 GB file.

Speech to text. OpenAI’s whisper-large-v3-turbo (October 2024, MIT) is still its newest Whisper. Newer open options are Qwen’s Qwen3-ASR at 0.6B and 1.7B (January 2026, Apache 2.0) and IBM’s granite-speech-5.0-2b (October 2026), which ships a GGUF.

What Pascal and Turing Cards Can and Cannot Run in 2026

The Quadro P5000 is Pascal, compute capability 6.1; the Quadro RTX 5000 is Turing, 7.5. The difference shows in four places (llama.cpp master, November 1, 2026):

  • Formats: both run every GGUF, including the new ones. Both always take llama.cpp’s int8 matrix path, which covers every K- and IQ-quant plus MXFP4 and NVFP4 (mmq.cu): Turing on int8 tensor cores, Pascal through byte-wise dot products; prompt processing runs 805 against 1,841 tokens a second for the P5000 and the laptop RTX 5000 on Llama 2. Native FP4 instructions are used only on Blackwell; nobody has published how fast MXFP4 runs on a P5000.
  • Flash attention: yes, without tensor cores. llama.cpp falls back to a generic tile kernel and handles the 512-wide heads of Gemma 4’s global layers (fattn.cu), so -fa on and a quantised V cache work on a P5000. Pascal’s FP16 math is slow, and common.cuh treats it so.
  • Builds: CUDA 12 for Pascal. llama.cpp build b12030 (November 1, 2026) ships CUDA 12.8 Linux and 12.4 Windows packages that carry Pascal code as PTX, and CUDA 13.4 packages that start at Turing. R580 is NVIDIA’s last driver branch for Pascal, with security updates through October 2028, as our Quadro generations article details.
  • Engines: vLLM starts at Turing. vLLM’s install guide requires “compute capability 7.5 or higher (e.g., T4, RTX20xx, A100…)”: the Quadro RTX 5000 and the RTX 2080 Ti qualify, the P5000 and P4000 do not. FP8 and NVFP4 checkpoints are made for engines like vLLM on newer cards, and NVIDIA’s own Nemotron 3.5 targets Ampere, Hopper and Blackwell, so on Pascal you take the GGUF build.

One gap we could not close: Ollama runs Qwen3.5 and newer on its own engine, and we have not verified its DeltaNet and flash-attention paths on a 6.1 card. llama.cpp’s are in its code; on a P5000, test Ollama with these models first.

When Used Hardware Is the Wrong Answer: a Mac, a New Card or an API

In depth, our shelf stops at 16 GB per card and 448 GB/s. Three alternatives beat it, each for a different reader.

Option Memory that holds the model Bandwidth Price
Quadro P5000 configurator option, for scale16 GB288 GB/s$344.99
Mac mini, M5 Pro 20-core GPU, 64 GB, 1 TB64 GB unified307 GB/s$3,199
Mac Studio, M5 Max 40-core GPU, 128 GB, 1 TB128 GB unified614 GB/s$5,399
Mac Studio, M5 Ultra 64-core GPU, 96 GB, 1 TB96 GB unified1.2 TB/s$5,499
NVIDIA DGX Spark128 GB unified273 GB/s$6,950 (4 TB)
GeForce RTX 5090, card only32 GB—$1,999 MSRP; $6,919 lowest US listing
GeForce RTX 4090, card only24 GB—$1,599 MSRP; $4,999 lowest US listing
Radeon AI PRO R9700, card only32 GB640 GB/s (press-reported)$1,299 launch price (press-reported)

Apple US store and spec pages (Mac mini, Mac Studio) and configured prices, November 4, 2026; DGX Spark from NVIDIA’s US store the same day (our inference-node article read $4,699 on September 20); RTX prices from Tom’s Hardware’s lowest US price index, page dated November 3, 2026, one listing each; R9700 price from AMD’s David McAfee as reported by Phoronix (October 23, 2025), bandwidth press-reported; our prices as of November 4, 2026. Cards need a host; the rest are whole machines.

A Mac for big models in one quiet box. A Mac Studio with 128 GB holds gpt-oss-120b, which no card we sell can, at 2.1 times a P5000’s bandwidth. But unified memory is not all GPU memory: Metal reports a recommendedMaxWorkingSetSize, which Apple calls “an approximation” and gives no fixed share for, and llama.cpp takes it as the GPU’s memory. The mlx-lm README raises the limit with sudo sysctl iogpu.wired_limit_mb=N, set above the model size and below the machine’s memory, on macOS 15 or later. And a 32 GB Mac mini M6, at 170 GB/s, has less bandwidth than one P5000.

A new card for speed. On llama.cpp’s Llama 2 scoreboard an RTX 5090 generates 290.02 tokens a second, six times a P5000. At the November 3 lowest listing it costs about 20 times our $349.99 loose P5000, or 5.7 times at MSRP; our GPU price article tracks why. AMD’s Radeon AI PRO R9700 brings 32 GB, and Ollama’s hardware page lists it among its ROCm cards. We found no Intel MSRP for the Arc Pro B60 24 GB; SPARKLE’s partner price was $799 in January 2026, as press-reported, and llama.cpp’s SYCL page names the Arc B580, not the B60. Any new NVIDIA card also brings the CUDA 13 and vLLM support Pascal lacks.

An API for occasional use. OpenRouter’s model list (November 4, 2026) prices gpt-oss-120b at $0.15 per million input tokens and $0.60 per million output, gpt-oss-20b at $0.030 and $0.130, and Qwen3.8-27B at $0.15 and $1.875. DeepSeek’s own API charges $0.15 and $0.60 off-peak, $0.30 and $1.20 at peak, for V4.1-Flash. At $0.60, the $349.99 of one loose P5000 buys about 580 million output tokens of a model that card cannot hold. Local wins on privacy, offline use and a fixed cost; our inference-node article works out the break-even.

Tell us which model you want to run

We will size the card, the system RAM and the memory channels for it, and say so when a Mac or an API is the better answer. One 16 GB machine we have in depth is an HP Z6 G4 with two Xeon Gold 6136, 64 GB and a Quadro RTX 5000: From $1,973.98 · 26 in stock, refreshed nightly. One-year warranty covering parts, labor and shipping, extendable up to five years. Orders ship within 1–2 business days.

See this Z6 G4 Browse all workstations

The Bottom Line

The best local AI model in November 2026 is the best one your memory holds at a context you set yourself, and on 16 GB that is a mixture of experts. On 8 GB that is Qwen3.5-9B; on 12 GB, Gemma 4 12B; on 16 GB, gpt-oss-20b or Gemma 4 26B-A4B, and with 128 GB of RAM behind the card, gpt-oss-120b in llama.cpp; on 32 GB, the dense Qwen3.8-27B at 128k. As of November 4, 2026 that is a $785.93 P520 with a P4000, a $1,973.98 Z6 G4 with a Quadro RTX 5000, or a Z8 G4 at $2,769.86 with one P5000 and 192 GB, $3,114.85 with two.

If the model you need is bigger than 32 GB and you want it in one quiet box, a 128 GB Mac Studio at $5,399 is the better buy. If you need it twice a month, an API is cheaper than any hardware, and we would rather say so.

?

Best Local AI Models: FAQ

What are the best local AI models in 2026?

By memory, as of November 4, 2026: Qwen3.5-9B for 8 GB, Gemma 4 12B for 12 GB, OpenAI’s gpt-oss-20b and Google’s Gemma 4 26B-A4B for 16 GB, Qwen3.8-27B for 24 to 32 GB, and gpt-oss-120b for 80 GB or a 16 GB card with 128 GB of system RAM. All are Apache 2.0; gpt-oss adds OpenAI’s usage policy.

What is the best LLM to run locally on a 16 GB GPU?

gpt-oss-20b: a 12.11 GB MXFP4 file with 3.6 billion active parameters whose full 131,072-token context totals 15.35 GB by our arithmetic. For writing, Gemma 4 26B-A4B (13.60 GB) runs 32k in 14.58 GB. The best dense model, Qwen3.8-27B, fits only at 3-bit.

How much memory does gpt-oss-120b need?

About 68.2 GB at its full 131,072-token context: the 63.39 GB MXFP4 file from ggml-org plus 4.83 GB of f16 cache at 36 KiB per token, before compute buffers; 64.6 GB at 32k. That fits one 80 GB GPU, a 96 to 128 GB Mac, a DGX Spark, or, in llama.cpp, a 16 GB card with its experts in 128 GB or more of RAM. Ollama cannot put experts in RAM.

What are the small Qwen3.5 models in Ollama?

There is no tag called small. As of November 1, 2026, Ollama lists qwen3.5:0.8b (1.2 GB), qwen3.5:2b (2.7 GB), qwen3.5:4b (3.3 GB) and qwen3.5:9b (6.6 GB, the latest tag), each with vision included and up to 256K of context. Ollama starts any card under 24 GiB at a 4k context unless you set OLLAMA_CONTEXT_LENGTH.

Can I run an LLM locally without a GPU?

Yes, best with a mixture of experts. Speed is capped at memory bandwidth divided by bytes read per token: one Xeon socket with six DDR4-2666 channels peaks at 128 GB/s, a ceiling near 70 tokens a second for Qwen3.6-35B-A3B and 7.6 for a dense 27B. One user’s dual-Xeon box reached 15 to 21% of paper bandwidth.

Can a Quadro P5000 run 2026 models?

Yes, as GGUF files in llama.cpp: its int8 kernels cover every K and IQ quant plus MXFP4, so gpt-oss runs, and flash attention works without tensor cores. It cannot run vLLM, which needs compute capability 7.5, or CUDA 13 builds. One user’s P5000 posted 48.61 tokens a second on Llama 2 7B; we found no posted 2026-model result.

Is a Mac better than a used GPU workstation for local AI?

For models above 32 GB, often yes: a Mac Studio with 128 GB at 614 GB/s was $5,399 on November 4, 2026 and holds gpt-oss-120b, though not all unified memory is GPU memory by default. For models up to 16 GB, our HP Z6 G4 with a Quadro RTX 5000 was $1,973.98 the same day.