Used GPU Server Buying Guide 2026: L40S vs A100 vs RTX 4090

Three market forces have converged to make this the best used-GPU buying window since the technology went mainstream:

• The Blackwell refresh is feeding the secondary market. As hyperscalers, AI labs, and infrastructure providers refresh parts of their fleets for B200/B300 and GB200 systems, more A100, H100, and L40S hardware is entering secondary-market channels. Those cards are retired for density and power efficiency, not because they stopped working — they remain strong for inference and mid-size training.
• Prices are trending down. Used A100 80GB PCIe cards that sold for $15,000+ new now appear on the secondary market well below that — often in the $8,000–$12,000 range (SXM parts run lower but need a compatible HGX board), and pricing varies widely by form factor, condition, and warranty. Several analysts expect further softening through 2026 as Blackwell volume ramps. Note the exception: the L40S still holds most of its value — demand for the current inference favorite keeps used discounts modest — so the deepest 40–70% savings show up on A100/H100-class cards and complete platforms rather than on the L40S itself.
• The memory shortage makes new builds harder to budget. With DRAM and HBM prices up sharply and vendors diverting wafers to HBM for AI accelerators, a new server can cost as much in RAM alone as a fully configured refurbished platform — and new orders can wait months on memory allocation. Refurbished hardware sidesteps the shortage: our platforms are in stock and build to spec in about one business day.

The counterintuitive part: used enterprise GPUs hold their value well — A100-class cards retain roughly 60–80% of purchase price — so a used GPU server is not just cheaper to buy, it depreciates gently if you resell later.

Six cards cover almost every real-world on-prem AI decision in 2026. Used prices vary widely by form factor (PCIe vs SXM), condition, warranty, and whether a card is bought standalone or inside a tested system — treat the ranges below as directional, not a quote. For a firm number on a specific configuration, ask the quote desk.

Above these sits the H100 80GB (HBM3, ~3.35 TB/s, used roughly $18,000–28,000), which is worth it only when you are training or need the absolute highest serving throughput. For pure inference, an L40S delivers dramatically better cost-per-token — more on that below.

One caveat on the L40S: it has no NVLink, so multi-card builds communicate over PCIe. That is ideal for horizontally scaling independent inference requests, but for tightly-coupled workloads or distributed training that depend on NVLink bandwidth, A100 and H100 remain the better fit.

An RTX 4090 is cheaper than an L40S — about a quarter of the price — and leads on raw FP32 (its FP16 tensor throughput is roughly on par, not ahead). So why does anyone buy the L40S for a server? Because a GPU server is a different animal from a gaming PC, and three factors decide whether a card belongs in a rack:

• Cooling design. Data-center cards (L40S, A100, H100) are passively cooled — no onboard fans — and rely on the server chassis pushing high-static-pressure air through them front-to-back. Consumer cards (4090/5090/3090) have their own axial fans and expect an open case with side airflow. Put a standard 4090 in a 2U rack and it will overheat and throttle within minutes. Rare blower-style (turbo) GeForce cards from third parties handle rack airflow better — but they still fall under the consumer license below, so they are not a supported path for a commercial deployment.
• Licensing. NVIDIA's GeForce driver EULA prohibits use of consumer cards in commercial data-center deployments. The L40S, A100, and RTX 6000 Ada carry data-center/professional licensing and avoid that restriction entirely. For a business, this is not a technicality — it is the difference between a supportable deployment and one that violates your driver license.
• Form factor and power. Enterprise cards use the server's standard power delivery and mounting; consumer cards need 12VHPWR connectors, three slots of clearance, and often more length than a rack chassis provides.

Building multi-GPU on a budget in a workstation or open homelab rig? One nuance the spec sheets bury: the RTX 3090 supports an NVLink bridge, while the 4090 dropped it and has peer-to-peer over PCIe disabled in the driver. For tensor-parallel serving, two 3090s can outrun two 4090s despite the slower per-card speed. If you like to build it yourself, the things that actually bite are card length and 12VHPWR clearance, PCIe lane allocation, and the chassis fan profile — and if you would rather skip that, our quote desk supplies the enterprise cards too.

Start from the model you need to run and the number of users, then pick the smallest GPU (and count) that fits — and the platform built to host it. Over-buying VRAM is the most common and most expensive mistake.

The H100 is faster than the L40S on paper. But inference economics are about tokens-per-dollar, not peak throughput, and here the L40S wins for most serving workloads. On comparable cloud pricing the L40S runs roughly 5× cheaper per hour than an H100 SXM — a useful proxy for relative value, since a similar ratio broadly holds on purchase price. For most serving (moderate batch, prefill-heavy RAG and chat) the L40S throughput deficit does not come close to closing that gap. The H100's edge widens mainly on large-batch decode, which is memory-bandwidth-bound: its ~3.35 TB/s versus the L40S's 864 GB/s is where the H100 pulls ahead. Buy the same logic outright: two used L40S cards cost less than a single used H100 and deliver more aggregate VRAM and more parallel request streams.

The H100's advantage shows up in two places: training (where raw FP16/FP8 matrix throughput dominates) and ultra-low-latency single-stream serving of very large dense models. If neither describes your workload — and for most RAG, chat, copilots, and document pipelines it does not — the L40S is the rational purchase.

A GPU is only as useful as the chassis that can power, cool, and physically fit it. These are the refurbished GPU-optimized platforms we build to spec — all available to configure on our GPU Servers page:

Every one of these ships as a Build-Your-Own configuration: pick the GPUs, CPUs, memory, storage, and networking (dual 10/25GbE standard; 100GbE / ConnectX options for multi-node serving), plus NVMe for fast model loading and checkpointing. We assemble and test the whole system before it leaves the building. Not sure which GPU count your workload needs? That is exactly what the quote desk is for.

This is the part first-time buyers underestimate. A GPU server is not an office-desk machine — plan the environment before the purchase:

• Power. GPUs dominate the draw: four L40S at ~350W each is ~1.4 kW of GPU alone, and a fully loaded 2U node (dual CPU, NVMe, four accelerators) lands around 2–2.7 kW. That exceeds a standard US 120V/15A circuit (~1.8 kW) — these systems ship with redundant high-wattage PSUs and generally want a 208–240V feed and an appropriately rated PDU.
• Noise and cooling. Rackmount servers use high-static-pressure fans that run loud under load — fine for a server room, colocation cage, or garage/basement rack, but not a shared office or living space. Budget for the heat too: ~2.5 kW of compute is roughly 8,500 BTU/hr to remove.
• No data center? No problem. If you have no server room, colocation is the usual answer for startups — you rent rack units, power, and cooling by the month. If the deployment truly has to sit in an office, a quiet workstation (Precision 7960/7875, HP Z8) with one or two active-cooled GPUs is the realistic path, not a rack server.

Used GPUs carry more risk than used CPUs or RAM — they run hot, and some spent their lives in crypto or 24/7 training. Buy from a vendor who addresses these, or check them yourself:

• Testing and burn-in. Cards should run sustained-load diagnostics (DCGM plus a GPU stress/burn tool), not just POST — with ECC (correctable/uncorrectable) and XID error counts logged and a steady-state thermal read under load. Ask to see the results before you buy.
• Thermal service. Enterprise cards can need fresh thermal pads/paste after years of duty. Reputable refurbishers service this; a marketplace bargain often has degraded thermals.
• BMC thermal qualification. In a PowerEdge or ProLiant, a GPU the iDRAC/iLO thermal table does not recognize forces fans to 100% or throttles the card. Confirm the GPUs are OEM-qualified — or that fan curves are validated for the card — so you do not receive a screaming, throttling box.
• Provenance and VBIOS. Confirm cards are genuine data-center parts with unmodified VBIOS — not re-flashed mining cards or gray-market conversions.
• Warranty. A used GPU with no warranty is a gamble. A tested, warrantied build removes the single biggest risk of buying used silicon.
• System-level integration and software. The GPU has to be qualified with the server's power, cooling, BIOS, and firmware — and a genuinely production-ready node ships with a validated NVIDIA data-center driver, CUDA, and container runtime, smoke-tested against a serving stack such as vLLM. Buying a tested server — not a loose card you install yourself — eliminates that qualification work.

On-prem is not automatically cheaper — it wins when utilization is steady. Cloud is elastic and priced for it; owned hardware is a fixed cost that only beats rental once you keep it busy. A quick way to frame it: running a single GPU around the clock is about 720 hours a month, so at roughly $0.70/hr for an L40S-class instance or ~$3/hr for an H100, a steadily-used card burns on the order of $500 or $2,000+ per month in rental — every month, indefinitely. Owned hardware converts that recurring bill into a one-time capital cost plus power, and after payback the marginal token is essentially free.

Buy a used GPU server when: you run inference or scheduled fine-tuning most of the day, your workload profile is predictable, you have data-residency or compliance reasons to keep data in-house, or your cloud GPU bill already runs into the thousands per month.

Stay on cloud when: usage is spiky or experimental, you are still chasing product-market fit, you need short bursts of the very latest H100/H200/Blackwell capacity, or you would rather not own infrastructure yet. If that is you, an honest guide says rent — and come back when your utilization is steady.

Realistic order of magnitude: an entry single-GPU inference build lands in the low five figures; a multi-GPU serving node runs into the mid-to-high five figures depending on the GPUs. The quote desk returns exact numbers for your configuration — and for the full cloud-vs-on-prem TCO model, see our Local LLM Hardware Guide.

We build custom refurbished GPU servers on Dell PowerEdge and HPE ProLiant platforms, configured to your exact GPU, CPU, memory, storage, and networking spec. Every system is assembled and burn-tested in-house — GPUs run sustained-load diagnostics with ECC and thermal checks, and the node ships with a validated NVIDIA data-center driver and CUDA. Each build carries a PCSP-backed warranty (1 year standard, extendable to 5) that does not depend on expired OEM or NVIDIA coverage, and we can share the validation details for your unit. You get current-generation-adjacent silicon at 40–70% below new pricing, without touching the memory shortage or the data-center EULA minefield — and the same chassis re-configures for a different GPU load-out as your needs change.

L40S vs A100 — which should I buy for inference?

For most inference, the L40S. It delivers far better cost-per-token, carries clean data-center licensing, and 48 GB fits the majority of quantized models. Choose the A100 80GB when you need to serve a 70B dense model at FP16 (which takes two cards), want NVLink bandwidth to shard a model across GPUs, or are fine-tuning — workloads where HBM bandwidth is the bottleneck. Two L40S cards often beat one A100 for pure throughput serving.

Can I put an RTX 4090 or 5090 in a rack server?

Generally no. Consumer GeForce cards are actively cooled and are not designed for the front-to-back airflow of a rack chassis, so they overheat in 1U/2U servers. NVIDIA's GeForce driver EULA also prohibits consumer cards in commercial data-center deployments. Use a 4090/5090 in a workstation (Dell Precision, HP Z8) or open homelab build; for a rackmount GPU server, choose L40S or A100.

How much does a used A100 80GB cost in 2026?

Roughly $8,000–$12,000 for PCIe cards on the secondary market as of mid-2026, down from $15,000+ new (SXM parts list lower but require a compatible HGX board). Pricing varies widely by form factor, condition, and warranty, and is expected to soften further through 2026 as enterprises move to Blackwell hardware. The A100 40GB runs lower, around $4,000–$7,800.

Is a used H100 worth it over an L40S?

Only for training or ultra-high-throughput serving of very large dense models. A used H100 costs several times more than an L40S and does not deliver a matching cost-per-token advantage for typical inference. For RAG, chatbots, copilots, and document pipelines, L40S cards are the better buy; reserve the H100 budget for workloads that are genuinely training-bound.

What GPU do I need to run a 70B model locally?

At 4-bit quantization, a 70B model's weights fit in roughly 40–48 GB, but production serving needs headroom for KV cache, longer context, batching, and framework overhead — so a single 48 GB card works for constrained inference, while 80 GB (or two cards) is safer for production. Full FP16 weights are ~140 GB, making 2× A100 80GB with careful sharding a practical minimum. For most teams, quantized inference on a single 48–80 GB data-center card is the right call.

Are refurbished enterprise GPUs reliable for 24/7 use?

Yes, when properly refurbished. Enterprise GPUs are built for continuous duty. The keys are load-testing under sustained thermal stress, servicing thermal pads/paste, verifying ECC health, and a warranty. Every GPU server PCSP ships is tested as a complete system and warrantied for 1–5 years.

How many GPUs can a 2U server hold?

GPU-optimized 2U platforms like the Dell PowerEdge R760xa/R750xa and HPE ProLiant DL380a hold up to four double-width accelerators (or more single-width cards). Dense nodes like the Dell C4140 pack four tightly-coupled GPUs. The exact count depends on GPU width, power draw, and cooling — which is why buying a qualified, tested server matters.

Do you sell the GPUs separately, or only inside a server?

We focus on complete, tested GPU servers because that eliminates the compatibility and thermal risk of installing loose cards. Enterprise GPUs (L40S, A100, H100, RTX 6000 Ada) are configured and priced through our quote desk, since availability and pricing move quickly — we typically respond within one business day with current stock and lead time.

What does the warranty cover, and how does support work?

Configured GPU servers carry a PCSP-backed warranty — 1 year standard, extendable to 5 — covering the system and its GPUs, independent of any expired OEM or NVIDIA coverage. Because these are business-critical systems we handle RMAs directly; contact us to confirm current turnaround and to discuss the purchase terms your procurement process needs.

If you are upgrading to Blackwell and decommissioning A100, H100, V100, or L40S hardware — or retiring GPU servers of any kind — we are actively buying. GPUs hold strong value in today's market. Get in touch and we will quote a fair price, handle NIST 800-88 data sanitization, and arrange pickup.

The quote desk turns your workload into a concrete configuration — GPU model and count, CPU, memory, storage, and platform. To get you two or three options within a business day, tell us as much of the following as you can:

• Model(s) you plan to run — and whether it is inference or fine-tuning
• Quantization level (Q4/Q8/FP16) and context length you need
• Expected concurrent users or a throughput / batch-size target
• Rack or office deployment, plus any power and noise constraints
• Budget range and whether you prefer to start small and scale later

Not sure on some of these? That is fine — give us the shape of the workload and we will fill in the rest.

• Local LLM Hardware Guide 2026 — model-by-model VRAM sizing, five build tiers, and the full on-prem inference picture.
• What Every IT Buyer Needs to Know Right Now — the 2026 memory and HBM shortage, why it is structural, and how to plan procurement.
• How to Use an AI Server for Your Business — the business case for on-prem AI, from data prep to serving.
• Support Your AI Stack With These Hardware Upgrades — networking, RAM, and OS upgrades for teams already running AI workloads.
• Case Study: Cost Savings Through Refurbished IT Equipment — real numbers on what refurbished enterprise hardware saves over a multi-year horizon.

Sources

• Hashrate Index, Used GPU Market: A100 & H100 Pricing and Depreciation, 2026
• Spheron, L40S vs H100 for AI Inference — Cost Per Token, 2026
• Spheron, RTX 5090 vs RTX 4090 for AI: Benchmarks, VRAM, and Cost Per Million Tokens, 2026
• Alibaba Electronics, NVIDIA A100 80GB Price Guide, 2026
• NVIDIA L40S, A100, and H100 product datasheets and driver EULA
• Dell PowerEdge R760xa / R750xa / C4140 and HPE ProLiant DL380a technical guides
• TrendForce, DRAM & HBM Price Outlook, 2026