Self Hosted AI Hardware CalculatorVRAM, GPU, power, and cloud breakeven months for local LLMs

The popular rule of thumb "you need 2 bytes per parameter at FP16" is correct as far as it goes - a 7B FP16 model weighs about 14 GB. What that rule misses is everything that also lives in VRAM: the KV cache that grows linearly with context length and concurrent users, the activation memory during prompt processing, and the CUDA workspace the runtime carves out. By the time you load a 70B Q4 model with a 32K context and serve five concurrent sessions, the "2 bytes per parameter" rule is off by 30-40 GB. People who follow that rule order a 4090, find that the model OOMs the first time anyone sends a long prompt, and end up reselling the card.

The calculator above does the full math. Model weights use the real bytes-per-parameter for the quantization scheme you picked (Q4_K_M includes its per-block metadata, FP16 does not). KV cache uses the actual num_layers and num_kv_heads for that model family, not a generic average. Concurrent users multiply the KV term. Overhead scales with parameter count. The number it returns is the working set you actually need to fit on the card, not the model file size.

The same logic applies in reverse to cloud breakeven. A common mistake is comparing the sticker price of an RTX 4090 to the hourly cost of an A100 instance and concluding cloud is always cheaper. In practice the 4090 lasts three years, draws power at most eight hours a day for a hobbyist, and the cloud A100 you would actually rent is closer to 600 dollars per month always-on. Run the numbers in the tool and the breakeven for a 4090 is typically 4-8 months at moderate use. That is the answer that lets you make a real buy-or-rent decision.

Self-hosted inference VRAM is the sum of three terms. The largest is model weights: number of parameters times bytes per parameter. For a Q4_K_M GGUF that is roughly 0.56 bytes per param (4 bits plus per-block scale and zero metadata), so 70 billion params lands near 39 GB. For FP16 that is exactly 2 bytes per param, so the same 70B model is 140 GB - which is why nobody runs 70B at FP16 outside a datacenter.

The second term is the KV cache. Every forward pass writes a key and value vector for each token, in every attention layer, for every attention head. The memory needed is 2 (K and V) x num_layers x num_kv_heads x head_dim x context_length x 2 bytes. Most modern models use grouped-query attention with 8 KV heads regardless of total head count, which keeps the cache surprisingly small - Llama 70B at 8K context is only about 2.5 GB of KV per session. Multi-session deployments multiply this term by concurrent users, which is where multi-user servers blow past the "just buy a 4090" answer.

The third term is activation and CUDA workspace overhead. During prompt processing, the model materializes intermediate tensors that vanish at the end of the forward pass. The calculator models this as a flat 2 GB plus 5 percent of the parameter count - small for a 7B, meaningful for a 70B. This is the term most rule of thumb estimates skip, and it is the term that turns a "close to fit" configuration into an OOM at runtime.

The calculator picks consumer cards first because that is where the price-per-VRAM curve is most generous. An RTX 3090 24 GB at street price (around 850 USD used in May 2026) is still the best dollar-per-GB GPU you can put in a consumer build. The 4090 jumps you to roughly 2.2x the cost for the same 24 GB but with 60-70 percent more tokens per second on most workloads, so the right way to choose between them is throughput-bound vs budget-bound, not VRAM-bound.

Above 24 GB the consumer market goes thin: the RTX 5090 ships with 32 GB, the RTX 6000 Ada has 48 GB at prosumer pricing (about 7,000 USD), and beyond that you are in datacenter SKUs - A100 40/80 GB, H100 80 GB - at 9,000 to 28,000 USD per card. The calculator only routes you to datacenter cards when single-GPU fit demands more than 48 GB. Below that ceiling, the answer is almost always "stack two 3090s on a board with x8/x8 PCIe lanes" for the price of one A100.

Multi-GPU has two non-obvious costs. The first is PCIe lane sharing: if your motherboard splits to x4/x4 because of an NVMe in the wrong slot, inference can lose 20-30 percent of its throughput compared to x8/x8. The second is power and case clearance: two 350 W 3090s need either a 1200 W PSU and an open-frame case or a dual-PSU build. The calculator factors in the extra 300 USD of riser cards and splitter cables that a real multi-GPU build needs, which is the cost most YouTube build-logs gloss over.

Apple Silicon is the only sane way to put 128 GB of GPU-accessible memory in a single machine without datacenter parts. An M3 Max 128 GB or M3 Ultra 192 GB lets you load a 70B FP16 model into unified memory, where the CPU and GPU share the same address space. The trade is throughput: Llama 7B Q4 hits roughly 100 tokens per second on a 4090, around 50 on an M3 Max, and 35 on an M3 Pro. Prompt processing (the time before the first token appears) is where Apple lags hardest, especially on long prompts.

The calculator routes to Apple Silicon when you pick that target, and ladders up through M3, M3 Pro, M3 Max, and M3 Ultra based on the unified memory you need. It leaves a 25 percent headroom because macOS reserves some of the unified pool for the OS and other processes - the 64 GB M3 Max only gives you about 48 GB of usable model memory under normal conditions. Apple ships a kernel boot argument (iogpu.wired_limit_mb) that raises the ceiling, but the calculator uses the out-of-the-box limit so the number it returns matches what your machine actually does.

Pick Apple Silicon for: interactive single-user, laptop portability, idle power, and anything above 64 GB without a server room. Pick a discrete GPU for: production serving, long-prompt latency, batch workloads, fine-tuning, and anything where you measure success in tokens per second per dollar.

The breakeven number is the most useful single output of the calculator and also the one most likely to surprise people. If you run an RTX 4090 ten percent of the time - call it three hours per day - the breakeven against a rented A6000 sits around 7-9 months. Below that usage, cloud wins; above it, DIY wins. The mistake people make is comparing peak rent to peak buy and forgetting that they will never actually run the cloud instance 24x7.

Three real-world adjustments the tool does not capture: cards lose 30-40 percent of their resale value per year, a homelab rig has incidental costs (UPS, networking, ventilation) that add 5-10 percent to the headline number, and your time setting up and maintaining the rig is not free. Even with those adjustments, the breakeven advantage for self-hosted is significant if you run inference for more than three hours per day - which most agents, code assistants, and bulk batch jobs easily exceed.

The other axis the breakeven misses is privacy and offline access. If your prompts contain customer data, source code, or anything you cannot ship to a third-party API, self-hosted is not a cost decision, it is the only option. Run the calculator with your model and context length, see the hardware bill, and treat the cloud breakeven as a sanity check on the total cost of ownership rather than the headline number.