Most GPU server quotes start with FLOPS and end with regret. In production inference the number that decides your budget is almost always memory bandwidth, followed closely by how much VRAM you can keep a single model resident in. This is the checklist our architects run before quoting any cluster.
Start from the model, not the card
A 70B parameter model at FP16 needs roughly 140 GB of weights alone, before KV cache. That single fact eliminates half the catalogue. Once you know the resident size, you can work backwards to the smallest node that holds the model plus the cache your concurrency demands.
Compute is the foundation, memory is the ceiling. Buy for the ceiling.
The three numbers that matter
- Resident size — weights plus KV cache at your target context length.
- Bandwidth per dollar — tokens per second is bandwidth-bound far more often than teams expect.
- Power and cooling per rack — the constraint nobody budgets for until the rack arrives.
Where teams overspend
The most common mistake is buying eight small cards when two large ones would hold the model natively. Sharding a model that fits in one node adds latency, networking cost and operational surface for no throughput gain. We see this in roughly one in three enquiries.
A five-minute sanity check
Take your model size in billions, multiply by two for FP16, add 20 percent for cache, and compare that to usable VRAM per node. If it fits in one node, buy one node. If it does not, the question becomes interconnect, and that is where a conversation with an architect saves real money.
If you want a second opinion, send us your model, batch size and latency target. We will return a configuration and a benchmark estimate free of charge.