Running a 14B model on a 12 GB GPU

VRAM on a single consumer GPU goes to three things: the weights, the KV cache, and a chunk of runtime overhead. Most "out of memory" surprises come from forgetting the second one.

1. Weights: parameters × bits

The formula is as boring as it gets:

weight_bytes = parameters × bits_per_weight / 8

The "≈" matters: quantized formats store per-block scale factors, so a "4-bit" file is really closer to 4.5–5 bits per weight on average. Check the actual file size; it is the ground truth.

2. The KV cache: the part everyone forgets

During generation the model keeps the keys and values of every previous token for every layer, so it doesn't recompute them. That cache grows linearly with context length:

kv_bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_value

Take a plausible 14B architecture: 48 layers, grouped-query attention with 8 KV heads, head dimension 128, cache stored in FP16 (2 bytes):

2 × 48 × 8 × 128 × 2 = 196,608 bytes ≈ 192 KiB per token

  8,192 tokens → ≈ 1.5 GiB
 32,768 tokens → ≈ 6.0 GiB

So the same model that comfortably runs at 8k context will not fit at 32k — not because of the weights, but because of the cache. Your model's config.json has all four numbers; plug them in rather than trusting a blog post (including this one).

3. Overhead

The CUDA context, compute buffers for activations, and the runtime itself typically take somewhere between 0.5 and 1.5 GB. Budget 1 GB and measure.

Putting it together

Q4_K_M weights      8.4 GB
KV cache @ 8k       1.5 GB
overhead            ~1.0 GB
---------------------------
total             ≈ 10.9 GB   → fits in 12 GB with a little headroom

At 32k context you would be around 15.4 GB. Two levers bring it down: quantize the KV cache to 8 bits (halves it to ≈ 3 GiB, usually with negligible quality loss), or offload a few layers to system RAM (works, but every offloaded layer costs real tokens per second).

With llama.cpp that looks like:

llama-server -m model-14b-Q4_K_M.gguf \
  -ngl 99 -c 16384 \
  --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0

-ngl 99 means "put every layer on the GPU". If it fails to load, lower it until it fits and watch throughput drop — that drop is your incentive to find the memory elsewhere.

What does 4-bit cost in quality?

Less than intuition suggests, down to a point. The usual pattern in perplexity measurements is a small, nearly flat penalty from 8 bits down to about 4–5 bits, then a cliff below roughly 3 bits, where models start losing coherence on long or precise tasks.

A rule of thumb that has held up well for me: a larger model at 4-bit usually beats a smaller model at 8-bit with the same memory budget. A 14B at Q4 tends to be a better assistant than a 7B at Q8. The exceptions are tasks that are very sensitive to exact recall — code with precise APIs, arithmetic, structured extraction — where it's worth running your own comparison.

A checklist before you download 9 GB

• Read num_hidden_layers, num_key_value_heads and head_dim (or hidden size ÷ attention heads) from the config.
• Decide the context length you actually need, not the maximum the model supports.
• Weights + KV cache + 1 GB ≤ VRAM − 0.5 GB of headroom for your desktop.
• If it doesn't fit: quantize the cache first, drop one quant level second, offload layers last.

None of this needs a benchmark harness. It needs a config file and a calculator — which is a nice change of pace in a field that usually needs both and a GPU cluster.