Practical guides

From model file to useful local service.

The goal is not merely to make a model load. A good local setup should be repeatable, measurable and stable under the prompt sizes you actually use.

1. Estimate memory before downloading

A quick first approximation for quantized dense models:

Q4 weight size is often around 0.55–0.7 GB per billion parameters after format overhead. Add extra memory for KV cache, runtime buffers and GPU workspace.

That means a 7B model is easy territory for many GPUs, while 27B–32B usually moves into 24 GB VRAM, high unified memory, or partial CPU offload.

2. Pick the runtime

llama.cpp is excellent for GGUF and mixed CPU/GPU execution. Ollama is convenient for simple model management. vLLM is aimed more at high-throughput serving on supported GPUs.

3. Quantization is a trade-off

4-bit formats are popular because they dramatically reduce memory. Higher-bit quantization typically preserves more fidelity but consumes more memory and bandwidth. The best choice depends on the task and model family.

4. Context has a cost

A model may advertise a very large context window, but local inference at maximum context can demand substantial additional memory. Benchmark the context length you actually need.

Local benchmark recipe

Measure four things.

Cold start

Measure model load time and initial memory use from a clean start.

Prompt processing

Use a fixed prompt of meaningful size and record prompt-processing throughput separately.

Generation

Generate a fixed number of tokens with the same sampling settings and record decoding tokens per second.

Quality check

Keep a small suite of your own tasks: coding fixes, extraction, reasoning, summarization or structured JSON. Speed without useful output is not a win.

Practical defaults

A sensible starting point.

Hardware	Comfortable model range	Typical strategy
8 GB VRAM	3B–7B	Q4/Q5, modest context, full or near-full GPU offload
12 GB VRAM	7B–14B	Q4, leave headroom for KV cache
16 GB VRAM / unified memory	9B–14B, some larger	Q4/Q5, multimodal models become practical
24 GB VRAM	14B–32B	Q4/Q5, strong local coding and reasoning setups
48 GB+ memory	32B+ and larger sparse models	workstation/server deployment, longer context

These ranges are practical approximations, not hard limits. CPU offload can run larger models at lower speed, while some runtimes use memory more efficiently than others.