01
Model fit
Match the model to the task. A compact coding model can be more useful locally than a much larger general-purpose model that barely fits in memory.
Local inference · open weights · practical testing
Run AI on your own hardware.
Local AI Lab is a compact field guide to modern open-weight models, realistic hardware choices, benchmark interpretation and self-hosted inference.
7B–32Bsweet spot for consumer systems
4-bitcommon local quantization target
12–24 GBuseful VRAM range for enthusiasts
100%static, self-hosted website
$ local-ai inspect --system
GPU: consumer NVIDIA / AMD / Apple
runtime: llama.cpp / Ollama / vLLM
privacy: local by default
network: optional
quantization: Q4 / Q5 / Q8
$ choose-model --goal coding
scanning model families...
balancing quality, memory and speed
What matters
Better than a leaderboard.
Raw benchmark scores are useful, but local deployment adds constraints that hosted-model charts often ignore: memory, quantization, context length, prompt processing and sustained generation speed.
02
Memory first
Weights are only part of the budget. KV cache, runtime buffers, context length and GPU offload settings also consume RAM or VRAM.
03
Measure locally
Tokens per second depend heavily on backend, quantization, prompt size and hardware. Published numbers should be treated as reference points, not guarantees.
Current landscape
Open models keep getting more efficient.
Recent families increasingly combine reasoning, coding, multimodality and long context while using denser small models or sparse Mixture-of-Experts designs to reduce active compute.
Consumer-friendly
Models around 7B–14B parameters remain the easiest way to get useful local chat, reasoning and coding on a single consumer GPU or modern unified-memory laptop.
Workstation class
Models around 27B–32B deliver a meaningful capability jump, but usually benefit from 24 GB+ VRAM, partial CPU offload, or higher-capacity unified memory.
Start here
A simple local AI workflow.
1
Pick a realistic size
Start from available memory, not benchmark hype. Leave headroom for context and the runtime.
2
Use a quantized build
For local experimentation, 4-bit or 5-bit quantization often gives a practical quality-to-memory balance.
3
Benchmark your own prompts
Measure prompt processing speed, generation speed, peak memory and output quality on workloads that resemble your actual use case.