Run LLMs Locally: Ollama, llama.cpp, and What You Need

You don't need the cloud to run a language model. Here's how to run open-weight LLMs on your own hardware — Ollama, llama.cpp, and the trade-offs.

Cloud models are convenient, but they cost per token, phone home, and go down. Open-weight models — Llama, Mistral, Qwen, DeepSeek — run on your own hardware, offline, for free. Here’s the practical path.

Start with Ollama

Ollama is the easiest entry point: one command to install, one to run. On Linux or macOS:

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.2

That downloads the model and drops you into a chat. It exposes an OpenAI-compatible API on localhost:11434, so tools that speak OpenAI’s protocol work with a one-line base-URL change.

For maximum control: llama.cpp

llama.cpp is the C/C++ inference engine that popularized the GGUF format — quantized weights that run on almost anything, CPU included. It’s the foundation under many tools. Use it when you want fine control over quantization, GPU offload, and batching, or when you’re building your own pipeline.

What hardware you actually need

  • CPU-only — runs 3–8B parameter models slowly but fine for experimentation.
  • A single consumer GPU — 12–24GB VRAM runs 7–13B models comfortably; 24GB+ reaches 30–70B quantized.
  • Apple Silicon — unified memory makes Macs surprisingly good for 7–30B models.
  • Rule of thumb — model size in GB ≈ parameters × bytes-per-weight. A 7B model at 4-bit is ~4GB.

Quantization is the whole game

Weights are usually stored as 16-bit floats. Quantizing to 4-bit roughly halves memory use with a small quality hit — that’s what makes local inference viable. Trade quality for speed with the Q4, Q5, Q8 ladder (higher = better, slower).

Why run local at all

  • Privacy — nothing leaves your machine.
  • Cost — fixed, not per-token.
  • Reliability — no rate limits, no outages, no API deprecation.
  • Learning — you see the whole stack, from tokenizer to sampler.

Still fuzzy on the vocabulary? See the AI glossary. Curious how the big players stack up? See Nvidia vs OpenAI.

Related

Guides & explainers

Filed under: Software & Open Source, Linux, LLM, NVIDIA, Open Source