Can I run it without a graphics card?

Yes. Ollama runs on the CPU and system RAM, but much more slowly. It suits small models and work that can wait. The PROCESSOR column in ollama ps shows how much of the model sits on the GPU versus the CPU.

Does 4-bit quantization make the model much worse?

Somewhat, and fewer bits lose more. q4_K_M is the default tag for many Ollama models because it is a widely used balance point. If you have the room, try q8_0 against your own task and decide.

Can it replace the API in my existing code?

Mostly. Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1/, so you change the client's base_url. Provider-specific features may be missing.

Running an LLM locally with Ollama: when it pays and how much VRAM you need

Running an LLM locally means downloading the model file and letting a program such as Ollama or llama.cpp run it on your own GPU or CPU. Data never leaves the machine, it works offline, and there is no per-token bill. In exchange you need enough memory and you accept that quality trails the big cloud models. This article helps you decide whether it is worth it and works out how much VRAM you need.

Facts were checked against the Ollama docs (latest release v0.40.8 as of October 2026), the Ollama model library, and the config.json of Llama 3.1 8B and Llama 3.3 70B on Hugging Face. Memory numbers are my own arithmetic from those values, shown step by step.

When it pays, when it doesn’t

Situation	Local	API
Confidential data that must stay in-house (customer documents, internal logs)	good fit	check the contract and retention policy
No internet (on site, factory floor, IoT gateway)	good fit	not possible
High-volume small tasks (classification, field extraction)	good fit, if a small model is good enough	also very cheap; see below
Multi-step reasoning, long code, agents	clearly weaker	good fit
Small team with nobody to run the box	a burden	good fit

The strongest reasons are data must not leave and no internet. The money argument is usually weaker than people expect, because small API models are very cheap.

Memory, part 1: the weights

A model is billions of parameters, each stored in some number of bits:

weights (bytes) = parameters × bits per parameter ÷ 8

Quantization lowers the bits per parameter, from 16 to 8 or 4, trading a little accuracy for a much smaller model.

Llama 3.1 8B (about 8.03 billion parameters):

fp16 (16 bits): 8.03 × 10⁹ × 16 ÷ 8 = 16.06 × 10⁹ bytes ≈ 16 GB, matching Ollama’s 16GB file
q8_0 (about 8.5 bits including scales): 8.03 × 10⁹ × 8.5 ÷ 8 ≈ 8.5 GB, matching the 8.5GB file
q4_K_M: the real file is 4.9 GB; working backwards, 4.9 × 10⁹ × 8 ÷ 8.03 × 10⁹ ≈ 4.9 bits per parameter (not exactly 4, because some layers are stored at higher precision and scales take space)

Real file sizes from the Ollama library:

Model	q4_K_M	q8_0	fp16
Llama 3.1 8B	4.9 GB	8.5 GB	16 GB
Llama 3.3 70B	43 GB	75 GB	141 GB

ollama pull llama3.1:8b gives you the q4_K_M build by default.

Memory, part 2: the KV cache

While answering, the model keeps data for every token in the context in a KV cache. Longer context, more memory. The formula (stored at 16 bits = 2 bytes):

per token = 2 (K and V) × layers × KV heads × head size × 2 bytes

From Llama 3.1 8B’s config.json: 32 layers, 8 KV heads, head size 128.

Per token: 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB
4,096-token context: 4,096 × 128 KiB = 512 MiB
32,768: 4 GiB
131,072 (the full 128K): 16 GiB, three times the q4 model itself

Llama 3.3 70B: 80 layers, 8 KV heads, head size 128 → 2 × 80 × 8 × 128 × 2 = 327,680 bytes = 320 KiB per token, so 1.25 GiB at 4K context.

Context is not free. Set only what the task needs; a full-length KV cache can be bigger than the model.

Llama 3.1 8B on a 6 GB card: context length decides whether it fits

Worked example: what fits on a 6 GB card

A 6 GB card (6,144 MiB) with Llama 3.1 8B q4_K_M:

Weights: 4.9 GB ≈ 4,673 MiB (4.9 × 10⁹ ÷ 1,048,576)
KV cache at 4K context: 512 MiB
Total: 4,673 + 512 = 5,185 MiB
Left: 6,144 − 5,185 = 959 MiB for compute buffers and the display

A tight fit. Raise the context to 32K (4 GiB of KV) and it no longer fits: some layers spill to the CPU and everything slows down a lot. The 70B at q4_K_M (43 GB) needs several GPUs, or a machine with a lot of RAM and patience.

Ollama picks the default context from VRAM: under 24 GiB gets 4K, 24-48 GiB gets 32K, 48 GiB and up gets 256K. ollama ps shows the real value.

Getting started with Ollama

Install on Linux (on macOS and Windows, download the installer from ollama.com/download):

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl status ollama      # check the service is running

Download and chat:

ollama pull llama3.1:8b
ollama run llama3.1:8b            # type /bye to exit
ollama ps                         # SIZE, PROCESSOR (GPU/CPU), CONTEXT
ollama ls                         # models on this machine
ollama stop llama3.1:8b

Call the API on port 11434 and measure real speed from eval_count and eval_duration (nanoseconds):

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Classify this message: shipping is so slow, still not here after 3 days",
  "stream": false,
  "options": { "num_ctx": 4096 }
}' | jq '{tokens: .eval_count, tok_per_s: (.eval_count / .eval_duration * 1e9)}'

Existing code on the OpenAI client only needs base_url="http://localhost:11434/v1/" and any api_key value (local Ollama ignores it). To set the context for the whole server, use OLLAMA_CONTEXT_LENGTH=8192 ollama serve.

If other machines at home should reach it, do not open the port to the internet; use a VPN as in the Tailscale article.

The money: is it really cheaper than an API?

Classifying 10,000 messages a day, 500 tokens in and 20 out each.

On the API (Claude Haiku 5.5, as of October 2026: $0.10 / $0.50 per million tokens):

Input: 10,000 × 500 = 5M × $0.10 = $0.50 a day
Output: 10,000 × 20 = 0.2M × $0.50 = $0.10 a day
Total $0.60 × 30 = $18 a month; if it can wait, the Batch API halves it to $9

Locally (assumptions: the machine draws 300 W while working, 3 hours a day):

0.3 kW × 3 h = 0.9 kWh a day × 30 = 27 kWh a month, times your electricity rate
Plus the GPU, upkeep, and the time spent testing new models

The real hours depend on your hardware; measure tok_per_s above first. Conclusion: at this volume money is not the main reason. Move local for data or offline reasons. To cut an API bill instead, see controlling LLM API costs.

The quality gap you have to accept

7-12B models do narrow jobs well: classification, short summaries, pulling fields out of text, simple translation
Multi-step work is clearly weaker: agent planning, multi-file code edits, long reasoning. Large API models do much better
Non-English quality varies by model: some answer stiffly or switch languages. Test with your real text
Knowledge stops at the training date and there is no built-in web search; feed it your data, as with RAG

The most honest test: collect 50-100 real examples with correct answers, run them locally and on the API, and count how many each gets right.

Common mistakes

Looking only at the model file size and forgetting the KV cache. Raise the context and the model spills to the CPU, several times slower
Not checking ollama ps. The PROCESSOR column shows whether it is 100% GPU or split with the CPU
A context far longer than the task needs, wasting VRAM
Exposing port 11434 to the internet. Anyone can use your model. Use a VPN or a reverse proxy with authentication
Assuming local means safe from prompt injection. It does not; text you feed in can still steer the model

Sources

Commands and settings were tested on sample systems. Try them on a test machine before production.