CloudSignal AI

Blog

Run a capable LLM on your laptop: a no-nonsense local setup

By Ashwin Rajendraprasad Tutorials

You do not need an API key, a GPU rental, or a signup form to run a genuinely capable language model. You need about 8 GB of free RAM and ten minutes. Here’s the whole thing, start to finish, on your own laptop — plus the one setting that decides whether it feels instant or unbearable.

1. Why run it locally at all

Four reasons, and none of them are “because it’s cool.” Your data never leaves the machine, which matters the moment you’re pasting in anything you wouldn’t email to a stranger. There’s no per-token bill, so you can be wasteful while you experiment. It works on a plane. And there are no rate limits, no surprise deprecations, and no model swapped out from under you mid-project. The tradeoff is real: a laptop model is smaller than a frontier hosted one, and you own the performance tuning. This post is about making that tradeoff go well.

2. The two-file mental model

Strip away the tooling and local inference is just two pieces: a runtime that executes the model, and a model file it loads.

The runtime almost everyone is standing on is llama.cpp (ggml.org), a C/C++ inference engine that runs quantized models fast on ordinary CPUs and consumer GPUs. Ollama wraps llama.cpp in a friendly CLI and a local server, and is the fastest way to get running, so we’ll use it. The model file is distributed in the GGUF format (llama.cpp’s container), and each file is baked at a particular quantization level, a label like Q4_K_M or Q8_0 that tells you how aggressively the weights were compressed. That’s the whole picture: pick a runtime, pull a GGUF, chat.

Update (July 2026): the “wraps llama.cpp” description has aged. Ollama 0.19 (March 2026) rebuilt Apple Silicon inference on top of MLX, Apple’s machine learning framework, shipping as a preview built around a single NVFP4-quantized model and asking for a Mac with more than 32 GB of unified memory. GGUF did not go anywhere: 0.30 (June 2026) widened GGUF support through llama.cpp and turned Vulkan on by default, work Ollama itself describes as augmenting the MLX engine rather than replacing it. The two-file mental model holds, and so does everything below. On a Mac, the runtime half may now be MLX instead of llama.cpp.

3. The fastest path: three commands

Install Ollama (on macOS and Windows it’s a normal app download from ollama.com; on Linux it’s one line), then pull a model and talk to it:

# 1. install (Linux; macOS/Windows use the app from ollama.com)
curl -fsSL https://ollama.com/install.sh | sh

# 2. pull Qwen3 (an open-weight model from Alibaba); pick a size that fits your RAM
ollama pull qwen3:8b

# 3. chat
ollama run qwen3:8b

A caveat on that first line: piping a remote script into sh executes unreviewed code as your user, and you are trusting whatever that URL serves at the moment you run it. On macOS and Windows, take the app installer from ollama.com/download instead. On Linux, fetch the script to disk first (curl -fsSL https://ollama.com/install.sh -o install.sh), read it, then run sh install.sh.

That’s it: you now have a working assistant in your terminal. We used Qwen3 (Alibaba, released April 2025, a “hybrid reasoning” family) as the example, but the whole point of open weights is that this line is swappable. Llama 4 (Meta) and DeepSeek’s open and distilled variants are other reasonable 2025 choices. New releases land monthly, so treat the exact model name as a variable, not gospel; the machinery underneath does not change.

One axis is worth understanding before you pick. Some models are tuned to reason: they generate a chain of internal deliberation before answering, trading latency and RAM for stronger results on hard problems. A plain chat model replies immediately. Qwen3’s “hybrid reasoning” framing refers to a model that can work either way. For a quick lookup, a chat model feels snappier; for a thorny question, a reasoning model’s extra tokens can be worth the wait. Both kinds run in the same runtime. The choice is about the task, not the plumbing.

4. Quantization without the math (and how much RAM you need)

A model’s weights are numbers. Train them and they’re 16- or 32-bit floats; quantization rounds them to fewer bits so the file is smaller and the math is faster. Lower bits mean less memory and more speed, paid for with a little quality. The rule of thumb the whole ecosystem has settled on: Q4_K_M (4-bit) is the sweet spot, near-original quality at roughly a quarter of the memory. Drop to Q8_0 (8-bit) when you have RAM to spare and want to shave off the last of the quality loss; go below 4-bit only when you’re desperate to fit.

Here’s the back-of-envelope that predicts whether a model will fit: at 4-bit, a weight costs roughly half a byte. So an 8-billion-parameter model is about 4–5 GB of weights, and you want headroom on top for context and the OS. The table below is rough (weights only, before context), but it’s the right mental model:

Model size4-bit on disk (~)Free RAM you’ll wantHow it tends to feel
1–4B~1–3 GB~4 GBsnappy, even CPU-only
7–8B~4–5 GB~8 GBcomfortable, the laptop sweet spot
13–14B~8–9 GB~12 GBfine if it fits in RAM
30B and up18 GB+24 GB+workstation territory

I’m deliberately not quoting tokens-per-second: it depends entirely on your CPU, GPU, and memory bandwidth, so measure it on your own machine rather than trusting anyone’s number. “Feel” above is qualitative on purpose.

5. Calling it from your own code

Ollama runs a local HTTP server on port 11434, so anything that can make a POST request can drive the model, no SDK required. Here’s the whole client in fifteen lines of Python:

# pip install requests==2.34.2
import requests

resp = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "qwen3:8b",
        "messages": [
            {"role": "user", "content": "Explain quantization in one sentence."}
        ],
        "stream": False,
    },
    timeout=120,
)
print(resp.json()["message"]["content"])

Because it’s just HTTP on localhost, the same request works from any language, and you can point a dashboard, a script, or a test suite at it. This local endpoint is also what you’ll aim retrieval and agent code at later: it’s the substrate for the RAG and fine-tuning tutorials in this series.

6. The sharp edges

Three things bite people, so flag them now.

Context length eats RAM. The model’s weights are a fixed cost; the context window is a variable one that grows with how much text you feed in. A long prompt or a long conversation can double your memory use before you notice. If you enlarge the context, watch your RAM.

“It’s slow” almost always means “it doesn’t fit.” When a model spills out of RAM, your OS pages it to disk and throughput collapses: from usable to unusable, not gradually. Nine times out of ten the fix isn’t a faster laptop, it’s a smaller model or a lower quant. Match the model to the table above before you blame the hardware.

“Open weights” is not the same as “open source.” You can download and run these models, but the license governs what you may do commercially, and some open-weight licenses carry restrictions. Before you ship anything on top of a local model, read its license. It’s a five-minute check that saves a legal headache later.

That’s the entire loop: a runtime, a GGUF at the right quant, and a model sized to your RAM. Once you’re comfortable here, the natural next steps are giving the model your own documents with retrieval-augmented generation and teaching it a house style with LoRA fine-tuning. If you’re moving from a laptop demo to something real (multi-user, latency-bound, on your own hardware), that’s the jump where an architecture review pays for itself, and it’s a conversation CloudSignal is glad to have.

Sources / further reading

Written by Ashwin Rajendraprasad for CloudSignal AI. The code above is free to reuse.