Can Your Laptop Run Its Own LLM? A Practical Ollama Guide

Every API call to a hosted LLM costs money, needs the internet, and sends your prompt to someone else's server.

Sometimes that's fine.

Sometimes it isn't: private code, offline travel, a side project you don't want to put a credit card behind, or simply curiosity about how these models actually run.

That's where Ollama comes in.

In this guide, you'll learn how to:

• Install Ollama on Windows, macOS, and Linux

• Pull and chat with your first local model

• Call the local REST API on port 11434 with curl and Python

• Customise a model with a Modelfile

• Pick a model size that fits your RAM or VRAM

• Plug Ollama into your editor and a small RAG pipeline

Let's get it running.

Quick Answer

Ollama is a free, open-source tool that downloads open-weight LLMs and runs them on your own machine, exposing them through a simple CLI and a local REST API at http://localhost:11434.

The whole workflow looks like this:

Install Ollama
      ↓
ollama pull llama3.2      (download the model once)
      ↓
ollama run llama3.2       (chat in the terminal)
      ↓
POST localhost:11434/api/chat   (use it from your code)

That's the core idea.

What Is Ollama?

Ollama is a local model runner.

It handles the boring parts for you: downloading model weights, storing them, loading them into GPU or CPU memory, applying the right prompt template, and serving responses over HTTP.

Under the hood, the Ollama GitHub repository lists llama.cpp as a supported backend.

You don't need to know that to use it.

You just type a model name.

The models themselves come from the Ollama model library, where each model has tags for different sizes and quantizations.

Model names follow a model:tag format.

For example:

llama3.2          → same as llama3.2:latest
llama3.2:1b       → a specific, smaller variant

The Ollama API docs confirm that the tag defaults to latest when you leave it out.

Why Use Ollama?

A few practical reasons:

• Privacy: prompts and documents stay on your machine

• No per-token bill: you pay with hardware, not API credits

• Offline work: once a model is downloaded, no internet is needed to run it

• Fast experiments: swap models with one command

• Standard interfaces: a native REST API plus OpenAI-compatible endpoints

One honest caveat:

A small local model is not a frontier cloud model.

It's a different tool for different jobs.

How to Install Ollama

Installation takes a couple of minutes on every major OS.

Windows

The official Windows docs list Windows 10 22H2 or newer as the minimum, and NVIDIA driver 551.61 or newer if you have an NVIDIA card.

Open PowerShell and run:

irm https://ollama.com/install.ps1 | iex

Prefer a normal installer?

Download OllamaSetup.exe and run it.

The install doesn't need Administrator rights and goes into your home directory by default.

After installing, Ollama runs in the background and the ollama command works in cmd, PowerShell, or any terminal.

macOS

The macOS docs require macOS Sonoma (v14) or newer.

Apple M-series Macs get CPU and GPU support; Intel Macs run on CPU only.

Install from the terminal:

curl -fsSL https://ollama.com/install.sh | sh

Or download the app from ollama.com/download and drag it into Applications.

Linux

Run the official script:

curl -fsSL https://ollama.com/install.sh | sh

The Linux docs recommend running Ollama as a systemd service.

Check that it's running:

sudo systemctl status ollama

If something looks wrong, read the logs:

journalctl -e -u ollama

Docker

There's also an official ollama/ollama image on Docker Hub.

That's handy for servers, but for learning, the native install is simpler.

Verify the Install

Run:

ollama --version

Then check the server responds:

curl http://localhost:11434/api/version

Expected result (your version number will differ):

{"version":"x.y.z"}

If you get a JSON version back, the server is up.

Step 1: Pull and Run Your First Model

Download a model:

ollama pull llama3.2

Start a chat:

ollama run llama3.2

You'll see a prompt:

>>> Send a message (/? for help)

Ask it something:

>>> Explain what a REST API is in two sentences.

Type /bye to exit.

That's it.

You're running an LLM on your own hardware.

If ollama run is called for a model you haven't pulled yet, Ollama downloads it first.

Step 2: Call the Local REST API

The CLI is nice for testing.

Your apps will use the API.

According to the Ollama FAQ, Ollama binds to 127.0.0.1 on port 11434 by default.

So only your own machine can reach it unless you change OLLAMA_HOST.

Chat with curl

Send a chat request:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "Why is the sky blue?" }
  ],
  "stream": false
}'

Expected shape of the response:

{
  "model": "llama3.2",
  "message": {
    "role": "assistant",
    "content": "The sky looks blue because..."
  },
  "done": true,
  "total_duration": 4883583458,
  "eval_count": 282,
  "eval_duration": 4535599000
}

Look at this line:

"stream": false

By default, /api/chat streams a series of JSON objects, one chunk at a time.

Setting stream to false returns one complete object, which is easier when you're starting out.

What the Numbers Mean

The API reference says all durations are in nanoseconds.

It also gives the formula for speed:

tokens per second = eval_count / eval_duration × 10^9

That's the fairest way to compare models on your own machine.

Other Endpoints Worth Knowing

The ones you'll use most:

• POST /api/generate: single-prompt completion

• POST /api/chat: multi-turn chat with a messages array

• POST /api/embed: embeddings for search and RAG

• GET /api/tags: models downloaded locally

• GET /api/ps: models currently loaded in memory

• POST /api/pull: download a model from code

Here's what happens when your app calls /api/chat:

What happens when your app calls Ollama /api/chat

The first request after a model is unloaded is slower because the weights must be loaded from disk.

After that, the model stays in memory for the keep_alive period, which defaults to 5 minutes.

Step 3: Use Ollama from Python

You have two good options.

Option A: The Official Python Library

Install it:

pip install ollama

Create:

chat.py

Add:

from ollama import chat

response = chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Give me 3 tips for writing clean Python."}],
)
print(response.message.content)

Run:

python chat.py

The ollama-python repository has streaming, async, and embedding examples too.

Option B: The OpenAI-Compatible Endpoint

Already have code written for the OpenAI SDK?

The OpenAI compatibility docs show you can point it at Ollama:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",  # required by the client, ignored by Ollama
)

reply = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Say this is a test"}],
)
print(reply.choices[0].message.content)

Only the base URL and model name change.

Ollama supports a subset of the OpenAI API, so test the features you depend on.

Step 4: Customise a Model with a Modelfile

A Modelfile is a small text file that describes a custom model.

Think of it as a Dockerfile for LLM behaviour.

Create:

Modelfile

Add:

FROM llama3.2
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
SYSTEM """You are a senior Java reviewer. Answer briefly and point out bugs first."""

Build it:

ollama create java-reviewer -f Modelfile

Run it:

ollama run java-reviewer

The instructions you'll use most, per the Modelfile reference:

• FROM: the base model (required)

• PARAMETER: settings such as temperature, num_ctx, top_k, top_p, stop, seed

• SYSTEM: the system message

• MESSAGE: example conversation turns

• TEMPLATE: the full prompt template (rarely needed for library models)

Want to see how an existing model is configured?

Run:

ollama show --modelfile llama3.2

That's a great way to learn what a good template looks like.

A Real Ollama Example: Ask Questions About Your Own Notes

Let's build something useful: a tiny, fully local "ask my notes" script.

It uses an embedding model for search and a chat model for answers.

That's the same retrieval-then-generate pattern explained in What Is RAG? Retrieval-Augmented Generation Explained.

Pull an embedding model:

ollama pull nomic-embed-text

Create:

ask_notes.py

Add:

import math
import ollama

notes = [
    "Annual leave: employees get 24 days per year.",
    "The VPN must be used on public Wi-Fi.",
    "Expense claims are due by the 5th of each month.",
]

def embed(texts):
    return ollama.embed(model="nomic-embed-text", input=texts).embeddings

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

note_vectors = embed(notes)
question = "How many vacation days do I get?"
q_vector = embed([question])[0]

best = max(range(len(notes)), key=lambda i: cosine(q_vector, note_vectors[i]))

answer = ollama.chat(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "Answer only from the provided note."},
        {"role": "user", "content": f"Note: {notes[best]}\n\nQuestion: {question}"},
    ],
)
print(answer.message.content)

Run:

python ask_notes.py

Expected result (wording will vary):

You get 24 days of annual leave per year.

No API key.

No data leaving your laptop.

If embeddings are new to you, What Are Embeddings? explains what those vectors actually capture.

For a fuller version with chunking and a vector store, follow Build a Simple RAG Application Using Python and swap the model calls for Ollama.

How to Choose a Model Size for Your Hardware

This is where most people get stuck.

The rule is simple:

The model, plus room for its context, has to fit in memory.

Fits fully in GPU VRAM: fast.

Spills into system RAM: it still works, but slower.

Some reference points from the official docs:

• The API docs list llama3.2 (3.2B parameters, Q4_K_M quantization) at about 2 GB on disk

• The Ollama quickstart says the Gemma 4 E2B download is about 7.2 GB and recommends 8 GB of available VRAM, or unified memory on a Mac

• The FAQ notes that larger context windows need more memory

Each model page on ollama.com shows the download size per tag.

Use that number as your starting point, not the parameter count alone.

Here's the decision path I use:

Choosing a model size that fits your hardware

The key check is ollama ps.

Run a model, then:

ollama ps

Example output from the FAQ:

NAME        ID            SIZE    PROCESSOR   UNTIL
llama3:70b  bcfb190ca3a7  42 GB   100% GPU    4 minutes from now

The PROCESSOR column tells you where the model landed:

• 100% GPU: loaded entirely into the GPU

• 100% CPU: loaded entirely into system memory

• 48%/52% CPU/GPU: split between both

If you see a split and responses feel slow, try a smaller tag or a smaller context.

Context Length Matters Too

The FAQ says Ollama uses a 4096-token context window by default.

You can raise it per session:

/set parameter num_ctx 8192

Or for the whole server:

OLLAMA_CONTEXT_LENGTH=8192 ollama serve

Bigger context means more memory.

Don't raise it "just in case".

Using Ollama with Your Editor and Agents

Ollama doesn't stop at the terminal.

The Ollama README lists editor integrations such as Continue and Cline, both of which can use a local Ollama model.

Recent versions also include an ollama launch command that configures supported tools for you.

According to the CLI reference, supported integrations include Claude Code, Codex, OpenCode, VS Code, and Droid.

Try:

ollama launch

If you're comparing local setups with hosted coding tools, our Claude Code Tutorial is a useful companion.

Useful Ollama Commands

List downloaded models

ollama ls

See what's loaded in memory

ollama ps

Unload a model right now

ollama stop llama3.2

Delete a model to free disk space

ollama rm llama3.2

Inspect a model's details

ollama show llama3.2

Start the server manually

ollama serve

See server environment variables

ollama serve --help

Common Ollama Mistakes

1. Pulling a model that doesn't fit

A model that spills heavily into system RAM can feel painfully slow.

Check the download size on the model page and confirm with ollama ps.

2. Exposing port 11434 to the internet

Changing OLLAMA_HOST to 0.0.0.0 makes the server listen on all interfaces.

Do that only on a trusted network, or put an authenticated reverse proxy in front.

3. Forgetting "stream": false

Without it, you get many JSON lines instead of one object.

Your json.loads() will fail on the first try.

4. Running out of disk space

The Windows docs warn that models can be tens to hundreds of GB in total.

Move them with the OLLAMA_MODELS environment variable if your system drive is small.

5. Expecting cloud-model quality from a tiny model

A 1B or 3B model is great for summaries, classification, and simple Q&A.

Give it a hard multi-step reasoning task and it will struggle.

Match the model to the job.

6. Setting environment variables in the wrong place

On Linux with systemd, a variable exported in your shell won't reach the service.

Use systemctl edit ollama.service, then reload and restart, as the FAQ describes.

Ollama Best Practices

A few habits that save time:

• Pin tags in code (llama3.2:1b), not just latest, so behaviour doesn't change silently

• Use a Modelfile for repeatable system prompts and parameters

• Use keep_alive to keep a model warm during development, or set it to 0 to free memory immediately

• Measure speed with eval_count / eval_duration, not by feel

• Keep the server on 127.0.0.1 unless you really need remote access

• Clean up unused models with ollama rm

Is Ollama Worth It?

For most developers, yes.

It's the quickest path from "I've heard about local LLMs" to a working API on your own machine.

It's especially worth it if you:

• Work with private code or documents

• Want to prototype RAG or agents without API costs

• Need something that works offline

It's less useful if:

• You need the strongest possible reasoning on every request

• Your laptop has little free memory and no GPU

Even then, a small model for embeddings and quick tasks is often enough.

Ollama FAQ

Is Ollama free?

The Ollama software is open source and free to download from ollama.com.

Each model has its own licence, so check the model page before commercial use.

Do I need a GPU to run Ollama?

No.

Ollama can run models fully on the CPU using system memory.

A supported GPU makes responses much faster.

Where does Ollama store models?

Per the FAQ: ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux, and C:\Users\%username%\.ollama\models on Windows.

How do I keep a model loaded permanently?

Send "keep_alive": -1 in an API request, or set the OLLAMA_KEEP_ALIVE environment variable on the server.

Can I use Ollama with LangChain or LlamaIndex?

Yes.

Both are listed as integrations in the Ollama README, and the OpenAI-compatible endpoint works with many other libraries.

Key Takeaways

Remember these points:

• Ollama runs open-weight LLMs locally behind a CLI and a REST API on localhost:11434

• Install with one command on macOS and Linux, or one PowerShell line on Windows

• Use /api/chat with "stream": false when you're starting out

• A Modelfile turns a base model into a repeatable, customised assistant

• Pick models by download size versus your VRAM or RAM, then confirm with ollama ps

• The same server powers editors, agents, and fully local RAG

Start small, measure, then scale up the model.

You May Also Like

• What Is RAG? Retrieval-Augmented Generation Explained: the pattern behind the notes example above

• What Are Embeddings?: what /api/embed is really returning

• Build a Simple RAG Application Using Python: turn the example into a full app

• Claude Code Tutorial: a hosted coding agent to compare with your local setup

Get New Guides in Your Inbox

Like practical guides like this one?

Subscribe to the free TheTechLane newsletter and get new tutorials on AI, tools, and development as soon as they're published.

Sources

• Ollama GitHub repository and README

• Ollama REST API reference (docs/api.md)

• Ollama Modelfile reference

• Ollama FAQ

• Ollama on Windows

• Ollama on macOS

• Ollama on Linux

• Ollama CLI reference

• Ollama OpenAI compatibility

• Ollama Python library

• Ollama model library

Have a Question?

If something isn't clear, or your model won't load, leave a comment below with your OS and the output of ollama ps.

Happy Coding! 🚀

Follow TheTechLane

Keep learning AI, development, and modern developer technologies with TheTechLane.

Learn • Build • Share

Follow @thetechlanehq for more developer guides, tutorials, and practical technology content.