Ollama

Overview

Ollama is an open-source local LLM running tool that makes it simple to run large language models on personal computers. Its core philosophy is to simplify model deployment: a single command can download and run a model. Ollama supports open-source models such as Llama 3, Mistral, Phi-3, Gemma, and Qwen. It provides a REST API and command-line interface, making it easy to integrate into applications. Ollama automatically handles complex tasks such as GPU detection, memory management, and model optimization. It supports macOS, Linux, and Windows.

Installation

Ollama supports macOS, Linux, and Windows. The installation process is simple: download the installer and run it. After installation, the environment is automatically configured, requiring no additional setup.

# macOS
# Download the installer from https://ollama.com/download

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download the installer from https://ollama.com/download

# Docker
docker pull ollama/ollama
docker run -d -p 11434:11434 ollama/ollama

Running Models

Running a model is very simple, using the ollama run command. The first run will automatically download the model. After downloading, the model is stored locally, and subsequent runs do not need to re-download. It supports interactive conversations and one-shot queries.

# Run a model (downloads on first run)
ollama run llama3.1

# Interactive conversation
>>> Hello, please introduce yourself

# One-shot query
ollama run llama3.1 "Explain quantum computing"

# List installed models
ollama list

# Delete a model
ollama rm llama3.1

Model Library

Ollama provides a rich model library, including Llama, Mistral, Phi, Gemma, Qwen, DeepSeek, and more. Models come in different sizes (7B, 13B, 70B, etc.) and can be selected based on hardware. It also provides embedding models, vision models, and more.

# Common models
ollama run llama3.1        # Meta Llama 3.1
ollama run mistral         # Mistral 7B
ollama run phi3            # Microsoft Phi-3
ollama run gemma2          # Google Gemma 2
ollama run qwen2.5         # Alibaba Qwen 2.5
ollama run deepseek-r1     # DeepSeek R1
ollama run llama3.1:70b    # 70B version (requires more memory)
ollama run nomic-embed-text # Embedding model

Custom Models

Ollama supports custom models. By defining a model via a Modelfile, you can import GGUF format models, set parameters, customize system prompts, and more. Importing models from Hugging Face is supported.

# Modelfile example
FROM llama3.1

# Set parameters
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096

# System prompt
SYSTEM """
You are a professional programming assistant skilled in Python and JavaScript.
"""

# Create the model
ollama create my-assistant -f Modelfile

# Run the custom model
ollama run my-assistant

API

Ollama provides a REST API on port 11434 by default. The API is compatible with the OpenAI format and can be easily integrated into applications. It supports generation, chat, embedding, and other endpoints. Streaming responses are provided.

# Generate endpoint
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Why is the sky blue?"
}'

# Chat endpoint
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [
    {"role": "user", "content": "Hello"}
  ]
}'

# Python SDK
import ollama
response = ollama.chat(model='llama3.1', messages=[
  {'role': 'user', 'content': 'Hello'}
])

Fine-tuning

Ollama itself does not provide fine-tuning capabilities, but it can run fine-tuned models. You can use techniques such as LoRA and QLoRA to fine-tune a model, then convert it to GGUF format and import it into Ollama. This is suitable for scenarios that require specific domain knowledge.

# Fine-tuning workflow
# 1. Fine-tune the model with transformers
# 2. Convert to GGUF format
python convert.py model.pth --outtype f16

# 3. Quantize (optional)
./quantize model.gguf model-q4.gguf q4_0

# 4. Create a Modelfile
FROM ./model-q4.gguf

# 5. Import into Ollama
ollama create fine-tuned-model -f Modelfile

GPU Setup

Ollama automatically detects and uses GPUs (macOS Metal, NVIDIA CUDA, AMD ROCm). GPUs significantly improve inference speed. You can force the use of CPU or a specific GPU via environment variables. Multi-GPU configurations are supported.

# Check GPU usage
ollama ps  # Show the GPUs in use

# Force CPU usage
OLLAMA_NO_GPU=1 ollama serve

# Specify GPU (Linux)
CUDA_VISIBLE_DEVICES=0,1 ollama serve

# macOS automatically uses Metal
# No additional configuration required

Docker

Ollama provides an official Docker image, suitable for server deployment. You need to configure GPU support and volume mapping. Docker deployment makes management and scaling easier.

# Run with Docker
docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

# GPU support (NVIDIA)
docker run -d \
  --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama
    ports: ["11434:11434"]
    volumes: ["ollama:/root/.ollama"]
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

Integration

Ollama can be integrated into various applications. It supports frameworks such as LangChain, LlamaIndex, and Open WebUI. Through the OpenAI-compatible API, it can replace OpenAI in most applications. It is suitable for building local AI applications.

# LangChain integration
from langchain_ollama import OllamaLLM
llm = OllamaLLM(model="llama3.1")
response = llm.invoke("Hello")

# OpenAI-compatible API
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Hello"}]
)

# Open WebUI (Docker)
docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Configuration

Ollama is configured through environment variables and the Modelfile. OLLAMA_HOST changes the listen address, OLLAMA_ORIGINS controls CORS, and OLLAMA_NO_GPU/CUDA_VISIBLE_DEVICES manage GPU selection. Models are stored under ~/.ollama/models by default. A Modelfile lets you create custom models by layering a system prompt, parameters, and a base model. The server exposes an OpenAI-compatible API at /v1, so most OpenAI SDKs work without changes.

# Environment variables
export OLLAMA_HOST=0.0.0.0:11434   # listen on all interfaces
export OLLAMA_ORIGINS="*"          # allow CORS from any origin
export OLLAMA_NO_GPU=1             # force CPU-only
export CUDA_VISIBLE_DEVICES=0,1    # pick specific GPUs

# Modelfile - build a custom model
FROM llama3.1
SYSTEM "You are a senior Python engineer. Be concise."
PARAMETER temperature 0.2
PARAMETER num_ctx 8192

# Build and run
ollama create my-assistant -f Modelfile
ollama run my-assistant

# Config file locations
# Linux/macOS: ~/.ollama/
# Windows: C:\Users\<you>\.ollama\

Models can be many GB—point OLLAMA_MODELS at a disk with enough space if your home partition is small.

FAQ

Common questions cover hardware requirements, model storage location, custom models, OpenAI API compatibility, and multi-user access. Ollama runs on CPU but is much faster on GPU; memory needs scale with model size (a 7B model needs ~8 GB, 70B needs ~40 GB). Models live under ~/.ollama/models. The /v1 endpoint is OpenAI-compatible, so existing SDKs work. For multi-user access, run Ollama behind a reverse proxy and set OLLAMA_ORIGINS.

Q: Do I need a GPU?
A: No, Ollama runs on CPU, but inference is much faster on a GPU. A 7B
   model needs ~8 GB RAM/VRAM; a 70B model needs ~40 GB.

Q: Where are models stored?
A: Under ~/.ollama/models by default. Change it with the OLLAMA_MODELS
   environment variable if you need more disk space.

Q: Can I create a custom model?
A: Yes—write a Modelfile (FROM base-model + SYSTEM + PARAMETER) and run
   'ollama create my-model -f Modelfile'.

Q: Is the API OpenAI-compatible?
A: Yes, Ollama exposes /v1/chat/completions; point any OpenAI SDK at
   http://localhost:11434/v1 with api_key="ollama".

Q: How do I serve multiple users?
A: Run Ollama on a server, set OLLAMA_HOST=0.0.0.0 and OLLAMA_ORIGINS,
   and put it behind a reverse proxy (nginx/Caddy) for TLS.

Use 'ollama list' to see installed models and 'ollama rm <model>' to free disk space.