Ollama

Lightweight, command-line local LLM runner and background daemon for effortlessly pulling, customizing, and serving open-source AI models via REST API

⚠️ CRITICAL VRAM & HARDWARE REQUIREMENT

Dedicated GPU VRAM is mandatory for real-time speed: While Ollama can run on standard CPU and system RAM, CPU-only token generation is painfully slow (often 1–3 tokens/second). High-performance local inference strictly requires sufficient dedicated GPU Video RAM (VRAM) to store the model's weight tensors and active conversation KV cache.

If a model's size exceeds your GPU's VRAM, Ollama automatically splits layers between the GPU and CPU system RAM. Once layers spill across the PCIe bus, processing speeds hit a dramatic bottleneck. Always verify model parameter sizes and quantization ratings against your available VRAM before pulling large models.

💡 TIP

Run ollama ps in your terminal while a model is running to see exactly how much GPU VRAM vs system RAM the model is consuming and confirm 100% GPU offloading.

Hardware Tiers & Recommended GPUs

Ollama delivers blazing fast inference when models fit entirely inside your GPU's high-speed memory. Below is an overview of hardware recommendations spanning from entry-level single GPUs to enterprise multi-GPU server racks, along with the model classes best matched for each setup.

Budget / Entry Single GPU

Tier 1: Budget

Example Hardware NVIDIA GeForce RTX 3060 12GB (approx. $200–$280 used) or RTX 4060 Ti 16GB

Dedicated VRAM Pool 12 GB – 16 GB GDDR6

Supported Model Tiers Lightweight Models (3B to 8B parameters) quantized to 4-bit (e.g., standard 4-bit quantizations) with 8k–16k context windows.

Performance & Role Generates 40–70 tok/sec. Ideal for shell scripting, home automation backends, terminal chat, and everyday question answering.

Mid-Range Workstation GPU

Tier 2: Mid-Range

Example Hardware NVIDIA GeForce RTX 3090 24GB (approx. $650–$750 used, community sweet spot) or RTX 4070 Ti Super 16GB / RTX 4080 16GB

Dedicated VRAM Pool 16 GB – 24 GB GDDR6X (up to 936 GB/s memory bandwidth on 3090)

Supported Model Tiers Mid-Tier Reasoning & Coding Models (14B to 32B parameters) at 4-bit, or dense 8B models at full uncompressed precision with 32k+ context.

Performance & Role Generates 30–60 tok/sec. High-speed local code generation, complex refactoring, and automated background API services.

High-End Prosumer Dual-GPU Array

Tier 3: High-End

Example Hardware Dual (2x) NVIDIA RTX 3090 24GB or Dual (2x) RTX 4090 24GB (48GB combined), or Apple Silicon Mac Studio (64GB–128GB unified memory)

Dedicated VRAM Pool 48 GB GDDR6X across dual PCIe slots (Ollama automatically splits layers across both GPUs)

Supported Model Tiers Flagship Heavyweight Models (70B parameters) at 4-bit quantization (~40–42GB VRAM needed), or 32B models at uncompressed 8-bit precision with 64k+ context.

Performance & Role Generates 15–28 tok/sec. Top-tier offline reasoning, complex multi-step agents, and deep autonomous coding workflows.

Extreme Multi-GPU Cluster

Tier 4: Enterprise

Example Hardware Quad (4x) RTX 3090/4090 (96GB VRAM), Dual/Quad NVIDIA RTX 6000 Ada (96GB–192GB ECC VRAM), or 8x NVIDIA H100 / A100 80GB SXM5 Nodes (640GB VRAM) ($10,000–$300,000+)

Dedicated VRAM Pool 96 GB to 640+ GB ultra-fast memory with NVLink / SXM interconnects (up to 3.2 TB/s)

Supported Model Tiers Massive Frontier Models & Mixture-of-Experts (MoE) Clusters (120B to 400B+ parameters), uncompressed FP16 70B models, and high-concurrency API serving.

Performance & Role Enterprise production performance supporting hundreds of simultaneous requests with massive context windows and zero quantization degradation.

•
Automatic Multi-GPU Layer Splitting: Ollama detects all available CUDA and Metal GPUs on boot and automatically balances model layer offloading proportionally across your GPUs without requiring custom scripts.
•
Inspect Active VRAM Consumption: Run ollama ps in any terminal to view active loaded models, their size in VRAM, the percentage offloaded to GPU, and when the model will automatically unload.
•
VRAM Sizing Formula: Memory needed ≈ (Total Parameters in Billions × Quantization Bits / 8) + Context Cache. A 70B parameter model at 4-bit requires ~38 GB for weights plus ~4–8 GB for context history.

CLI Commands & REST API

•
One-Line Model Pull & Execution: Run ollama run [model] to automatically download weights from the Ollama library and initiate an immediate interactive shell session.
•
Built-In Background REST Daemon: Ollama runs as a background service listening on http://localhost:11434. Native API endpoints include /api/generate, /api/chat, and /api/tags for instant integration into developer scripts.
•
OpenAI API Compatibility: Seamlessly integrate with any OpenAI SDK or frontend by pointing base URLs to http://localhost:11434/v1 (/v1/chat/completions). Works out-of-the-box with Cursor, Cline, and Open WebUI.
•
Custom Modelfiles: Package custom system prompts, stop tokens, and parameters into reusable Dockerfile-style bundles:
FROM llama3.2
PARAMETER temperature 0.3
SYSTEM """You are a specialized code review and refactoring assistant."""
•
Ecosystem & WebUI Integration: Easily connects to Open WebUI, AnythingLLM, Obsidian plugins, Docker containers, and local Home Assistant voice pipelines for complete homelab automation.

Performance Tuning & Environment Variables

•
Enable Flash Attention: Set the environment variable OLLAMA_FLASH_ATTENTION=1 before starting Ollama to drastically reduce VRAM usage on context tokens and accelerate generation.
•
Keep Models Loaded in VRAM: Set OLLAMA_KEEP_ALIVE=-1 to prevent Ollama from unloading idle models from GPU VRAM after the default 5-minute timeout. Perfect for zero-latency home automations.
•
Tune Parallel Request Concurrency: Set OLLAMA_NUM_PARALLEL=2 (or higher) to allow simultaneous processing of multiple incoming API requests without queuing, provided your VRAM has capacity for multiple context buffers.
•
Manage Disk Storage: Models are cached in ~/.ollama/models. Run ollama list to inspect disk usage and ollama rm [model] to reclaim space from unused models.