How to run AI locally
Updated · What changed: First published with Gemma 4 E4B on Ollama as the default pick.
By The AI Loadout editors · How we pick
The easiest way to run AI locally is Ollama: install it, then run one command such as ollama run gemma4:e4b. Gemma 4 E4B (about 7 GB) is the best starting model for a 16 GB laptop. With 24 to 32 GB, step up to Gemma 4 26B or Qwen 3.5 27B. Prefer a desktop app? LM Studio does the same with a model browser and a built-in chat.
| Slot | Pick |
|---|---|
| Tool | Ollama |
| Model | Gemma 4 E4B |
| Skill | Agent Project Manager |
| Persona | The Local Assistant |
Free, no signup needed. Want changes in your inbox? Get the daily email.
The loadout (why each pick)
Tool: Ollama
Ollama installs on Mac, Windows and Linux, pulls a model with one command and serves a local API on port 11434 that most coding agents can use. Its library lists download sizes and context windows for every tag.
Model: Gemma 4 E4B
Gemma 4 is Apache 2.0, takes image input on every size and ships in sizes from E2B to 31B, so you can move up without changing tools. E4B is a 6.6 GB download at Q4 with a 128K context window, which fits a 16 GB laptop.
Agent skills
- Agent Project Manager: Routes easy tasks to your local model and only sends hard ones to a paid cloud model, so local AI saves real money.
Bot persona: The Local Assistant
Small local models do better with short, explicit instructions.
Comparison table
| Option | Best for | Model(s) | Price | Free plan | Last checked |
|---|---|---|---|---|---|
| Gemma 4 (E4B to 31B) | Best all-rounder at every size | Google, Apache 2.0 | Free, 6.6 GB (E4B Q4) | Yes | |
| Qwen 3.5 (0.8B to 122B) | Coding and many languages, widest size range | Alibaba, Apache 2.0 | Free, 6.6 GB (9B Q4) | Yes | |
| Qwen 3.8 27B | Newest Qwen for agentic work on 32 GB | Alibaba, Apache 2.0 | Free, 18 GB (Q4) | Yes | |
| gpt-oss 20B | Reasoning on a 16 GB GPU or Mac | OpenAI, Apache 2.0 | Free, 14 GB | Yes | |
| Muse Glimmer 30B | Always-on local agents with tools | Meta, Apache 2.0 | Free, 18 GB (Q4) | Yes | |
| Devstral Small 2 24B | Local coding agent, 384K context | Mistral, Apache 2.0 | Free, 15 GB (Q4) | Yes | |
| Mistral Small 3.2 24B | General assistant with vision | Mistral, Apache 2.0 | Free, 15 GB (Q4) | Yes | |
| Ministral 3 (3B to 14B) | Low-memory laptops | Mistral, Apache 2.0 | Free, 3.0 GB (3B Q4) | Yes | |
| DeepSeek-R1 8B | Step-by-step reasoning and math | DeepSeek, MIT | Free, 5.2 GB (Q4) | Yes | |
| Llama 3.3 70B | Proven general model for 48 GB and up | Meta, Llama 3.3 Community License | Free, 43 GB (Q4) | Yes |
Prices come from the linked vendor or source page and can change. Check the vendor before you buy.
Copy this setup
Paste this persona into your tool's custom instructions or project instructions:
You are a concise local assistant. Answer in plain language. If a task needs facts you may not know or data newer than your training, say so instead of guessing. Keep answers under 200 words unless asked for more. No em dashes.- Check your memory: the model download size plus 2 to 4 GB should fit in your RAM (Mac) or GPU VRAM (PC).
- Install Ollama from ollama.com, or LM Studio from lmstudio.ai if you prefer a desktop app.
- Run ollama run gemma4:e4b, or search Gemma 4 in LM Studio and click download.
- Point your tools at the local API: http://localhost:11434 for Ollama, or start LM Studio with lms server start and use http://localhost:1234/v1.
- Try a bigger model only if the small one is too weak for your task.
Alternatives by situation
- 8 GB laptop: Ministral 3 3B (3.0 GB) or Gemma 4 E2B (4.6 GB). Expect simple chat and summaries, not hard coding.
- 16 GB: Gemma 4 E4B, Qwen 3.5 9B or DeepSeek-R1 8B. gpt-oss 20B fits if little else is open.
- 24 to 32 GB: Gemma 4 26B, Qwen 3.5 27B, Qwen 3.8 27B, Devstral Small 2 or Muse Glimmer 30B.
- 48 GB and up: Llama 3.3 70B or Qwen 3.5 122B (81 GB). gpt-oss 120B needs about 65 GB plus context.
- Coding agents: Both Ollama and LM Studio expose an OpenAI compatible API on localhost, so agents and editors that accept a custom base URL can use a local model. Devstral Small 2 is built for this.
What changed
- : Page launched with initial picks. Sources are listed at the bottom of the page.
FAQ
Is running AI locally free?
Yes. Ollama, LM Studio and the models listed here cost nothing to download and use. You pay only for your hardware and electricity.
What is the best local model right now?
For most people, Gemma 4: E4B on a 16 GB laptop and 26B A4B on a 32 GB machine, as of Oct 15, 2026. Qwen 3.5 and Qwen 3.8 are the main alternatives, especially for coding.
How much RAM do I need?
Roughly the model download size plus 2 to 4 GB. A 7 GB model wants about 10 GB free; on a Mac that is shared memory, on a PC it should be GPU VRAM for good speed.
Ollama or LM Studio?
Ollama if you live in the terminal or want an API for agents. LM Studio if you want a desktop app with a model browser. Both serve an OpenAI compatible API, so tools work with either.
Are local models as good as ChatGPT or Claude?
Not for the hardest tasks. The best cloud models still lead on complex coding and reasoning. Local models are strong for private data, offline work, summaries and routine edits.
Sources
- Ollama library: Gemma 4
- Ollama library: Qwen 3.5
- Ollama library: gpt-oss
- LM Studio: Gemma 4
- LM Studio docs: OpenAI compatibility endpoints
- Ollama docs: API
Disclosure: links marked "affiliate" may earn us a commission, and sponsored slots are always labeled. Neither affects our picks. Read our methodology.