Local model tools

llama.cpp on GitHub: what it does and how to use it with AI agents

The C/C++ engine for running LLMs locally on almost any hardware.

Stars, license and last push from the GitHub API, checked 2026-10-26

Repo	ggml-org/llama.cpp
Stars	132,400+
License	MIT
Last push	2026-10-26
Language	C++

What llama.cpp does

llama.cpp runs GGUF models on CPUs and GPUs, including Apple Silicon. Its llama-server program exposes an OpenAI-compatible API, so agents can talk to a local model as if it were a hosted one.

Who it is for

Tinkerers who want full control over quantization, context size and speed on their own hardware.

How to use llama.cpp with Claude Code, Codex or Cursor

Install a prebuilt release or build from source as described in the README.
Download a GGUF model from Hugging Face.
Run llama-server -m model.gguf to start the local API.
Point an OpenAI-compatible tool at the server URL.

Tip: Ask Claude Code or Codex to pick quantization and context flags for your exact machine, then benchmark.

The loadout for this repo

See the full tool stack, prompts and costs in Best setup to run AI locally.

Related repos

Ollama: Run open models like Qwen, DeepSeek, gpt-oss and Gemma on your own machine.
LangGraph: LangChain framework for long-running, stateful agents.

llama.cpp FAQ

Is llama.cpp free?

Yes. It is open source under MIT.

Does llama.cpp have an API?

Yes. llama-server provides an OpenAI-compatible HTTP API.

What model format does it use?

GGUF.

Star counts change daily. Always read a repo's license and code before running it. All repos