llama.cpp
High-performance C/C++ implementation for running LLM inference locally on consumer hardware, including CPUs, with minimal dependencies.
- Actively maintained
- Well documented
- Permissive license
Overview
llama.cpp runs large language models efficiently on ordinary hardware - including plain CPUs, not just GPUs - using quantization techniques that shrink model size and memory requirements with minimal quality loss, all in a dependency-light C/C++ codebase that's easy to embed or cross-compile. It's the inference engine that many higher-level local-LLM tools are built on top of.
llama.cpp is a fit for developers who want direct, low-level control over local LLM inference performance and hardware targeting, the underlying engine that Ollama (also in this catalog) wraps with a friendlier setup experience - power users often reach for llama.cpp directly when they need more control.
Why it's in this catalog
Catalog rating: ★★★★☆ (4/5)
The foundational CPU/edge-optimized LLM inference engine underlying much of the local-LLM ecosystem.
- Mature
- Community-driven
- Categories
- AI & Machine Learning
- Languages
- C++, C
- License
- MIT
Related Projects
vLLM
Apache-2.0High-throughput, memory-efficient inference and serving engine for large language models, built for running LLMs in production at scale rather than on a single local machine.
Actively maintained
- AI & Machine Learning
SGLang
Apache-2.0Fast serving framework for large language models and vision-language models, with a structured generation language for complex LLM programs.
Actively maintained
- AI & Machine Learning
LangChain
MITFramework for developing applications powered by large language models, agents and retrieval-augmented generation.
Actively maintained
- AI & Machine Learning
Ollama
MITRun large language models locally with a simple command-line interface and API.
Actively maintained
- AI & Machine Learning
Spotted an error? Suggest an edit on GitHub.