llama.cpp
High-performance C/C++ implementation for running LLM inference locally on consumer hardware, including CPUs, with minimal dependencies.
Overview
llama.cpp runs large language models efficiently on ordinary hardware - including plain CPUs, not just GPUs - using quantization techniques that shrink model size and memory requirements with minimal quality loss, all in a dependency-light C/C++ codebase that's easy to embed or cross-compile. It's the inference engine that many higher-level local-LLM tools are built on top of.
llama.cpp is a fit for developers who want direct, low-level control over local LLM inference performance and hardware targeting, the underlying engine that Ollama (also in this catalog) wraps with a friendlier setup experience - power users often reach for llama.cpp directly when they need more control.
- Categories
- AI & Machine Learning
- Languages
- C++, C
- License
- MIT
Related Projects
SGLang
Apache-2.0Fast serving framework for large language models and vision-language models, with a structured generation language for complex LLM programs.
- AI & Machine Learning
vLLM
Apache-2.0High-throughput, memory-efficient inference and serving engine for large language models, built for running LLMs in production at scale rather than on a single local machine.
- AI & Machine Learning
KServe
Apache-2.0Kubernetes-native platform for serving machine learning models at scale, standardizing model deployment across frameworks.
- AI & Machine Learning
CAMEL
Apache-2.0Open-source framework for research and building systems of multiple communicating, role-playing AI agents that collaborate autonomously.
- AI & Machine Learning
Spotted an error? Suggest an edit on GitHub.