Search: "llm inference" — FindOpenSource

6 projects found for "llm inference"

Related category: AI & Machine Learning (6)

Try also:model serving llm-inference model-serving llm-serving

vLLM
Apache-2.0
High-throughput, memory-efficient inference and serving engine for large language models, built for running LLMs in production at scale rather than on a single local machine.
- AI & Machine Learning
llama.cpp
MIT
High-performance C/C++ implementation for running LLM inference locally on consumer hardware, including CPUs, with minimal dependencies.
- AI & Machine Learning
SGLang
Apache-2.0
Fast serving framework for large language models and vision-language models, with a structured generation language for complex LLM programs.
- AI & Machine Learning
LangChain
MIT
Framework for developing applications powered by large language models, agents and retrieval-augmented generation.
- AI & Machine Learning
Ollama
MIT
Run large language models locally with a simple command-line interface and API.
- AI & Machine Learning
KServe
Apache-2.0
Kubernetes-native platform for serving machine learning models at scale, standardizing model deployment across frameworks.
- AI & Machine Learning