KServe
Kubernetes-native platform for serving machine learning models at scale, standardizing model deployment across frameworks.
Overview
KServe provides a standard Kubernetes custom resource for deploying trained models - from scikit-learn, PyTorch, TensorFlow, and others already in this catalog - with built-in autoscaling (including scale-to-zero), canary rollouts, and a consistent inference API regardless of which framework trained the model. It's a CNCF project built specifically for general-purpose ML model serving rather than LLMs specifically.
KServe is a fit for platform teams standardizing how models get deployed across many data science teams and frameworks, a broader-scope serving platform than vLLM or SGLang (also in this catalog), which focus specifically on LLM inference.
- Categories
- AI & Machine Learning
- Languages
- Python, Go
- License
- Apache-2.0
Related Projects
Kubeflow
Apache-2.0Machine learning platform for Kubernetes that bundles pipelines, model training, hyperparameter tuning, and serving into one toolkit.
- AI & Machine Learning
llama.cpp
MITHigh-performance C/C++ implementation for running LLM inference locally on consumer hardware, including CPUs, with minimal dependencies.
- AI & Machine Learning
MLflow
Apache-2.0Open-source platform for managing the machine learning lifecycle: experiment tracking, model packaging, versioning, and deployment.
- AI & Machine Learning
SGLang
Apache-2.0Fast serving framework for large language models and vision-language models, with a structured generation language for complex LLM programs.
- AI & Machine Learning
Spotted an error? Suggest an edit on GitHub.