Promptfoo
Open-source tool for testing and evaluating LLM prompts, comparing outputs across models and catching quality regressions before deployment.
- Actively maintained
- Well documented
- Permissive license
Overview
Promptfoo runs a set of test cases against one or more LLM prompts or models, scoring outputs against defined criteria and showing side-by-side comparisons - so a team can catch a prompt change that degrades quality, or compare how different models perform on the same task, before shipping. It works as a CLI that fits into existing CI pipelines.
Promptfoo is a fit for teams iterating on prompts or comparing LLM providers who want data-driven evaluation instead of manual eyeballing, complementing DeepEval (also in this catalog) - both cover LLM evaluation, with Promptfoo leaning toward prompt and model comparison specifically.
Why it's in this catalog
Catalog rating: ★★★☆☆ (3/5)
Widely used LLM prompt-testing and evaluation CLI, a practical developer-first tool.
- Mature
- Community-driven
- Categories
- AI & Machine Learning
- Languages
- TypeScript
- License
- MIT
Related Projects
DeepEval
Apache-2.0Open-source testing framework for evaluating LLM outputs, treating LLM application quality checks like conventional unit tests.
Actively maintained
- AI & Machine Learning
Milvus
Apache-2.0Open source vector database built for scalable similarity search and AI applications.
Actively maintained
- AI & Machine Learning
- Database
MLflow
Apache-2.0Open-source platform for managing the machine learning lifecycle: experiment tracking, model packaging, versioning, and deployment.
Actively maintained
- AI & Machine Learning
PyTorch
BSD-3-ClauseOpen source machine learning framework that accelerates the path from research prototyping to production deployment.
Actively maintained
- AI & Machine Learning
Spotted an error? Suggest an edit on GitHub.