Promptfoo
Open-source tool for testing and evaluating LLM prompts, comparing outputs across models and catching quality regressions before deployment.
Overview
Promptfoo runs a set of test cases against one or more LLM prompts or models, scoring outputs against defined criteria and showing side-by-side comparisons - so a team can catch a prompt change that degrades quality, or compare how different models perform on the same task, before shipping. It works as a CLI that fits into existing CI pipelines.
Promptfoo is a fit for teams iterating on prompts or comparing LLM providers who want data-driven evaluation instead of manual eyeballing, complementing DeepEval (also in this catalog) - both cover LLM evaluation, with Promptfoo leaning toward prompt and model comparison specifically.
- Categories
- AI & Machine Learning
- Languages
- TypeScript
- License
- MIT
Related Projects
DeepEval
Apache-2.0Open-source testing framework for evaluating LLM outputs, treating LLM application quality checks like conventional unit tests.
- AI & Machine Learning
CAMEL
Apache-2.0Open-source framework for research and building systems of multiple communicating, role-playing AI agents that collaborate autonomously.
- AI & Machine Learning
CrewAI
MITMulti-agent orchestration framework for building teams of collaborating, role-based AI agents that work together autonomously to complete complex tasks.
- AI & Machine Learning
DeepSpeed
Apache-2.0Deep learning optimization library from Microsoft for training and running very large models efficiently across multiple GPUs.
- AI & Machine Learning
Spotted an error? Suggest an edit on GitHub.