DeepEval
Open-source testing framework for evaluating LLM outputs, treating LLM application quality checks like conventional unit tests.
Overview
DeepEval provides metrics and test primitives specifically for LLM outputs - answer relevancy, faithfulness to source documents, hallucination detection, and more - so a team can write automated tests for an LLM application's quality the same way they'd write unit tests for regular code, and run them in CI. It integrates with common testing frameworks like Pytest.
DeepEval is a fit for teams that want systematic, automated quality checks on LLM or RAG application outputs rather than manual spot-checking, complementing Promptfoo (also in this catalog) - both address LLM evaluation, with DeepEval leaning toward a pytest-native unit-testing workflow.
- Categories
- AI & Machine Learning
- Languages
- Python
- License
- Apache-2.0
Related Projects
Promptfoo
MITOpen-source tool for testing and evaluating LLM prompts, comparing outputs across models and catching quality regressions before deployment.
- AI & Machine Learning
CAMEL
Apache-2.0Open-source framework for research and building systems of multiple communicating, role-playing AI agents that collaborate autonomously.
- AI & Machine Learning
CrewAI
MITMulti-agent orchestration framework for building teams of collaborating, role-based AI agents that work together autonomously to complete complex tasks.
- AI & Machine Learning
DeepSpeed
Apache-2.0Deep learning optimization library from Microsoft for training and running very large models efficiently across multiple GPUs.
- AI & Machine Learning
Spotted an error? Suggest an edit on GitHub.