DeepEval
Open-source testing framework for evaluating LLM outputs, treating LLM application quality checks like conventional unit tests.
- Actively maintained
- Well documented
- Permissive license
Overview
DeepEval provides metrics and test primitives specifically for LLM outputs - answer relevancy, faithfulness to source documents, hallucination detection, and more - so a team can write automated tests for an LLM application's quality the same way they'd write unit tests for regular code, and run them in CI. It integrates with common testing frameworks like Pytest.
DeepEval is a fit for teams that want systematic, automated quality checks on LLM or RAG application outputs rather than manual spot-checking, complementing Promptfoo (also in this catalog) - both address LLM evaluation, with DeepEval leaning toward a pytest-native unit-testing workflow.
Why it's in this catalog
Catalog rating: ★★★☆☆ (3/5)
LLM evaluation is a growing, still under-served niche even within a saturated ai category.
- Emerging
- Vendor-backed
- Categories
- AI & Machine Learning
- Languages
- Python
- License
- Apache-2.0
Related Projects
Promptfoo
MITOpen-source tool for testing and evaluating LLM prompts, comparing outputs across models and catching quality regressions before deployment.
Actively maintained
- AI & Machine Learning
Milvus
Apache-2.0Open source vector database built for scalable similarity search and AI applications.
Actively maintained
- AI & Machine Learning
- Database
MLflow
Apache-2.0Open-source platform for managing the machine learning lifecycle: experiment tracking, model packaging, versioning, and deployment.
Actively maintained
- AI & Machine Learning
PyTorch
BSD-3-ClauseOpen source machine learning framework that accelerates the path from research prototyping to production deployment.
Actively maintained
- AI & Machine Learning
Spotted an error? Suggest an edit on GitHub.