Skip to content

RAGChecker v0.1.0
Compare
Choose a tag to compare
@HuXiangkun HuXiangkun released this 18 Jul 04:39
· 21 commits to main since this release
v0.1.0
cda8234

RAGChecker v0.1.0 Release Note

We are excited to announce the initial release of RAGChecker, version 0.1.0. RAGChecker is a comprehensive evaluation framework designed for in-depth analysis and diagnostics of Retrieval-Augmented Generation (RAG) systems.

Key Features

  • Fine-grained Evaluation: Utilizes claim-level entailment checking for detailed analysis of RAG system performance.
  • Comprehensive Metric Suite: Includes metrics for overall performance, retriever effectiveness, and generator capabilities.
  • Flexible Model Integration: Supports various LLMs for claim extraction and checking, including AWS Bedrock models.
  • Easy-to-use CLI: Provides a command-line interface for quick evaluation of RAG outputs.
  • Python API: Offers a Python API for seamless integration into existing workflows and scripts.

Metrics Included

  • Overall: Precision, Recall, F1 Score
  • Retriever: Claim Recall, Context Precision
  • Generator: Context Utilization, Noise Sensitivity, Hallucination, Self-knowledge, Faithfulness

Getting Started

To start using RAGChecker, install it via pip:

pip install ragchecker
python -m spacy download en_core_web_sm

For detailed usage instructions and examples, please refer to our GitHub repository: https://github.com/amazon-science/RAGChecker

Feedback and Contributions

As an open-source project, we welcome feedback, bug reports, and contributions from the community. Please use the GitHub issues section for reporting bugs or suggesting enhancements.

Thank you for your interest in RAGChecker. We look forward to seeing how it helps improve RAG systems across various applications!

Assets 2
Loading