Error Loading RagChecker

[ss8319]

Setup
Python version 3.11
Windows Machine
pip install ragchecker
python -m spacy download en_core_web_sm

Its seems like there is trouble connecting with Azure OpenAI or utilising it. I used the same checking_inputs.json as the one provided

with open(r"Script/checking_inputs.json") as fp:
    rag_results = RAGResults.from_json(fp.read())

# set-up the evaluator
evaluator = RAGChecker(
    extractor_name="azure/gpt-4o",
    checker_name="azure/gpt-4o",
    batch_size_extractor=32,
    batch_size_checker=32
)

Is this convention correct?
"azure/model-deployment-name"

USER@MSI MINGW64 /d/LLM-Database-And-Notebook/Database/Copilot (main)
$ python Script/rag.py
Warning: scikit-learn-intelex not installed, sklearn acceleration for the RepC checker is not enabled.
2024-08-30 12:08:28.602 | INFO     | ragchecker.evaluator:extract_claims:101 - Extracting claims for response of 2 RAG results.
  0%|                                                                                              | 0/1 [00:00<?, ?it/sl 
itellm.APIError: AzureException APIError - argument of type 'NoneType' is not iterable [sleep 10 seconds]
litellm.APIError: AzureException APIError - argument of type 'NoneType' is not iterable [sleep 10 seconds]

I have also checked my API authentication by running some inferences on the completions API. There isn't any issue there.

[alexlorenzo]

Have you find a solution on this ?

import os
from refchecker import LLMExtractor, LLMChecker

# GPT-4o from OpenAI
model = 'gpt-4o'

os.environ["OPENAI_API_KEY"] = ""

evaluator = RAGChecker(
    extractor_name="openai/gpt-4o-mini",
    checker_name="openai/gpt-4o-mini",
    batch_size_extractor=8,  
    batch_size_checker=8
    )

This way it works

[ss8319]

I think that way uses OpenAI and not AzureOpenAI

[KannamSridharKumar]

its not working even using the openai. Pls post if you found a work around..

AttributeError: 'LLMExtractor' object has no attribute 'split'

[HuXiangkun]

RAGChecker uses litellm for invoking the LLMs. Please check their doc for various providers: https://docs.litellm.ai/docs/providers

If you are using OpenAI, just use the model name "gpt-4o" for extractor_name and checker_name.

[hoahoa1808]

Hi everyone,
I am try use extractor as 'gpt-4o' but got an issues (in the pic)

Have any gotten the similar issue?
cc @HuXiangkun

[Juanpa0128j]

Have the same issue as @hoahoa1808. Is there a way around this?

[alexlorenzo]

from ragas.metrics import faithfulness
from ragas import evaluate

from langchain.chat_models import ChatOpenAI

api_key = os.getenv("OPENAI_API_KEY")
gpt_4 = ChatOpenAI(model='gpt-4o',
                   openai_api_key=api_key,
                   temperature=0)

data_samples = {
    'question': question_list,
    'answer': answer_list,
    'contexts' : [context]
}

dataset = Dataset.from_dict(data_samples)```

Than you can launch your evaluation part : 
`score = evaluate(dataset, metrics=[faithfulness], llm=gpt_4, raise_exceptions=False)`

[Juanpa0128j]

@alexlorenzo is that a solution? Could you develop more on your explanation please? Thanks

[alexlorenzo]

Yes it is, I modified it.
If you have a correct Open API Key it should work