Hallucination

Hallucinations are generated responses that are incorrect or unfaithful given the user input and the source knowledge supplied to the model.

Types of hallucination

Hallucinations fall into two sub-categories.

Intrinsic hallucination

Given source documents and a user prompt, the LLM's output directly contradicts information in the documents.

Extrinsic hallucination

The LLM's output may or may not be factually correct, but the information it provides is not present in the context given to the model.

The GenAI Engine approach

In real-world use cases a hallucination misinforms users, which can lead to bad business decisions. Applications that supply proprietary or domain-specific data to the model through retrieval are the ones where this matters most, because the user expects the answer to reflect that data.

The hallucination rule checks for intrinsic and extrinsic hallucinations by fact-checking the LLM's output against the context it was given. The check uses a proprietary prompting technique with an LLM to decide whether each claim in the response is supported. The LLM it calls is the one configured for your engine; see Configure Model Provider.

Claim-by-claim evaluation

Before calling the LLM evaluator, the engine parses the response into chunks of text so it can localize which chunk, if any, contains hallucinated information.

Once evaluation is complete, the rule returns Fail if any chunk contains a hallucination. The rule result includes a verdict for every chunk, so you can see whether the hallucination runs through the whole response or is confined to a single claim.

Detecting claims

Not every chunk is sent to the LLM evaluator. A classifier trained by Arthur decides whether a chunk contains a claim that needs evaluation. It returns False for conversational text such as "Let me know if you have any further questions!" and for responses that decline to answer, such as "I'm sorry, I don't have information about that stock price in the reference documents." It returns True when the LLM is actually answering the user's question from the context.

A chunk that is not classified as a claim appears in the rule result with "valid": true and "reason": "Not evaluated for hallucination."

Evaluating claims

A chunk classified as a claim is passed to the LLM evaluator together with the context.

If the evaluator finds no hallucination, the claim appears with "valid": true and "reason": "No hallucination detected!". If it does find one, the claim appears with "valid": false and a "reason" written by the evaluator explaining what is unsupported.

{
  "id": "22af5293-b2fd-4dad-a418-47ce2c9b729a",
  "name": "Hallucination",
  "rule_type": "ModelHallucinationRuleV2",
  "scope": "task",
  "result": "Fail",
  "latency_ms": 2105,
  "details": {
    "score": false,
    "message": "At least one claim was unsupported by the context.",
    "claims": [
      {
        "claim": "You can return unworn items within 30 days for a full refund.",
        "valid": true,
        "reason": "No hallucination detected!",
        "order_number": 0
      },
      {
        "claim": "Sale items can be returned within 90 days.",
        "valid": false,
        "reason": "The context states sale items are final and cannot be returned.",
        "order_number": 1
      },
      {
        "claim": "Let me know if you have any other questions!",
        "valid": true,
        "reason": "Not evaluated for hallucination.",
        "order_number": 2
      }
    ]
  }
}

Requirements

Both the response and the context it was grounded on must be sent to validate_response.

Context is required for the hallucination rule to run. If it is missing, the rule returns Skipped with the message "Please pass context for running hallucination rule" rather than failing the request. If your application does not provide context to the LLM, disable the hallucination rule for that task to avoid the extra latency of a check that cannot run.

The rule also returns Skipped when the response and context together exceed the engine's hallucination token limit, set by GENAI_ENGINE_HALLUCINATION_CHECK_MAX_TOKEN_LIMIT. If some claim batches cannot be evaluated because the LLM call fails, the rule returns Partially Unavailable with the claims that were evaluated.

Benchmarks

The benchmark datasets we assembled contain samples from the following sources.

• The Dolly dataset from Databricks. This dataset was compiled by Databricks employees in early 2023 by prompting an LLM to respond to questions while providing a context. The original responses mostly contained claims supported by their context, so we extended the dataset internally with examples of hallucinated responses, synthetically generated by gpt-3.5-turbo-0314.
• The Wikibio GPT-3 Hallucination dataset from the University of Cambridge. This dataset was compiled by prompting GPT-3 to write biographies, many of which contain hallucinated facts. Those hallucinations are labeled in the public dataset on Hugging Face.

When running the benchmarks, we sample N = 50 rows from each dataset.

Each row contains a context, an llm_response, and a binary_label for each claim in the response (the binary_label for a row is a string of one or more labels, one per claim). The label False for a claim means the claim is not supported by its context. It does not mean the claim is false on its own: claims that are true in the real world are still counted as hallucinations if the context does not support them.

The contexts can get long, so we benchmark only LLMs with a context window of at least 16k tokens.

Response-level and claim-level performance

We report hallucination detection performance at two levels.

If an LLM response contains three sentences and the second contains a hallucination, response-level detection classifies the whole response as a hallucination. Claim-level detection classifies the second sentence as a hallucination and the first and third as supported.

Response-level performance looks only at the rule result: did the rule fail, and was there a hallucination somewhere in the response? Perfect response-level performance does not require labeling individual claims correctly, only failing whenever a hallucination is present.

Claim-level performance is stricter: which claims did the rule label valid or invalid, and are those the claims that are actually faithful to the context?

You can typically expect response-level performance to be stronger than claim-level performance.

Which evaluator model you configure affects both accuracy and cost; see Configure Model Provider.

Benchmark data

Note on cost and latency: numbers are totals for the full benchmark set of N = 50. Cost is the number of prompt and completion tokens used multiplied by the provider's prices at the time. Cost and latency are reported only on the response-level rows, since they do not change between the two levels.

Note on the positive class: for precision, recall, and F1, the positive class is hallucinated claims (claims not supported by their context) and the negative class is supported claims. This is the reverse of the dataset labels, which mark a claim True when it is supported.

Configuration

No additional configuration is required. The rule type is ModelHallucinationRuleV2, with apply_to_prompt set to false and apply_to_response set to true. See the Rule Configuration Guide for the request body and for enabling or disabling the rule by default or per task.