Toxicity

Definitions of toxicity

Toxic language from an LLM covers a broad range, from stereotypes about specific groups to hateful content to harmful or abusive language. Because of the explicit nature of toxic speech, no examples are given here.

The GenAI Engine approach

The toxicity rule narrows that broad definition to three categories, each handled by its own detector.

Hate speech and toxic content

The engine uses a model-based approach to capture toxicity according to this characterization of what it means for text to be toxic:

📘

Hateful or abusive speech towards group characteristics, such as ethnicity, race, gender, sexual orientation, or religion.

A classifier (s-nlp/roberta_toxicity_classifier) scores each piece of text for toxicity on a scale of [0, 1]. The score is compared against a threshold to produce a pass or fail, and that threshold is something you can tune for your use case.

The rule runs on both prompts and responses. Catching toxic responses is the obvious goal, but catching toxic prompts matters too: LLMs predict the next token, so a prompt containing toxic language raises the chance of a toxic response. Checking both ends lets you block before the model is called.

Profanity

Text is flagged as toxic even when it contains no hateful or threatening speech if it contains representations of profanity. Profanity detection combines a set of regular expressions covering common spellings and obfuscations with a profanity classifier. When profanity is found, the rule fails with a toxicity score just under 1.0 regardless of the threshold.

Harmful or illegal requests

As shown in Universal and Transferable Adversarial Attacks on Aligned Language Models, LLMs can be induced to assist with explicitly malicious or illegal requests despite their alignment training. Text containing such requests should fail the toxicity check. The rule reserves a harmful_request violation type for this category.

Violation types

When the rule fails, toxicity_violation_type in the rule result tells you which category triggered it:

• profanity
• harmful_request
• toxic_content (hate speech and other discriminatory language)

When the rule passes, the violation type is benign. The result also includes toxicity_score, the highest score any detector produced for the text.

{
  "id": "94ebc731-1e50-4bf7-a7ba-9bfd645d0323",
  "name": "Toxicity",
  "rule_type": "ToxicityRule",
  "scope": "default",
  "result": "Fail",
  "latency_ms": 62,
  "details": {
    "message": "Toxicity detected",
    "toxicity_score": 0.93,
    "toxicity_violation_type": "toxic_content"
  }
}

Long inputs are split into chunks and the rule reports the maximum score across chunks. Text over the toxicity token limit (GENAI_ENGINE_TOXICITY_CHECK_MAX_TOKEN_LIMIT) returns Skipped. If the classifier has not finished loading after engine startup, the rule returns Model Not Available; treat this as transient and retry.

Requirements

The toxicity rule runs on validate_prompt, validate_response, or both. We generally recommend checking both, though some use cases only need one.

Benchmarks

The benchmark dataset we assembled contains samples from:

• The Toxic-Chat dataset from LMSYS
• The WikiToxic dataset from Oxford
• The harmful behaviors dataset from the paper Universal and Transferable Adversarial Attacks on Aligned Language Models

Configuration

No additional configuration is required. The rule type is ToxicityRule, and apply_to_prompt and apply_to_response can each be true or false. See the Rule Configuration Guide for the request body and for enabling or disabling the rule by default or per task.

Choosing a threshold

You can set threshold in the rule's config. If you do not, it defaults to 0.5. Valid values are in (0, 1): 0 is the most stringent and 1 the least.

The rule's performance depends on picking the right threshold for your data. A grid search that maximizes weighted F1 (or whichever metric matters to you) on a labeled sample of your own traffic is a reliable way to find it. Weight the evaluation to match the real prevalence of toxic inputs: if you see one toxic prompt per ten benign ones, the sample should reflect that ratio, or the chosen threshold will be tuned for the wrong distribution.