Prompt Injection

Prompt injection is an attempt by a malicious user to manipulate the behavior of an LLM or an LLM application through strategic prompting, in order to produce responses the application was designed to prevent.

Types of prompt injection

Prompt injections fall into several sub-categories.

1. Jailbreaking. Attempting to override the LLM's system prompt (its underlying instructions) to elicit inaccurate, biased, or forbidden responses. Jailbreaks are further categorized by mechanism: role play (also called double character or virtualization), obfuscation, payload splitting, and adversarial suffixes.
2. Instruction manipulation. Attempting to leak or ignore the LLM's system prompt or the application's prompt template, which can reveal sensitive information or inform future attacks.

The range of prompt injection techniques keeps growing as attackers explore new approaches, so these definitions continue to evolve.

The GenAI Engine approach

The prompt injection rule uses a binary classification model fine-tuned on prompt injection data. The engine ships with ProtectAI/deberta-v3-base-prompt-injection-v2, a DeBERTa-v3 classifier trained on a broad corpus of injection and jailbreak examples. Detection focuses primarily on role play and instruction manipulation attacks. The classifier runs locally inside the engine and does not call an external LLM.

Long prompts are split into overlapping windows of 512 tokens (with a 256-token stride) and each window is classified separately. The rule fails if any window is classified as an injection, so an attack buried in the middle of a long prompt is still caught.

The rule returns Pass or Fail with no additional detail fields. If the classifier has not finished loading, which can happen shortly after the engine starts, the rule returns Model Not Available; treat this as transient and retry.

Requirements

The prompt injection rule runs on validate_prompt. Only the user prompt needs to be passed.

We recommend passing only the user-supplied portion of the prompt rather than the full assembled prompt. The fixed instructions your application adds are not the attack surface, and leaving them out reduces noise for the classifier.

Benchmarks

Benchmarks draw on HackaPrompt and DeepSet.

Configuration

No additional configuration is required. The rule type is PromptInjectionRule, with apply_to_prompt set to true and apply_to_response set to false. See the Rule Configuration Guide for the request body and for enabling or disabling the rule by default or per task.