Sensitive Data Leakage

Sensitive data leakage is the inadvertent exposure of confidential or private information. There are two places where you may want to catch it.

1. Sensitive data in the prompt. If your application uses a public or externally hosted LLM, everything sent to that model may be retained, used for training, or exposed across contexts. You want to make sure users are not pasting confidential company information into prompts.
2. Sensitive data in the response. Teams add private data to LLMs as context (through retrieval or fine-tuning) so the model can answer more questions. But the end user may not have permission to see that data at a granular level, and the application should not return it to them.

Types of sensitive data

Broadly, sensitive data is confidential information stored, processed, or managed by an organization that should only be accessible to authorized users. What falls into that category is use-case specific.

For example, imagine an LLM augmented with client investment portfolio data. Internal users should be able to ask broad questions across portfolios, but responses that reveal a specific client's holdings to someone without access to them should be blocked.

Personally identifiable information (PII) is another kind of data teams often treat as sensitive, whether common PII such as social security numbers or use-case-specific identifiers such as API keys. The PII Leakage rule covers the former and Custom Rules cover the latter.

The GenAI Engine approach

The sensitive data rule uses LLM prompting with few-shot examples as a general-purpose detector for sensitive data that is hard to capture with concrete regex patterns. The LLM it calls is the one configured for your engine; see Configure Model Provider.

To use the rule, prepare a set of examples that should pass or fail according to the category of sensitive data you need to detect. Those examples become the rule's configuration, and the rule uses them to judge every prompt or response it sees from then on.

Preparing examples

Positive and negative examples

Collect both positive and negative examples. If you want to detect medical data, positive examples are instances of medical text and negative examples are similar-looking text that is not medical. Negative examples reduce the false positive rate by showing the LLM what not to flag.

Enough examples, but not too many

Two to twenty examples is the useful range. Performance typically improves as you add examples, but we have observed a drop-off when there are too many, usually beginning around 10 to 15.

Test the rule on a validation set before putting it into production to find the example set that works best for your use case.

Adding a hint

Add a short description of the type of sensitive data you are looking for as the hint when creating the rule. In our benchmarks a hint lifts performance across every dataset while adding very few tokens.

Requirements

The sensitive data rule runs on validate_prompt, validate_response, or both.

Text over the sensitive data token limit (GENAI_ENGINE_SENSITIVE_DATA_CHECK_MAX_TOKEN_LIMIT) returns Skipped.

Benchmarks

The benchmark datasets we assembled contain samples from the AI4Privacy PII-Masking dataset. Each dataset focuses on one category from AI4Privacy: passwords, usernames, company names, and job titles. These are categories the PII Leakage rule does not cover.

When running the benchmarks, we sample N = 30 rows from each dataset.

Each row contains a text and a boolean label. label == True means the text contains the category of sensitive data that dataset focuses on. A text containing a username is labeled True in the username dataset, but a text containing a username and no company name is labeled False in the company name dataset.

Recommendation

Based on these benchmarks, roughly 8 examples plus a hint on a small, fast model was the best balance of performance, cost, and latency. Which models are available depends on your provider; see Configure Model Provider.

Benchmark data

Note on cost and latency: numbers are totals for the full benchmark set of N = 30. Cost is the number of prompt and completion tokens used multiplied by the provider's prices at the time.

Configuration

The rule type is ModelSensitiveDataRule, and apply_to_prompt and apply_to_response can each be true or false. The config must include examples, and may include a hint. Because the examples are specific to each use case, create sensitive data rules at the task level rather than as default rules. See the Rule Configuration Guide for the request body.