PII dataset into external model

ID

pii-dataset-into-external-model

Severity

high

Remediation Complexity

medium

Remediation Risk

medium

Remediation Effort

medium

OWASP LLM

LLM02:2025 — Sensitive Information Disclosure

Family

LLM02 — Sensitive Information Disclosure

Red-team vectors

Data Exfil via Markdown

Tags

ai_security

Description

A dataset classified as PII/PHI flows — through the in-scan lineage graph ai_dataset → ai_model → ai_inference_endpoint — to an external model where either:

  • the endpoint’s data_sharing_mode is train_on_prompts (the provider trains on the data), or

  • a data-residency allowlist is configured (--data-residency) and the model’s provider region falls outside it.

Either path discloses sensitive data to a third party, potentially permanently (trained into the provider’s model). The dataset is the protected object, so it is the finding’s primary asset; the model and endpoint travel as related assets, each with an evidence pointer so a triager can verify the join without re-running the scan.

The residency leg is off by default: with no allowlist configured, only the train_on_prompts leg fires, so no assumption is made about the customer’s data boundary.

How a dataset is classified

The dataset load call (load_dataset, read_csv, read_parquet, pd.read_*, …) is classified from the text of its line — the file name, the variable it is assigned to and a trailing comment — or, when that line says nothing, from a comment-only line right above it. The classification uses the platform’s shared sensitive-data vocabulary (the same one API Security uses for parameters and response fields), so it reports one of:

  • PII — personal identity data: names, e-mail, phone, national id, passport, date of birth, address, … or a label such as pii, gdpr, personal_data, customer_data.

  • PCI — financial data: card number, CVV, IBAN, bank account, tax id, SSN, … or credit_card.

  • PHI — health data: MRN, EHR/EMR, patient or medical records, … or phi, hipaa, patient, medical.

An explicit classification: pii|phi|sensitive|restricted comment is honoured too. The class is reported in the finding’s dataset_sensitivity property. Credential-like names (token, password, key) do not make a dataset personal data; those are covered by the Secrets scanner.

Examples

# customer_pii.csv is a dataset classified as PII
ds = load_dataset("customer_pii.csv")            (1)
llm = ChatMistralAI(model="mistral-large")
endpoint = { "base_url": "https://api.mistral.ai", "data_sharing_mode": "train_on_prompts" }  (2)
1 PII dataset declaration (the primary asset).
2 Endpoint trains on prompts — the disclosure path.

Mitigation / Fix

  • De-identify or tokenize PII/PHI before it reaches an external model.

  • Route to an endpoint with training opted out.

  • Keep the provider region inside your data-residency boundary; set --data-residency to enforce it.

© 2026 Xygeni. All rights reserved