PII dataset into external model
ID |
pii-dataset-into-external-model |
Severity |
high |
Remediation Complexity |
medium |
Remediation Risk |
medium |
Remediation Effort |
medium |
OWASP LLM |
LLM02:2025 — Sensitive Information Disclosure |
Family |
LLM02 — Sensitive Information Disclosure |
Red-team vectors |
Data Exfil via Markdown |
Tags |
ai_security |
Description
A dataset classified as PII/PHI flows — through the in-scan lineage graph ai_dataset → ai_model → ai_inference_endpoint — to an external model where either:
-
the endpoint’s
data_sharing_modeistrain_on_prompts(the provider trains on the data), or -
a data-residency allowlist is configured (
--data-residency) and the model’s provider region falls outside it.
Either path discloses sensitive data to a third party, potentially permanently (trained into the provider’s model). The dataset is the protected object, so it is the finding’s primary asset; the model and endpoint travel as related assets, each with an evidence pointer so a triager can verify the join without re-running the scan.
The residency leg is off by default: with no allowlist configured, only the train_on_prompts leg fires, so no assumption is made about the customer’s data boundary.
Examples
# customer_pii.csv is a dataset classified as PII
ds = load_dataset("customer_pii.csv") (1)
llm = ChatMistralAI(model="mistral-large")
endpoint = { "base_url": "https://api.mistral.ai", "data_sharing_mode": "train_on_prompts" } (2)| 1 | PII dataset declaration (the primary asset). |
| 2 | Endpoint trains on prompts — the disclosure path. |