High-stakes domain without grounding

high-stakes-domain-without-grounding

Severity

high (catalogue Medium; platform severity model has no 'medium'; confidence reduced)

Remediation Complexity

medium

Remediation Risk

low

Remediation Effort

medium

OWASP LLM

LLM09:2025 — Misinformation

OWASP ASI

ASI09:2026

Family

LLM09 — Misinformation

Red-team vectors

(none — LLM09 carries no red-team vector)

Description

Code generates content in a high-stakes domain — legal advice, medical/clinical content, or financial/investment guidance (matched by the high-stakes-domain heuristic pack over its prompt body) — with no grounding edge to a retrieval source and no citation requirement. Hallucinations are then delivered exactly where an error does real harm.

Two shapes anchor the rule:

a constructed agent (create_react_agent, initialize_agent, AgentExecutor, …);
a direct model call — a file that talks to a provider SDK with no agent at all (chat.completions.create, messages.create, generate_content, or a LangChain expression chain built on a chat-model constructor). This is by far the more common shape in application code.

Per open question #7, Phase 1 ships only the absent-safeguard fact — "the content is wrong" is not statically decidable. The domain classification is heuristic, so the finding carries reduced confidence.

Any one of these clears the finding: a retrieval/grounding source, a wired citation guardrail, or a citation directive written into the prompt body itself ("cite the specific clinical guideline for every claim"). The directive is read from prompt text only — comments and docstrings are ignored, so a file that documents its own gap ("no instruction to cite sources") does not thereby silence its own finding.

Examples

agent = create_react_agent(llm, tools)
prompt = SystemMessage("Answer questions about medication dosage and treatment.")   (1)

1	Medical-domain prompt with no retrieval grounding and no citation requirement — flagged.

completion = client.chat.completions.create(                                       (1)
    model="gpt-4o",
    messages=[{"role": "system", "content": "You are a legal advisor. Give legal advice."}],
)

1	No agent is constructed, but the call generates legal advice ungrounded and uncited — flagged.

Mitigation / Fix

Ground high-stakes answers in a vetted retrieval source and require citations.
Add a factuality / citation guardrail on the output; restrict the domain scope where grounding is not possible.
Where a retrieval pipeline is not available, instruct the model in the prompt to cite a named source for every claim and to refuse claims it cannot source:

msg = client.messages.create(
    system=(
        "You are a medical information assistant, not a doctor. For every "
        "possible condition you mention, cite the specific clinical guideline "     (1)
        "or peer-reviewed source it comes from."
    ),
)

1	An explicit citation directive in the prompt body — no finding.