LLM Red-Teaming in Production: Threat Models for Enterprise Generative AI Deployments
Adversarial prompt injection, training-data exfiltration, and model supply-chain attacks — threat modeling patterns for security teams.
The attack surface nobody scoped
Most enterprises deployed generative AI faster than they threat-modeled it. A customer-support copilot with retrieval access to internal knowledge bases is, structurally, an unauthenticated query interface to your document store — mediated by a model that can be persuaded. Traditional application security reviews rarely capture this, because the vulnerability is not in the code path; it is in the instruction-following behavior of the model itself.
Three threat classes that matter in production
Across our LLM red-team engagements, findings cluster into three classes. First, indirect prompt injection: adversarial instructions embedded in content the model will later read — support tickets, web pages, inbound email — that redirect its behavior. Second, data exfiltration through retrieval: coaxing a RAG pipeline into surfacing documents outside the user's authorization scope. Third, tool abuse: agentic deployments where a hijacked model invokes connected tools (email, ticketing, code execution) with the privileges of the application, not the attacker.
Supply chain: the quiet risk
Model supply-chain attacks remain under-defended. Weight files pulled from public hubs, fine-tuning datasets assembled from scraped content, and third-party adapters all enter the enterprise with less scrutiny than a typical npm dependency. Provenance verification and integrity checks on model artifacts should be treated with the same discipline as code signing.
What a defensible posture looks like
Treat the model as an untrusted component: least-privilege tool access, authorization enforced outside the model at the retrieval and action layer, input/output filtering as defense-in-depth rather than the primary control, and adversarial regression testing wired into the deployment pipeline so guardrail fixes survive model upgrades. Teams that operationalize red-team findings into automated regression suites hold their posture; teams that patch prompts ad hoc regress within two model versions.