Tag
red-team
22 entries, including 2 patterns.
A decoy puzzle is a denial of wallet
OverThink plants an ordinary-looking puzzle in a page a reasoning model will retrieve. The answer stays correct, the user notices nothing, and the model burns many times the reasoning tokens you are billed for.
A hallucinated package name is an open registration
Coding models invent dependency names, and they invent the same ones again. An attacker does not need to guess what your agent will install. They sample the model, find the names nobody owns, and register them first.
A poisoning budget does not scale with the corpus
Defenders assumed a bigger training set dilutes an attacker's samples. Anthropic, the UK AI Security Institute and the Alan Turing Institute found that about 250 documents backdoored models from 600M to 13B parameters, and the larger model saw twenty times more clean data.
A skill's prerequisites section is the payload
ClawHavoc filled an agent skill marketplace with polished documentation that ended in "install this first." The malware was never in the skill's code. It was a sentence the user, or the agent, was expected to follow.
A URL fragment is a prompt the server never sees
HashJack hides instructions after the "#" of a legitimate link. The fragment never leaves the browser, so the site, the WAF and the server logs see a clean request while the AI assistant reads the full URL as context.
Every response is a training label
Anthropic reports about 24,000 fraudulent accounts and more than 16 million exchanges used to distill Claude. Model extraction does not break the API. It uses it as designed, at a volume and in a pattern the product was never meant to serve.
The guardrail judged the message, not the conversation
Crescendo reaches a forbidden answer through turns that each look harmless. A per-message filter scores every one of them clean, because the violation exists only in the sequence.
The issue body ran with the maintainer's token
PromptPwnd found AI agents in CI that paste issue titles and pull request bodies straight into the prompt, then run with a write token and a shell. The attacker needs only a GitHub account and the ability to open an issue.
The MCP server added one BCC
The first widely reported malicious MCP server in the wild did not exploit the agent. It was a working email tool that, from one release on, quietly copied every message to the attacker.
The trusted config was the one the attacker could write
Two Cursor vulnerabilities targeted the same file. One let a prompt injection write a new MCP server into it without approval. The other let a repository swap an already-approved entry for a different command. In both, the config was the payload.
A checkpoint is not inert data
Loading a pickle-based model file runs whatever the file tells the unpickler to run. Attackers have shipped reverse shells on Hugging Face that fire on load, and one campaign broke its own file so the scanner errored after the payload had already executed.
Prompt secrecy is not a security control
OWASP LLM07 exists because a system prompt is configuration the model will happily recite. Treating its contents as a secret delegates the actual security decision to a text generator that has no mechanism to keep one.
A retrieved chunk is not a trusted source
RAG delegates judgment to retrieval, and retrieval answers with whoever sits nearest in vector space. An attacker who controls a single document controls the top-k that reaches the model — before any instruction is ever written down.
- Pattern
Retrieved Evidence Promotion Gate
Keep source claims and embedded instructions out of application authority, even when the poisoned chunk wins retrieval.
A retrieved chunk is not a data point
RAG poisoning turns your own retriever into the injection vector. A chunk that ranks top-k for a benign query is data to the pipeline and instructions to the model — and nothing in the stack can tell the difference.
- Pattern
Retrieved Context Action Boundary
Treat every retrieved chunk as evidence with provenance, while reserving tool and policy authority for a separate application-controlled gate.
A cache hit is not a trust grant
LLM serving caches the prefix and the answer, not the decision about them. What one tenant writes into the shared cache, another inherits — and the latency tells your adversary what you cached.
The LLM API reseller is the new bulletproof host
The South Korea ARTEX campaign reached DeepSeek through xcai.pro, a proxy reseller. That one hop is why attribution and blocking both fail, and it is the supply-chain gap nobody is watching.
The screenshot is a prompt, not a witness
Vision-language agents act on what they read in pixels. An attacker who controls a single image controls the instruction stream.
A retry is not a replay
The harness tells the model to try again. The model has no memory of the key it minted. At-least-once becomes at-least-twice, and that repetition is an attacker primitive, not a reliability footnote.
A tool result is not a system message
The return value of a tool call rides into context carrying execution authority. When the source is not yours, that value is a covert prompt channel.
A blocklist is not a jailbreak gate
String-match filters judge the surface of a prompt. Adversaries operate on the tokens the model actually reads.