Tag
ai-security
14 entries.
A decoy puzzle is a denial of wallet
OverThink plants an ordinary-looking puzzle in a page a reasoning model will retrieve. The answer stays correct, the user notices nothing, and the model burns many times the reasoning tokens you are billed for.
A hallucinated package name is an open registration
Coding models invent dependency names, and they invent the same ones again. An attacker does not need to guess what your agent will install. They sample the model, find the names nobody owns, and register them first.
A poisoning budget does not scale with the corpus
Defenders assumed a bigger training set dilutes an attacker's samples. Anthropic, the UK AI Security Institute and the Alan Turing Institute found that about 250 documents backdoored models from 600M to 13B parameters, and the larger model saw twenty times more clean data.
A skill's prerequisites section is the payload
ClawHavoc filled an agent skill marketplace with polished documentation that ended in "install this first." The malware was never in the skill's code. It was a sentence the user, or the agent, was expected to follow.
A URL fragment is a prompt the server never sees
HashJack hides instructions after the "#" of a legitimate link. The fragment never leaves the browser, so the site, the WAF and the server logs see a clean request while the AI assistant reads the full URL as context.
Every response is a training label
Anthropic reports about 24,000 fraudulent accounts and more than 16 million exchanges used to distill Claude. Model extraction does not break the API. It uses it as designed, at a volume and in a pattern the product was never meant to serve.
The guardrail judged the message, not the conversation
Crescendo reaches a forbidden answer through turns that each look harmless. A per-message filter scores every one of them clean, because the violation exists only in the sequence.
The issue body ran with the maintainer's token
PromptPwnd found AI agents in CI that paste issue titles and pull request bodies straight into the prompt, then run with a write token and a shell. The attacker needs only a GitHub account and the ability to open an issue.
The MCP server added one BCC
The first widely reported malicious MCP server in the wild did not exploit the agent. It was a working email tool that, from one release on, quietly copied every message to the attacker.
The trusted config was the one the attacker could write
Two Cursor vulnerabilities targeted the same file. One let a prompt injection write a new MCP server into it without approval. The other let a repository swap an already-approved entry for a different command. In both, the config was the payload.
A checkpoint is not inert data
Loading a pickle-based model file runs whatever the file tells the unpickler to run. Attackers have shipped reverse shells on Hugging Face that fire on load, and one campaign broke its own file so the scanner errored after the payload had already executed.
Prompt secrecy is not a security control
OWASP LLM07 exists because a system prompt is configuration the model will happily recite. Treating its contents as a secret delegates the actual security decision to a text generator that has no mechanism to keep one.
A retrieved chunk is not a trusted source
RAG delegates judgment to retrieval, and retrieval answers with whoever sits nearest in vector space. An attacker who controls a single document controls the top-k that reaches the model — before any instruction is ever written down.
A cache hit is not a trust grant
LLM serving caches the prefix and the answer, not the decision about them. What one tenant writes into the shared cache, another inherits — and the latency tells your adversary what you cached.