Tag
threat-model
55 entries, including 11 patterns.
A decoy puzzle is a denial of wallet
OverThink plants an ordinary-looking puzzle in a page a reasoning model will retrieve. The answer stays correct, the user notices nothing, and the model burns many times the reasoning tokens you are billed for.
A hallucinated package name is an open registration
Coding models invent dependency names, and they invent the same ones again. An attacker does not need to guess what your agent will install. They sample the model, find the names nobody owns, and register them first.
A poisoning budget does not scale with the corpus
Defenders assumed a bigger training set dilutes an attacker's samples. Anthropic, the UK AI Security Institute and the Alan Turing Institute found that about 250 documents backdoored models from 600M to 13B parameters, and the larger model saw twenty times more clean data.
A URL fragment is a prompt the server never sees
HashJack hides instructions after the "#" of a legitimate link. The fragment never leaves the browser, so the site, the WAF and the server logs see a clean request while the AI assistant reads the full URL as context.
The guardrail judged the message, not the conversation
Crescendo reaches a forbidden answer through turns that each look harmless. A per-message filter scores every one of them clean, because the violation exists only in the sequence.
The MCP server added one BCC
The first widely reported malicious MCP server in the wild did not exploit the agent. It was a working email tool that, from one release on, quietly copied every message to the attacker.
A checkpoint is not inert data
Loading a pickle-based model file runs whatever the file tells the unpickler to run. Attackers have shipped reverse shells on Hugging Face that fire on load, and one campaign broke its own file so the scanner errored after the payload had already executed.
Prompt secrecy is not a security control
OWASP LLM07 exists because a system prompt is configuration the model will happily recite. Treating its contents as a secret delegates the actual security decision to a text generator that has no mechanism to keep one.
A retrieved chunk is not a trusted source
RAG delegates judgment to retrieval, and retrieval answers with whoever sits nearest in vector space. An attacker who controls a single document controls the top-k that reaches the model — before any instruction is ever written down.
The screenshot is a prompt, not a witness
Vision-language agents act on what they read in pixels. An attacker who controls a single image controls the instruction stream.
A tool result is not a system message
The return value of a tool call rides into context carrying execution authority. When the source is not yours, that value is a covert prompt channel.
- Pattern
Purpose-Bound Handoff Packets
Give each specialist only the authenticated fields its task requires, and test the input it actually receives.
The specialist inherited the whole case
A handoff can transfer far more than the task. Decide what the receiving agent may see before it receives history.
A blocklist is not a jailbreak gate
String-match filters judge the surface of a prompt. Adversaries operate on the tokens the model actually reads.
- Pattern
Replay-Fenced Agent Side Effects
Give one approved intent one durable target-side identity, regardless of workflow retries, forks or restarts.
- Pattern
Revocable A2A Push Subscribers
Make a task's outbound update subscription expire with its subscriber's entitlement, including queued retries.
The checkpoint did not undo the transfer
Replaying an agent workflow can repeat a real-world action unless the target recognizes the same operation.
The webhook kept listening
An A2A task can keep sending updates to a once-authorized subscriber after that subscriber loses access.
- Pattern
Adjudicated Feedback Promotion
Keep production complaints useful for finding failures without letting them decide what a secure agent should do.
- Pattern
Revocation-Aware Policy Gates
Make policy freshness an explicit precondition for privileged agent actions, and prove revocations take effect at every serving gate.
The revocation that never reached the gate
A centrally approved policy change does not revoke an agent's access until every consequential gate stops using the old revision.
When feedback becomes the test
A complaint can reveal a failure; it should not be able to rewrite the acceptance test that decides whether an agent is safe to release.
- Pattern
Enforced Filesystem Scope for MCP Servers
Treat selected directories as intent and constrain the server process to the files it may actually open.
- Pattern
Same-User Elicitation Binding
Require the browser principal completing an MCP URL elicitation to match the authenticated principal that initiated it.
The folder picker did not confine the server
MCP roots describe where a server should work; they do not restrict what its process can read.
The link connected the wrong account
MCP URL elicitation keeps credentials out of the agent, but a copied authorization link can still bind a victim's account to the attacker's session.
A task ID is not an authorization boundary
A2A task handles identify work. They do not decide who may read its artifacts, cancel it, or receive its updates.
Inspected is not executed
A safety check on the raw command or code string does not prove the runtime will execute that same string.
- Pattern
Task-Bound A2A Authorization
Bind every task read, stream and mutation to the authenticated caller's ownership and action rights, not possession of its ID.
A registry spawn is not a catalog read
Accepting an MCP client registration that carries a command is process start, not browsing a tool list.
Non-empty is not authenticated
A non-empty Authorization header is not proof that a verifier ran or that the credential is valid.
Allowlisted is not constrained
Matching the first path token against an allowlist does not prove the process will only run that program's intended verbs.
Denied on the sandbox is not denied on the loopback
A sandbox tools.deny entry is not a gate if the MCP loopback never applies it.
Rejected is not blocked
A finishReason of tool_rejected does not prove the tool never ran.
A browsing agent inherits localhost
An Origin allowlist for 127.0.0.1 does not stop JavaScript that a local agent just rendered.
A valid token is not your audience
A signature and expiry check does not prove the token was issued for this MCP server.
Validated is not pinned
A hostname that passed an SSRF check can still dial a different address on the next lookup.
URL validation is not egress
A deny-list in the fetch helper is not a network policy. The agent host still reaches what the process can reach.
A session ID is not a credential
Conversation continuity tracks state. It does not prove who is calling.
Syntax is not intent
A well-formed tool call can still be the wrong action. Clear syntax is not a policy verdict.
Visible text is not the filter string
What a person reads on screen is not the string that keyword, signature, or model-ingestion controls evaluate. Invisible Unicode can split both.
Approved metadata is not a frozen contract
A tools/list snapshot from install time is not a lasting grant. After ordinary use, the same server can rewrite what the agent trusts.
Unset is not deny
When the webhook secret is absent, an if-secret check never runs. A warning is not a closed gate.
Memory without provenance is delayed authority
A write that can steer later tool calls is a control-plane event. Without provenance and audit, you cannot govern what fires next.
An isolate is not a host boundary
A V8 isolate or a NodeVM box can still leave the host process writable.
Logging that lies
A green success log is not proof that a policy gate ran.
A2A without a receipt
An authenticated agent call still needs a record of the policy that allowed it.
- Pattern
Skill Grant on the Hop
Enforce skill-level policy at the receiving agent, then bind a durable authorization receipt to the call before it runs.
The guardrail that waved traffic through
A green light on the model path is not containment if the tool can run without that path.
- Pattern
Verified Dispatch Path
A control pattern for security architects and platform teams who run agent harnesses with model-layer guardrails. It treats a green dashboard as worthless unless tool dispatch requires a trusted execution record and an explicit policy decision — so forged history and co-opted judges cannot wave traffic through.
Sandbox on is not containment
A sandbox label does not mean the agent is cut off from the estate. If the host can reach the VPN and the cloud APIs, so can the agent.
- Pattern
Verified Agent Containment
A control pattern for security architects and platform teams who run coding agents or MCP-backed tools on shared hosts. It replaces the comfort label “sandbox on” with an explicit, checkable isolation boundary so the agent cannot silently inherit the estate’s network paths and cloud identity.
Task utility is not harness safety
High task completion can sit next to high attack success. Safety is a property of the model and harness configuration, not the model alone.
Same-origin is not a tool-ownership boundary
The page origin is a DNS name, not a registrant. Ownership has to live in the browser, and the inspector cannot hold tools.
A hook is not a security boundary
If the host is trusted, a contract can prove that an action stopped. It cannot contain a host that never asks.