Detecting Malicious AI Agent Activity: A Guide for Detection Engineers

Your SOC spent the last two years learning to use AI. Now it needs to learn to watch it.

Every team we talk to has at least one AI agent running in production somewhere — a coding assistant with repo access, a support bot that can issue refunds, an internal copilot that can query customer data. Almost none of them log what those agents actually did. That’s the gap attackers are starting to use.

The pattern showing up in the field this year isn’t attacks on the model. It’s attacks through the model — a poisoned document, a malicious ticket, or a compromised tool response that gets the agent to do something its owner never approved. The model itself is rarely the vulnerable part. The tool calls it makes are.

The short version

Why this is a detection engineering problem, not just an AI problem

Frameworks like MITRE ATLAS and the OWASP Top 10 for LLM Applications both converge on the same finding: the highest-impact failures are indirect prompt injection and excessive agency, not model jailbreaks. An attacker doesn’t need to “hack” your agent. They plant instructions in content the agent will read — a web page, a PDF attachment, an email, a ticket comment — and wait for the agent to act on them with whatever permissions it holds.

That means the defense isn’t a better model. It’s the same discipline you already apply to human identities: log every action, know what normal looks like, and alert on deviation. The agent is just a new kind of actor in your environment, and it needs an audit trail like any other.

Step 1: Log tool calls like you log API calls

If your agent framework doesn’t emit a structured event per tool invocation, add a logging wrapper before you do anything else. At minimum, capture:

{
  "timestamp": "2026-08-31T14:02:11Z",
  "agent_id": "support-copilot",
  "session_id": "sess_9f2a",
  "actor": "customer_facing_bot",
  "tool_name": "query_customer_records",
  "tool_input": { "customer_id": "C-88214", "fields": ["email", "plan"] },
  "tool_output_hash": "sha256:...",
  "tool_output_size_bytes": 412,
  "source_content_origin": "support_ticket#44210",
  "approved_scope": ["query_customer_records", "issue_refund"],
  "outcome": "success"
}

Two fields do most of the work here: source_content_origin, which tells you what untrusted content the agent was processing right before it decided to act, and approved_scope, which tells you what that agent is actually supposed to be able to do. Almost every useful detection below is built from those two fields plus the tool name.

Ship this event stream into your SIEM the same way you’d ship CloudTrail or identity provider logs — as its own log type, normalized, queryable, and retained long enough to support an investigation.

Step 2: Detect scope escalation

An agent calling a tool outside its approved scope is the agentic equivalent of privilege escalation. It’s cheap to detect and catches a large share of real incidents:

SELECT agent_id, tool_name, count(*) AS calls
FROM agent_tool_calls
WHERE timestamp > now() - interval '1 hour'
  AND NOT has(approved_scope, tool_name)
GROUP BY agent_id, tool_name
HAVING calls > 0

Any hit here is worth a P1 alert. There’s no legitimate reason for an agent to invoke a tool it was never granted.

Step 3: Detect injection via untrusted content

This is the harder, higher-value detection: an agent reading untrusted input and then taking a sensitive action within the same session, with no corresponding action in its normal baseline.

for each session:
  if session contains a read from an untrusted source
     (email, ticket, uploaded doc, third-party API response)
  and session later contains a sensitive tool call
     (send_email, transfer_funds, modify_permissions, export_data)
  and this read -> sensitive-call sequence has not occurred
     for this agent_id in the last 30 days
  then flag: possible indirect prompt injection

The key design choice: don’t try to detect injection in the text itself. Detect the behavioral consequence — untrusted input immediately followed by an unusual privileged action. That’s far more durable than pattern-matching on prompt content, which attackers rotate constantly.

Step 4: Baseline tool-call sequences per agent

Treat each agent like a service account with a known job. Build a simple baseline of which tools it calls, in what order, and at what volume per session, then alert on drift:

You don’t need a machine learning pipeline for this. A rolling seven-day baseline per agent_id and a set of threshold rules catches the majority of real-world agent abuse, and it’s the same approach you already use for identity-based detection.

Step 5: Watch the tools, not just the agent

If your agents connect to external tool servers, log the connection and registration events too — not just the calls. A compromised or spoofed tool server is functionally identical to a malicious insider: it can return poisoned content to every agent that trusts it. Alert on any new tool server being added to an agent’s allowed list outside of your change process, and on any tool server whose responses suddenly change shape or size.

Final thought

AI agents are going to keep getting more access, not less — that’s the whole point of deploying them. The organizations that stay ahead of this aren’t the ones with the most sophisticated model, they’re the ones treating every agent as an identity with an audit trail, a scope, and a baseline. Start by logging tool calls. Everything else in this post builds on that one decision.

If you want help wiring agent tool-call logs into your existing detection pipeline, contact us.