How to Build an LLM-Powered SOC Alert Triage Pipeline

The bottleneck in most SOCs isn’t detection coverage. It’s the ten minutes an analyst spends per alert just gathering context before they can decide anything.

Every vendor is shipping an “AI SOC analyst” this year. Most of them are a chat window bolted onto a ticket queue. The useful version is narrower and less flashy: a pipeline that takes a raw alert, pulls the context a human would pull, and produces a structured verdict an analyst can accept or override in seconds — not a paragraph they have to re-verify from scratch.

This post walks through that architecture: what an LLM triage step actually does well, how to keep it from hallucinating a verdict, and where to draw the human-in-the-loop line.

The short version

Where LLMs actually help

An LLM is good at exactly one thing in this pipeline: turning unstructured or semi-structured signal into a structured judgment call, fast. It is good at reading an alert plus a pile of contextual JSON and asking “does this pattern match a known-benign baseline, or not.”

It is bad at three things people keep asking it to do anyway:

• Remembering facts across calls. It doesn’t know your environment. It knows what’s in the prompt.
• Deciding severity from vibes. Without a rubric, two calls on the same alert can disagree.
• Being the source of truth for “did this happen.” It summarizes evidence; it doesn’t generate it.

Design around those limits instead of hoping a bigger model fixes them.

Architecture: two-tier triage

Run a cheap, fast model as a first-pass filter, and reserve a stronger model for alerts that need real investigation. This mirrors how a human SOC works — a tier-1 analyst does the initial read, escalates what’s ambiguous.

Alert fires (Risk Service / scheduled check)
   -> Tier 1: fast triage model
        - classify: benign_known_pattern | needs_review | escalate
        - grounded in: entity history, rule metadata, prior verdicts
   -> if needs_review or escalate:
        Tier 2: investigation model
        - pulls raw events, related alerts, asset/identity context
        - produces: summary, MITRE mapping, recommended action
   -> result written back to the SIEM with a confidence score
   -> analyst accepts / overrides -> feedback recorded

The tier-1 classifier should run on nearly every alert — its job is to kill the 60-80% that match a known-benign pattern (a service account’s regular API calls, a recurring false positive from a specific rule) without ever touching the expensive model.

A minimal tier-1 prompt template, kept deliberately narrow:

TRIAGE_SYSTEM_PROMPT = """
You are a SOC tier-1 triage assistant. You classify security alerts using
ONLY the data provided in the prompt. You do not know anything about this
environment beyond what is given to you.

Output one of exactly three verdicts:
- benign_known_pattern: matches a documented baseline for this entity/rule
- needs_review: insufficient context to rule out malicious activity
- escalate: matches a high-confidence indicator of compromise

Always cite which piece of provided context drove your verdict.
Never invent entity history, IP reputation, or prior alerts not in the prompt.
"""

def build_triage_prompt(alert: dict, entity_context: dict, rule_baseline: dict) -> str:
    return f"""
Alert:
{json.dumps(alert, indent=2)}

Entity history (last 30 days, from SIEM):
{json.dumps(entity_context, indent=2)}

Rule baseline (known false-positive patterns for this rule):
{json.dumps(rule_baseline, indent=2)}
"""

The system prompt does two jobs: it constrains the output to a fixed set of verdicts you can route on programmatically, and it explicitly forbids inventing context. That second part matters more than it looks — it’s the difference between a triage tool and a hallucination generator.

Grounding: retrieval before generation

The single biggest failure mode in LLM-based triage is a model asserting something about the environment that isn’t true — “this IP has no prior history” when it actually fired three alerts last week. The fix isn’t a better prompt. It’s making sure the model never has to guess, because you handed it the answer.

That means a retrieval step runs before every triage call, not after:

def gather_context(alert: dict, ch_client) -> dict:
    entity = alert["source_ip"] if alert.get("source_ip") else alert["actor"]

    prior_alerts = ch_client.query("""
        SELECT rule_name, severity, verdict, created_at
        FROM siem_alerts
        WHERE entity = %(entity)s
          AND created_at > now() - INTERVAL 30 DAY
        ORDER BY created_at DESC
        LIMIT 20
    """, {"entity": entity})

    rule_history = ch_client.query("""
        SELECT verdict, count() AS n
        FROM siem_alerts
        WHERE rule_name = %(rule)s
        GROUP BY verdict
    """, {"rule": alert["rule_name"]})

    return {"prior_alerts": prior_alerts, "rule_history": rule_history}

This is the part teams skip when they’re moving fast, and it’s the part that determines whether the system is trustworthy in month two. A model with no grounding will sound just as confident whether it’s right or wrong — retrieval is what makes confidence correlate with correctness.

Closing the feedback loop

Every triage verdict an analyst overrides is a labeled training example you’re currently throwing away. Log it:

Feed disagreements back into the rule baseline used at retrieval time — not by fine-tuning the model, which is slow and heavy for most teams, but by updating the structured context the next call gets. If analysts keep marking a specific service account’s S3 access as benign, that becomes part of rule_baseline for that rule, and the model stops flagging it without anyone touching a prompt.

This is the mechanism that turns “an LLM that sometimes helps” into a system that gets measurably better at your specific environment over time — the model doesn’t improve, the context it’s grounded in does.

Where to keep humans in the loop

Automate the classification. Don’t automate the response.

The one line worth being strict about: an LLM can recommend isolating a host or disabling a credential. It should never execute that action directly. Keep the write path to production systems behind an explicit human approval, even when the model’s confidence is high — the cost of a false-positive containment action (locking out a legitimate admin mid-incident) is usually worse than the cost of a slightly slower response.

Final thought

The pitch for AI in the SOC isn’t “replace the analyst.” It’s “stop making the analyst do the same ten minutes of context-gathering for the two hundredth time this week.” Build the pipeline around retrieval and structured verdicts, keep the model narrow and grounded, and measure it against human overrides from day one. That’s a system that earns trust instead of asking for it.

If you want help wiring AI-assisted triage into your detection pipeline without turning it into another thing your team has to babysit, contact us.