The bottleneck in detection engineering was never writing the rule. It was reading the advisory, mapping it to your log fields, and figuring out if it would fire ten thousand times a day before you shipped it.
Every week brings a new CTI report, a vendor advisory, or a write-up of a fresh technique. Most of it never becomes a detection rule — not because the technique isn’t worth catching, but because turning three paragraphs of prose into working logic against your actual schema takes an hour a detection engineer doesn’t have. LLMs are a genuinely good fit for that translation step. They are a bad fit for deciding what ships to production unsupervised. This post covers a workflow that uses them for the first job and never the second.
The short version
| Step | What the model does | What a human does |
|---|---|---|
| Draft | Reads the advisory, proposes detection logic against your schema | Nothing yet |
| Validate | Nothing | Backtests the draft against real log history |
| Review | Explains its reasoning and known blind spots | Approves, edits, or rejects |
| Ship | Nothing | Deploys, sets severity, owns the rule |
The model never writes directly to your rule store. It writes to a staging queue that a human has to clear.
Why this is a good LLM task in the first place
Reading an advisory and mapping it to a schema is exactly the kind of work large language models are strong at: unstructured text in, structured output against a known schema out. A typical advisory paragraph looks like this:
The actor authenticates with a valid but dormant service account, then
lists all IAM roles and immediately assumes one with administrative
scope, from an ASN never previously seen for that account.
That’s three fields and a join condition once you know your event schema: an auth event, a list-roles call, an assume-role call, all from the same principal within a short window, with an ASN not in that principal’s history. A model that has been given your field names can turn that into a first-draft query in seconds. A human doing the same thing is mostly just typing field names — the model is not replacing judgment, it’s replacing typing and schema lookup.
Step 1: constrain the draft to your actual schema
The single biggest failure mode is a model inventing field names that don’t exist in your event store. Don’t let it guess. Pass the schema explicitly as part of the prompt, and ask for output in a fixed intermediate format rather than a specific query language — you’ll compile it yourself.
SYSTEM:
You draft detection logic. Output only the JSON schema below.
Never reference a field not listed in AVAILABLE_FIELDS.
If the advisory can't be expressed with these fields, say so and stop.
AVAILABLE_FIELDS:
principal_id, principal_type, event_name, source_asn,
source_ip, auth_method, mfa_used, target_resource, timestamp
OUTPUT_SCHEMA:
{
"conditions": [ {"field": str, "op": str, "value": any} ],
"join_key": str,
"window_seconds": int,
"confidence_notes": str,
"known_blind_spots": str
}
That known_blind_spots field matters more than it looks. Ask the model to name what its own logic would miss — attacker behavior it isn’t checking for, edge cases in the advisory it glossed over. Models are noticeably better at critiquing a draft than a blank page, and this forces that critique to happen before a human even looks at it.
Step 2: backtest before a human reviews the logic, not after
Don’t hand a reviewer raw draft logic and ask “does this look right?” Run it against 30 to 90 days of real history first and hand them the hit count and a sample of matches instead.
draft_rule -> compile_to_query()
-> run_against_log_history(days=30)
-> if hits > threshold: flag "likely noisy, needs narrowing"
-> if hits == 0: flag "never fired historically, confirm logic reaches real events"
-> attach sample_matches[0:10] for reviewer
A rule that would fire 4,000 times a day is a tuning problem, not a detection. Surfacing that before review turns a 20-minute logic review into a 2-minute “reject, too noisy” decision — the reviewer’s time goes to the rules that are actually close to correct.
Step 3: the review gate is not optional, and it’s not a formality
The reviewer isn’t rubber-stamping. They’re checking three things a model can’t verify itself: whether the advisory’s technique actually applies to your environment, whether the proposed severity matches your existing rule set, and whether the blind spots the model flagged are acceptable or need a second rule to cover. Log the outcome — approved as-is, edited, or rejected and why — in the same place you keep your rule history. That log becomes the training signal for tightening the prompt over time: if a category of draft keeps getting rejected for the same reason, that’s a prompt fix, not a one-off correction.
Where this breaks if you skip a step
Skip the schema constraint and you get plausible-looking rules referencing fields that don’t exist — they compile, they never fire, and nobody notices for months. Skip backtesting and reviewers approve rules on how reasonable the logic reads, not how it behaves against real traffic — which is how a “P1 critical” rule ends up paging someone 200 times in its first week. Skip the review gate and you’ve just moved the alert-fatigue problem one step upstream, from noisy alerts to noisy rules generating them automatically.
Final thought
The efficiency gain here is real — advisories that used to sit in a backlog for weeks can go from read to backtested draft in an afternoon. But the gain comes from removing the tedious middle step, not from removing the judgment on either end. Someone still decides what’s worth detecting, and someone still decides what ships. The model just stops being the reason that decision takes so long to get to.