Most threat intel reports die the moment they’re read. Someone on the team reads a vendor write-up, nods, and moves on — because turning three paragraphs of attacker behavior into a working detection rule takes an hour they don’t have.
That gap — read intel, write rule — is one of the most common failure points in detection engineering. It’s not that teams don’t value intel. It’s that the translation step is manual, tedious, and easy to defer indefinitely. A pipeline that uses an LLM to do the first draft, with a human gate before anything ships, closes that gap without lowering the bar on quality.
This post walks through that pipeline end to end: extraction, mapping to your log schema, rule generation, and — the part most people skip — automated validation before a human ever sees it.
The short version
| Point | Why it matters |
|---|---|
| Extraction is a narrow, structured task | LLMs are reliable at pulling behaviors and IOCs out of prose into a fixed schema — this is not the same as trusting them to reason about your environment |
| Ground rule generation in your actual field names | The model should draft against a schema and real examples, never guess field names from memory |
| Validate before a human reviews | Syntax-check and backtest every draft against historical logs so reviewers only see rules that are plausible, not broken |
Why this fits an LLM well
Detection engineers underuse LLMs here for a fair reason: nobody wants a model inventing detection logic against a live environment. But rule drafting from intel is a narrower problem than that. The report already contains the logic — a named technique, a described sequence of API calls, a list of file hashes or domains. The work is translation, not judgment: turning “the attacker disabled logging before exfiltrating data” into a query against your actual event schema.
That’s a structured extraction and mapping task, which is exactly what current models are good at, as long as you constrain the output format and give them the vocabulary to work with.
Step 1: Extract behaviors and indicators into a fixed schema
Don’t let the model free-write a rule from the raw report text. First pass: extract structured facts only.
{
"technique_id": "T1562.008",
"technique_name": "Disable or Modify Cloud Logs",
"behaviors": [
{
"description": "Logging service stopped or deleted shortly after a new access key is created",
"log_fields_needed": ["actor.identity", "event.action", "event.target", "event.time"]
}
],
"indicators": {
"domains": ["exfil-relay.example"],
"file_hashes": [],
"ip_ranges": []
},
"confidence": "high"
}
Ask for this JSON shape explicitly and reject anything that doesn’t validate against it. This step alone kills most hallucination risk — the model isn’t inventing a rule yet, it’s summarizing text it was given.
Step 2: Map behaviors to your schema, not the model’s guess
Feed the extracted behavior, plus your actual field reference, into the next step. This is the grounding step people skip, and it’s the one that determines whether the output is usable.
prompt_context = {
behavior: extracted.behaviors[0],
available_fields: schema_reference["cloud_audit_log"],
worked_examples: retrieve_similar_rules(extracted.technique_id, k=3)
}
Retrieving two or three existing rules for the same or adjacent MITRE ATT&CK technique gives the model a pattern to follow instead of inventing syntax. This is the same grounding principle that makes retrieval-based triage reliable: never let the model rely on memorized field names when you can hand it the real ones.
Step 3: Generate the draft rule
With the behavior, schema, and examples in context, generate a draft in your detection format of choice — Sigma is the practical default since it stays portable across backends:
title: Cloud logging disabled shortly after new access key creation
status: experimental
logsource:
category: cloud_audit
detection:
key_created:
event.action: "CreateAccessKey"
logging_disabled:
event.action:
- "StopLogging"
- "DeleteTrail"
timeframe: 15m
condition: key_created followed by logging_disabled
level: high
tags:
- attack.t1562.008
Notice this stays generic — Sigma’s logsource and detection blocks compile down to whatever query language your backend actually speaks. The model never needs to know your backend’s syntax, which keeps the pipeline portable if you ever change platforms.
Step 4: Validate before a human ever sees it
This is the gate that makes the pipeline trustworthy. Two checks, both automated:
- Syntax validation — parse the rule against the Sigma spec (or your rule format’s schema). Reject anything malformed.
- Backtest against real history — replay the rule against the last 30 to 90 days of logs. A rule that fires thousands of times an hour is either too broad or mapped to the wrong field. A rule that never fires against known-benign traffic and also never fires against a red-team replay of the technique is likely broken in the other direction.
result = backtest(rule, window="30d")
if result.match_rate > noise_threshold:
return "reject: too broad, refine field scoping"
if result.match_count == 0 and replay_available:
return "reject: rule did not fire against simulated technique"
route_to_review_queue(rule, result.match_samples)
Only rules that pass both checks land in front of an analyst, alongside a sample of what they’d actually match. That’s the difference between “here’s an idea” and “here’s something ready to approve.”
Where humans stay in the loop
Nothing in this pipeline auto-deploys. The output at every stage is a draft: extracted facts, a mapped rule, backtest results. A detection engineer reviews the final rule with its match samples and decides whether to tune, approve, or discard it. The LLM’s job is to remove the hour of transcription work between “we read the report” and “we have something to review” — not to remove the review.
Teams that adopt this pattern tend to see the biggest gain in coverage of second-tier intel: the reports that are clearly relevant but never urgent enough to justify blocking an engineer’s day. Those are exactly the ones that pile up unactioned, and exactly the ones a constrained, validated pipeline can clear quickly.
Final thought
The risk with LLMs in detection engineering was never that they can’t write a plausible-looking rule — they can, easily. The risk is shipping a plausible-looking rule that’s wrong. Extraction into a fixed schema, grounding in your real field names, and a mandatory backtest before human review turn “plausible” into “verified.” That’s the bar that makes this worth running in production, not just as a demo.
If you want to see how this kind of pipeline fits into a managed detection workflow, contact us.