How to Continuously Validate Detection Rules with Open-Source Adversary Emulation

A detection rule you haven’t tested since the day you wrote it isn’t a control. It’s a belief.

Most teams find out a rule stopped firing during the incident it was supposed to catch. A log field got renamed upstream. A parser silently dropped a source. Someone “temporarily” disabled a noisy rule six months ago and never turned it back on. None of that shows up in a dashboard that only tells you alerts are quiet — quiet looks identical whether nothing bad happened or your detection pipeline is broken.

Continuous detection validation closes that gap: instead of trusting that a rule works because you wrote it correctly once, you regularly generate the exact behavior the rule is supposed to catch and check that an alert actually comes out the other end.

The short version

Why “the rule looks right” isn’t enough

A detection rule has four places it can silently fail, and code review only checks one of them:

• The technique isn’t emulated correctly — you tested the wrong behavior.
• The log source doesn’t capture the technique — the agent or integration never records the field the rule depends on.
• The parsing/normalization step drops or renames a field — the rule references a field name that no longer exists after a schema change.
• The rule logic itself is wrong — the part everyone actually reviews.

Only the last one shows up when someone reads the rule. The other three show up when a real attacker does the thing and nothing happens.

Open-source tools that generate the signal

You don’t need to build attack emulation yourself. Two mature, freely available projects do the hard part — safely executing a known technique and mapping it to MITRE ATT&CK:

• Atomic Red Team — a library of small, scoped “atomic tests,” one or more per ATT&CK technique, each with an explicit cleanup step. Good for testing a single technique on demand.
• MITRE Caldera — an adversary emulation platform for chaining multiple techniques into a scenario, useful once single-technique validation is solid and you want to test detection across a multi-step attack path.

Start with Atomic Red Team. It maps cleanly to “one rule, one technique,” which is the granularity you want for a validation pipeline.

Building the validation loop

The pipeline has four stages, and none of them are exotic:

pick technique (ATT&CK ID)
  -> run the atomic test in an isolated account/host
  -> wait for the event to land in your log pipeline
  -> query your detection engine for the expected alert
  -> assert: alert fired, correct rule_id, within expected latency
  -> record result + cleanup

A minimal runner, in pseudo-code, ties an ATT&CK technique to the rule that should catch it and the atomic test that should trigger it:

# validation_map.yaml
- technique_id: T1078.004
  rule_id: cloud_root_activity_outside_console_login
  atomic_test: "Atomics/T1078.004/T1078.004.yaml#test-1"
  expected_latency_seconds: 120

- technique_id: T1562.001
  rule_id: security_logging_disabled
  atomic_test: "Atomics/T1562.001/T1562.001.yaml#test-3"
  expected_latency_seconds: 300

And the check itself, stripped down to the shape that matters:

def validate_technique(mapping: dict, detection_client, atomic_runner) -> dict:
    run_id = atomic_runner.execute(mapping["atomic_test"])

    deadline = time.time() + mapping["expected_latency_seconds"]
    alert = None
    while time.time() < deadline:
        alert = detection_client.find_alert(
            rule_id=mapping["rule_id"],
            correlation_id=run_id,
        )
        if alert:
            break
        time.sleep(5)

    atomic_runner.cleanup(mapping["atomic_test"])

    return {
        "technique_id": mapping["technique_id"],
        "rule_id": mapping["rule_id"],
        "fired": alert is not None,
        "latency_seconds": alert["latency"] if alert else None,
    }

Run this on a schedule — nightly is a reasonable default — against a dedicated test account or tenant, never against production data. The correlation_id matters more than it looks: without a way to tie a specific alert back to a specific test run, you’re eyeballing whether “an alert that looks about right” showed up, which defeats the point.

Tracking coverage, not just pass/fail

The output that’s actually useful to a team isn’t “12 of 15 tests passed last night.” It’s a coverage matrix that tells you what’s validated, what’s stale, and what was never tested at all:

That last row is the one most teams don’t have until they build this. A gap in ATT&CK coverage that nobody wrote a rule for is a very different problem from a rule that exists and quietly broke — but a plain rule inventory can’t tell them apart, and a coverage matrix can.

Where humans still have to be involved

Automated adversary emulation is not something you point at a production account and walk away from:

• Run it in an isolated account or tenant with its own IAM boundary, never against customer or production data.
• Every atomic test needs a cleanup step, and you should verify cleanup ran — a validation pipeline that leaves a backdoor account behind because a script errored out mid-test is its own incident.
• Gate any test that touches something destructive (disabling logging, deleting resources) behind explicit approval, even in a test account — the point is confidence in detection, not a second source of chaos.
• New techniques still need a human to decide they’re worth building a rule for. Emulation validates coverage you’ve already committed to; it doesn’t replace the judgment call about what’s worth detecting.

Final thought

Most detection engineering effort goes into writing rules and almost none into proving they still work. That asymmetry is exactly backwards, because rules don’t break when you write them — they break quietly, months later, when a schema changes or an integration gets reconfigured. A validation pipeline built on open-source adversary emulation turns “we have a rule for that” into something you can actually stand behind, on a schedule, without waiting for a real incident to be the test.

If you’re building out detection coverage and want a second pair of eyes on how to structure validation against it, contact us.