Detection-as-Code: Treating Your Alert Rules Like Production Software

A detection rule that nobody tested, nobody reviewed, and nobody can explain is not a control. It is a liability wearing a control’s badge.

Most teams start detection engineering the same way: someone opens the alerting UI, writes a query, clicks save. It works fine for the first ten rules. By rule fifty, nobody remembers why half of them exist, two are silently broken, and the on-call engineer is afraid to touch any of them before a big customer demo. That is not a tooling problem. It is a process problem, and the fix looks a lot like how you already manage application code.

The point-and-click ceiling

Rules edited directly in a console share three failure modes once a team scales past a handful of analysts:

• No history. Someone tightens a threshold at 2 a.m. to kill a noisy alert, and six months later nobody knows the rule used to catch something real.
• No review. A typo in a field name silently turns a detection into a rule that never fires. Nothing tells you until an incident review asks “why didn’t we catch this?”
• No regression testing. You fix a false positive for one log source and break detection for a related one you didn’t think to check.

None of this is a UI problem specifically — it’s what happens any time production logic lives outside version control.

What detection-as-code actually means

The idea is simple: detection logic lives as text files in a git repository, next to fixtures that describe the events each rule should and should not match. A pipeline lints, tests, and promotes changes the same way it would for a backend service.

detections/
├── rules/
│   ├── identity/
│   │   ├── root_console_login_new_geo.yaml
│   │   └── mfa_disabled_then_login.yaml
│   └── cloud/
│       └── guardrail_disabled.yaml
├── fixtures/
│   ├── root_console_login_new_geo.cases.yaml
│   └── guardrail_disabled.cases.yaml
└── pipeline.yaml

Each rule file is plain, declarative logic — field conditions, thresholds, a severity, a MITRE ATT&CK mapping. Nothing here is tied to a specific query language; the same pattern works whether you’re writing Sigma, a vendor DSL, or a small Python function.

# rules/identity/root_console_login_new_geo.yaml
id: root_console_login_new_geo
severity: critical
mitre: T1078 - Valid Accounts
condition:
  all:
    - field: user_identity_type
      equals: "Root"
    - field: event_name
      equals: "ConsoleLogin"
    - field: src_country
      not_in: ["known_admin_countries"]

Unit tests for detection logic

This is the part most teams skip, and it’s the part that pays off first. Every rule ships with fixtures: sample events labeled as should-match or should-not-match.

# fixtures/root_console_login_new_geo.cases.yaml
- name: root_login_from_unexpected_country
  event:
    user_identity_type: "Root"
    event_name: "ConsoleLogin"
    src_country: "RU"
  expect: MATCH

- name: root_login_from_known_admin_location
  event:
    user_identity_type: "Root"
    event_name: "ConsoleLogin"
    src_country: "IL"
  expect: NO_MATCH

- name: non_root_console_login_ignored
  event:
    user_identity_type: "IAMUser"
    event_name: "ConsoleLogin"
    src_country: "RU"
  expect: NO_MATCH

A test runner loads every rule, evaluates it against every fixture, and fails the build if a single expectation is wrong. This catches the two most common ways detections rot: a rule that stops matching after a schema change upstream, and a rule that starts matching everything after a careless edit.

detect-test run --rules ./rules --fixtures ./fixtures
# 42 rules, 118 fixtures — 118 passed, 0 failed

The pipeline

Once tests exist, the promotion path looks like any other deploy pipeline:

lint  →  unit test  →  shadow deploy  →  promote to production

Lint catches structural problems: missing severity, unmapped ATT&CK technique, a condition referencing a field that doesn’t exist in your schema. Unit test runs every fixture. Shadow deploy runs the new or changed rule against live traffic without alerting anyone, so you can compare its match rate to expectations before it can page someone. Promote merges to the branch your production detection engine reads from.

Rollback is the underrated benefit

The pipeline above pays for itself the first time a rule change causes an incident — a threshold tightened too far that goes silent, or a broadened condition that floods the queue. Instead of reconstructing what changed from memory, you revert the commit and redeploy. The fixture set also becomes living documentation: reading the test cases tells a new analyst exactly what a rule is meant to catch, faster than reading the condition logic alone.

None of this requires a specific vendor. It requires treating your detection rules as what they are: code that decides whether your team gets paged. Give it the same discipline you’d give anything else that ships to production, and the false-positive fire drills mostly go away on their own.

If you want a second opinion on how your current detection rules are managed and tested, contact us.