Every SIEM migration re-teaches the same lesson: the detection logic was never the hard part. Rewriting it in a new query language is.
The problem: every backend speaks its own language
A detection engineer who knows how to spot a root account login from an unexpected ASN has to express that logic differently depending on where it runs. In Splunk it’s SPL. In Elastic it’s a KQL or EQL query. In a homegrown pipeline it might be a Python function that inspects a JSON event. The underlying idea — “flag it when user_identity_type is Root and the event isn’t a normal console login” — never changes. The syntax around it does, every time.
That mismatch is expensive in three specific ways:
- Vendor lock-in. Years of detection content become a migration project the moment you switch platforms.
- No shared detection content. A rule a threat intel team publishes for one SIEM can’t be dropped into another without a rewrite.
- Duplicated engineering. Teams running both a real-time engine and a batch/scheduled query layer end up writing the same logic twice, in two languages, and keeping both in sync by hand.
Sigma is the open-source answer: a YAML-based, generic signature format for log events, with a converter (pySigma / sigma-cli) that compiles a single rule into dozens of query backends.
The short version
| Point | Why it matters |
|---|---|
| Sigma separates detection logic from query syntax | Write the rule once, target Splunk, Elastic, or a custom backend without touching the logic |
| SigmaHQ maintains 3,000+ community rules with MITRE ATT&CK tags | You get baseline coverage for free instead of starting from zero |
pysigma backends are pluggable | If your SIEM isn’t supported, you write one backend class, not thousands of rules |
| It doesn’t solve aggregation-heavy detections well | Rate-based and stateful rules (baselines, brute force) still need a real query engine behind them |
Anatomy of a Sigma rule
Here’s a rule for the CloudTrail root-activity detection almost every AWS-monitoring team eventually writes:
title: AWS Root Account Activity Outside Console Login
id: 7c1e2b6a-9f3d-4a2e-8b7a-3e1f2c9d4a11
status: stable
description: Detects any AWS API call made by the root account other than a normal console login.
references:
- https://attack.mitre.org/techniques/T1078/004/
author: Xpernix Detection Engineering
date: 2026-07-20
tags:
- attack.privilege-escalation
- attack.t1078.004
logsource:
product: aws
service: cloudtrail
detection:
selection:
userIdentity.type: Root
filter:
eventName: ConsoleLogin
condition: selection and not filter
falsepositives:
- Legitimate root usage during account recovery or billing changes
level: critical
Three things worth calling out:
logsourcedescribes what kind of data the rule applies to (product + service), not the vendor. The backend maps that to an actual index, sourcetype, or table name.detectionis a set of named selections combined with a booleancondition. No query syntax, no field-quoting rules, no vendor-specific operators.tagscarry the MITRE ATT&CK mapping directly in the rule (attack.t1078.004— Valid Accounts: Cloud Accounts). That mapping now travels with the rule everywhere it’s compiled.
Compiling it: one rule, multiple backends
Install the CLI and pull the community rule pack:
pip install sigma-cli
sigma plugin install splunk elasticsearch
git clone https://github.com/SigmaHQ/sigma.git sigma-rules
Compile the rule above to Splunk SPL:
sigma convert -t splunk -p aws \
--without-pipeline \
rules/aws_root_activity.yml
(userIdentity.type="Root") NOT (eventName="ConsoleLogin")
Compile the same rule to an Elastic Lucene query:
sigma convert -t elasticsearch -p ecs_cloudtrail \
rules/aws_root_activity.yml
userIdentity.type:"Root" AND NOT eventName:"ConsoleLogin"
Notice what didn’t change: the rule file. Only the target (-t) and the field-mapping pipeline (-p) did. The pipeline is the piece that knows your specific schema — for example, that Elastic’s ECS-mapped CloudTrail index calls the field aws.cloudtrail.user_identity.type instead of userIdentity.type.
Bringing it to a stack that isn’t natively supported
Most teams running ClickHouse, Loki, or an in-house pipeline won’t find an official pysigma backend. That’s fine — the SDK is built to be extended. A minimal ClickHouse backend only needs to implement how Sigma’s internal condition tree renders to WHERE clauses:
from sigma.conversion.base import TextQueryBackend
from sigma.conversion.state import ConversionState
from sigma.rule import SigmaRule
from sigma.conditions import ConditionFieldEqualsValueExpression
class ClickHouseBackend(TextQueryBackend):
eq_token = " = "
and_token = " AND "
or_token = " OR "
not_token = "NOT "
field_quote = "`"
str_quote = "'"
def convert_condition_field_eq_val_str(self, cond, state: ConversionState) -> str:
return f"`{cond.field}` = '{cond.value}'"
That’s a simplified sketch, not the full class — pysigma’s TextQueryBackend handles most operators for you, so a working backend for a SQL-like engine is typically 60-100 lines. Once it exists, every rule in the community pack (and every rule your team writes going forward) compiles straight to a WHERE clause you can drop into a scheduled query:
SELECT event_time, event_name, actor_arn, source_ip
FROM cloudtrail_events
WHERE `userIdentity.type` = 'Root'
AND NOT `eventName` = 'ConsoleLogin'
AND event_time > now() - INTERVAL 1 HOUR
That last query is functionally identical to the kind of single-event check a real-time rules engine evaluates — the difference is Sigma generated the WHERE clause instead of a human hand-writing it for the third time this quarter.
Where Sigma runs out of road
Sigma is deliberately scoped to single-event, stateless detection. It’s honest about what it doesn’t do:
| Detection type | Sigma fit | Why |
|---|---|---|
| Single-event signature (root login, disabled GuardDuty, known-bad process) | Strong | Maps directly to selection + condition |
| Field correlation within one event | Strong | near and multi-selection conditions cover most cases |
| Rate-based (5 failed logins in 2 minutes) | Weak | Sigma’s correlation rules are newer and backend support is inconsistent |
| Baseline/anomaly (user logging in from a new country) | Not supported | Requires historical state Sigma’s format has no concept of |
That split maps cleanly onto the real-time-vs-scheduled detection split most SIEM architectures already have. Use Sigma to generate the stateless, single-event rules that run against your streaming layer. Keep the stateful, aggregation-heavy detections — brute force, baselining, rare-value analysis — as hand-written scheduled queries where you have full control over windowing and state. Trying to force a rate-based rule into Sigma’s condition grammar produces something more fragile than just writing the SQL.
Final thought
Sigma won’t replace a detection engineer’s judgment about what’s worth alerting on. What it removes is the tax of re-encoding that judgment every time a query language changes underneath it. For any team maintaining detection logic across more than one backend — or planning to migrate off one — that’s hours back every week, and a rule library that outlives whatever platform it started on.
If you’re evaluating how to structure detection content across a real-time and scheduled pipeline, contact us — it’s the kind of architecture decision worth getting right before you’ve written three hundred rules on top of it.