Detections should have tests like code does

A detection rule that someone edited in a console has no review, no rollback, and no test. It can stop firing after an EDR upgrade or a parser change, and nobody finds out until the next purple team cycle scores it as a miss. Version 1.2 of the Detection Evidence Framework adds a section on fixing that: ship backlog fixes as detection as code and runbook as code.

I wrote the framework as an open teaching document under CC BY 4.0. Its premise fits on one line. Purple teaming is the method. Evidence is the product. This post walks through what the cycle asks for, what changed in 1.2, and what the new section is honest about not proving.

The cycle in brief

The framework runs an engagement as six phases: scope, select procedures, safety gate, execute and observe, score, then backlog and report. The safety gate is a signature. The Engagement Lead and the Control Owner sign before anything runs, and the framework says not to skip that phase or the backlog phase.

Scoring is where most programs drift. The framework uses six outcome labels per sub-technique or named procedure: Prevented, Detected-actionable, Detected-weak, Logged only, Missed, and Not testable. Those labels are never averaged into a single purple score. The framework's reason is short: averages hide misses. A report that says coverage improved tells an executive nothing about the one path where telemetry existed and no alert fired.

Every weak hit or miss you choose to fix becomes a backlog item with a named owner, a priority with a one-line reason, a status, and an acceptance test. The acceptance test is the exact purple retest that will prove the fix, and it gets scheduled. Without it, the framework says, "fixed" means "someone believes they deployed a rule." I have watched that belief pass for evidence in plenty of status meetings. A fix is closed when the retest passes.

Detection as code

Section 6.1 starts from a plain observation. A backlog item closed by a console edit is hard to review, hard to roll back, and easy to break by accident later. So where your tooling allows it, close the item with a change that lives in version control.

For detections, that means a few concrete habits:

• Store each rule (Sigma, a SIEM query, an EDR custom rule, whatever your platform supports) in a repository with an owner, the ATT&CK sub-technique or named procedure it covers, and the backlog ID that created it.
• Review changes by pull request, so a second person checks logic, scope, and noise before deployment.
• Deploy through a pipeline, so production matches what was reviewed and a bad change can be rolled back.
• Attach a test to every rule: a replay of the purple procedure, or sample telemetry, that must produce the expected alert.
• Run those tests on a schedule and after platform changes such as a new EDR version, a parser change, or a SIEM migration.
• Retire or rewrite rules that no longer fire, no longer apply, or produce only noise, and record why.

The test line is the one I care about most. The framework calls the rule's test the machine-readable form of the acceptance test. The backlog already demanded a retest. Code lets you run that retest on Tuesday after a vendor update instead of waiting months for the next cycle to discover the break. A silent failure shows up as a red test before it shows up as a miss.

The acceptance test field in the backlog now says so directly: when practical, write it as an automated test.

Runbook as code

Response gets the same treatment. The framework asks you to write the response to a procedure as an executable, or at least version-controlled, playbook: triage questions, enrichment steps, containment options, escalation, and who approves what.

The sequencing matters. Automate enrichment and low-risk steps first. Keep a named human approval gate on any step that disrupts users, systems, or patient or payment operations. Log every run: what ran, who or what approved it, and the result. Those logs feed the evidence package the framework already asks you to retain.

Then rehearse the runbook in the next purple cycle. This is the other small change in 1.2. The Response field captured during execute and observe now records the runbook name and version, or notes that no runbook existed. When a response is slow, you can tell whether the runbook was wrong, out of date, or missing, and the version ties the result to a specific change you can review.

When an AI agent runs a step

More teams are putting AI agents into response workflows. The framework does not treat that as a special category with its own rules. An agent that executes any runbook step gets the same rules as any other automation, plus three additions: least privilege, a human approval gate for consequential actions, and a complete action log.

It also asks you to test the agent's behavior with the same shared-scenario method as any other control. That is the right default. If offense and defense must work the same scenario to prove a detection, they should work the same scenario to prove what an agent does with the alert. An agent with broad permissions and no log is an unreviewed change running in production. For leaders deciding how far to trust AI in security operations, this is a readiness question before it is a tooling question, and it is the kind I write about at carbene.ai.

What a passing test does not prove

The section I would read twice is the limits paragraph. A passing automated test shows the detection fires on the replayed behavior in your test conditions. It does not show that an adaptive attacker will be detected, that the telemetry source is complete in production, or that analysts will act on the alert.

So the framework keeps scoring real outcomes with the six labels. Automated replay supports the retest. It does not replace execute and observe. A green pipeline is a regression check. Detected-actionable still means a signal fired in time and someone who could stop the activity was able to act on it.

The derivation behind the section is two sentences I keep coming back to. Purple teaming proves a detection worked on one day. Code keeps it provable on the next.

Where to start

The framework's teaching default for adoption is deliberately small. Take the Critical and High backlog items from one cycle. Convert those detections and their runbooks to code, attach a test to each, and run the retest from the pipeline. Expand only when that works. If you have no pipeline, a repository with reviewed rules and a manual deploy checklist already gives you history and review.

Version 1.2 leaves the outcome labels, the scoring unit, and the backlog fields unchanged. Sector sections for healthcare, financial services, and public safety technology still apply the same cycle. The full text is on GitHub. For hands-on purple team and tabletop practice, that work lives at Cyber Defense Tactics. If you want to rehearse what happens when an AI agent sits in the response path, the free AI tabletop exercise kit has one ready-to-run scenario with a facilitator guide.

Pick one rule that closed a backlog item this year. Can you show the test that proves it still fires today?