Experiment: AI vs Human Alert Triage

The 30-second explanation

Take a batch of old alerts where you know the real outcome. Have the assistant triage them, hide the answers, then compare its suggestions with what actually happened.

How it works

1. 1

   1. Build the set

   Closed alerts with known outcomes, redacted.

2. 2

   2. Run

   Same prompt and output format for each.

3. 3

   3. Compare

   Suggestions against known outcomes.

4. 4

   4. Review

   Study misses and weak reasoning first.

Choose a small set of closed alerts with known outcomes, including harmless ones and real incidents, then strip out anything sensitive. Give each one to the assistant using the same structured prompt. Record its suggested priority and reasoning, and set that beside the analyst's final verdict. Pay the most attention to real incidents the assistant downgraded, since those are the misses that actually cost something. Also note whether its reasoning held up even on the alerts where the verdict happened to be right. Nothing here is a finished result. It's a way to find out, on your own alert set, exactly how much you can lean on the assistant's judgement.

Real-world example

How to spot it

• Only measuring accuracy

  Ignoring which errors were the costly kind.

• Test set too easy

  Mostly obvious alerts.

• Leaked answers

  The outcome visible in the alert text.

• Small, unrepresentative samples

  Conclusions drawn from a handful of cases.

What to do

1. 1Include hard and real incidents in the set, not only routine alerts.
2. 2Track missed real threats separately from false alarms.
3. 3Re-run the test when the model, prompt or alert types change.