Projects / 04
AI-Native SOC Lab
Active research into AI-assisted triage, agent-assisted investigation and human-in-the-loop automation inside a working security operations lab.
- Alert rules
- 55
- Alert groups
- 18
- Dashboards
- 6
Count of rules in the lab's own monitoring configuration. Descriptive of this environment only; not a coverage or efficacy measure.
Count of alert groups in the lab's own monitoring configuration. Descriptive of this environment only.
Count of dashboards in the lab's own monitoring configuration. Descriptive of this environment only.
Lab measurement from a tested configuration in an experimental environment. Not a benchmark, and not portable to other hardware, models or workloads.
Overview
The AI-Native SOC Lab is a working security operations environment used to study where agentic assistance genuinely helps an analyst and where it quietly makes things worse. It runs its own detection, enrichment and investigation pipeline so that experiments are conducted against real telemetry rather than replayed samples alone.
We investigate four things in particular: AI-assisted triage of alerts, agent-assisted investigation across data sources, automated enrichment, and the design of human-in-the-loop automation. Alongside those sits a fifth concern that the others depend on — observability of the agents themselves.
The problem
Security operations has a volume problem and an interpretation problem, and they are usually confused with one another. Adding automation to the volume problem produces more decisions that still need interpreting. Adding a language model to the interpretation problem produces confident narratives that may not be supported by the underlying evidence.
Triage is where this is most acute. An assistant that summarises an alert well most of the time trains analysts to trust its summaries, which is precisely the condition under which its occasional wrong summary does damage. Any honest evaluation has to measure the failures, not the average.
Investigation adds a second issue: agents that query data sources take actions with real consequences — load on production systems, access to sensitive records, changes to case state. That makes agent observability a prerequisite for the research rather than a follow-on to it.
Architecture
The architecture separates detection, enrichment, investigation and decision. Detection produces alerts from telemetry. Enrichment attaches context automatically and deterministically where it can. Investigation is where agent assistance is applied. Decision remains with a person, and the boundary between the last two is drawn explicitly.
Agent activity is instrumented as its own telemetry stream: which sources were queried, what was returned, what was placed in front of the analyst, and what the analyst then did. That stream feeds the same monitoring pipeline as everything else, which means the assistant is observable on the same terms as the systems it is helping to defend.
- AI-assisted triage
- Summarisation and prioritisation proposals attached to an alert, always presented alongside the evidence they were derived from rather than in place of it.
- Agent-assisted investigation
- Agents that gather and correlate context across sources during an investigation, under explicit permissions and with every query recorded.
- Enrichment
- Deterministic context attachment that runs before any model is involved, so that the assistant reasons over enriched data rather than replacing enrichment.
- Human-in-the-loop automation
- Automation with defined suspension points where an action stops and requires a person, chosen by consequence rather than by confidence score.
- Agent observability
- Traces, metrics and logs describing agent behaviour, retained so that an assisted investigation can be reconstructed after the fact.
Security model
Alert content, log data and any document an agent retrieves during an investigation are treated as untrusted input. This matters more in a SOC than elsewhere: an attacker who can write to a log the assistant reads has a path to influence the investigation of their own activity.
- Trusted
- The detection pipeline configuration, the case record and the audit log of agent activity, all administered outside the agent's reach.
- Untrusted
- Alert payloads, log content, retrieved documents, third-party threat intelligence and all model output.
- Analyst
- Treated as the decision authority, not as a rubber stamp. The design fails if it produces so many approval requests that analysts stop reading them.
Agent permissions to query data sources are scoped and recorded. An assistant that can reach any source an analyst can reach inherits the analyst's blast radius, which is not an acceptable default in an environment holding sensitive records.
Current status
This is active research. The lab's detection and monitoring configuration is operational and stable enough to support experiments; the AI-assisted triage and investigation work is ongoing and has not produced conclusions we would present as findings.
The figures below describe the lab's own monitoring configuration. They are a statement of what has been built in this environment, not a performance claim, an efficacy result or a benchmark.
Limitations
One limitation is scale. A single lab environment does not generate the alert volume or the adversary diversity of a real estate, so results about triage assistance may not hold where analyst attention is genuinely scarce.
A second is evaluation. We do not have a ground-truth labelled corpus for this environment, which means claims about triage quality would rest on our own judgement of our own alerts. That is why no efficacy figures are published here.
A third is that automation bias is difficult to measure with a small number of operators. The failure mode we most want to detect — analysts accepting a wrong summary because previous ones were right — needs more participants than a lab has.
A fourth is that the lab's telemetry is generated by the lab's own systems, so detection content tuned here is tuned to a narrow environment and should not be assumed portable.
Roadmap
- Define an evaluation method for triage assistance that measures failure cases rather than average quality.
- Instrument the analyst decision point so that assistant proposals and final decisions can be compared.
- Scope and record agent data-source permissions explicitly for every investigation path.
- Test log-mediated influence on an assisted investigation as a controlled experiment.
- Align the agent telemetry schema with the agent security control plane work.
- Publish the human-in-the-loop suspension criteria for critique before broadening automation.
Detail
Stack
- Prometheus
- Grafana
- Loki
- Tempo
- PostgreSQL
- OpenTelemetry
Topics
- security operations
- detection engineering
- triage
- agent observability
- SIEM
- human-in-the-loop