KOR IT LAB / 04
Observability Lab
The Observability Lab treats telemetry design as a first-class engineering problem. Its central question is what a system must emit for an operator or an analyst to reconstruct what happened, and what it costs to emit it.
- Infrastructure telemetry
- AI telemetry
- Agent telemetry
- Network telemetry
- Security telemetry
- Stack
- Prometheus · Grafana · Loki · Tempo
- Scope
- Host, inference, agent runtime and security events in one estate
One telemetry estate for performance and security
Question
Can the same instrumentation answer both operational and security questions about an agentic system, or do the two require separate pipelines?
Method
The lab emits host metrics, inference performance, agent runtime events, policy decisions and security events into a single observability stack, then poses both kinds of question against it: why was this slow, and what influenced this decision.
- Alert rules
- 55
- Alert groups
- 18
- Dashboards
- 6
Lab measurement from a tested configuration in an experimental environment. Not a benchmark, and not portable to other hardware, models or workloads.
Findings
These figures describe the lab's own monitoring configuration — the Prometheus, Grafana, Loki and Tempo stack built to observe agent behaviour in this environment. They are not a customer deployment and not a benchmark. They say how much instrumentation exists, not how well anything performs.
- Performance and security questions want the same events at different granularities: the tool call that explains a latency spike is the same event that explains an action.
- Trace context is what makes an agent's activity reconstructable. Without a shared identifier across retrieval, inference and tool invocation, the events exist but cannot be joined.
- Silent failure is the primary concern in this environment. An agent that stops retrieving, or a check that stops running, produces no signal at all unless absence is itself instrumented.
Limitations
Next experiments
Written down before they are run, so the results can be judged against what we set out to find rather than what we happened to notice.
Measure the cardinality and retention cost of agent-level tracing at realistic session volumes.
Instrument absence explicitly — build detections for retrieval that stopped, checks that stopped running and agents that went quiet.
Test whether a single trace identifier can survive the boundary between agent runtime and downstream tool systems.