KOR IT LAB / 01
AI Lab
The AI Lab exists to answer operational questions about agentic systems that cannot be answered from documentation: what a real agent workload costs on a specific host, where the binding constraint actually sits, and what an agent's behaviour looks like when every step is instrumented.
- Local inference
- Open models
- Agent architectures
- Memory
- Performance
- Sovereign AI
- Host
- AMD EPYC 7451 · 24C/48T · 256 GB ECC · NVMe · Proxmox
- Inference
- llama.cpp with Qwen-family open models
- State
- PostgreSQL · Qdrant · MinIO
Agent workload characterisation at 64K context
Question
For an agent loop that rebuilds a long, largely repeated prompt on every turn, where is the binding performance constraint on a CPU-only host?
Method
We measured the configuration as it is actually used rather than a synthetic best case. Prefill and generation are reported separately because a single blended token-per-second figure hides the cost structure of agent workloads, where the prompt is long, largely repeated, and rebuilt on every turn.
- Context length under test
- 64K
- Prefill throughput
- 22.5 tok/s
- Aggregate generation throughput
- 33.5 tok/s
- Prefix cache hit rate
- 94.4%
Lab measurement from a tested configuration in an experimental environment. Not a benchmark, and not portable to other hardware, models or workloads.
Findings
- Prefill, not generation, is the binding constraint for this workload: 22.5 tok/s prefill against 33.5 tok/s aggregate generation at 64K context, which inverts the intuition carried over from chat-style usage.
- Prefix caching dominates agent-loop responsiveness at long context. Because agent turns rebuild a largely identical prompt, the measured 94.4% hit rate affects perceived latency more than raw generation throughput does.
- NUMA placement is not a tuning detail on a four-domain host. Inference performance here is governed by memory locality, and placement has to be decided deliberately rather than left to the scheduler.
Limitations
- Throughput was captured under lab load, not under sustained concurrent multi-agent use.
- Model behaviour is specific to the Qwen-family models tested; no claim is made about other open models or about hosted models.
Tool mediation as a behavioural record
Question
Does routing every agent tool call through a single mediating gateway produce a better description of agent behaviour than inspecting model output?
Method
All tool invocations in the lab pass through one gateway that records the call, its arguments, the policy decision and the result. We then compared what that record says an agent did against what the agent's own narration claimed it did.
Findings
- The sequence of tool invocations is a far better description of what an agent did than its narration of what it intended to do.
- Mediation makes agent behaviour legible in a way that inspecting model output does not — the gateway record is complete by construction, whereas narration is a generated artefact.
- Once mediation exists, policy has somewhere to live: allow, review and quarantine become implementable decisions rather than prompt instructions.
Limitations
- The lab has no external adversary. Untrusted content is introduced by us, so the threat model is exercised against attacks we already thought of.
- Sandboxing is tested against ordinary code execution, not against a determined escape attempt by a capable attacker.
Next experiments
Written down before they are run, so the results can be judged against what we set out to find rather than what we happened to notice.
Measure throughput and cache behaviour above and below 64K to characterise where prefill cost becomes prohibitive on this host.
Instrument concurrent multi-agent execution to find the point at which NUMA-aware placement stops holding.
Compare agent completion behaviour with tool permissions enforced at the gateway versus enforced only by prompt instruction.