The tiering decision: what a security data platform keeps hot

Every security data platform eventually reduces to the same decision, made repeatedly and usually without being named: for each stream of data arriving, how quickly does it need to be queryable, and for how long. Everything else — the choice of store, the normalisation strategy, the retention policy, the licence negotiation — follows from how that question is answered and by whom.

Answering it badly has a characteristic shape. Data that is expensive and rarely queried sits in the hot tier because it arrived through a connector that only writes there. Data that is needed during an incident sits in cold storage because it was voluminous and the decision was made on volume alone. Neither outcome was chosen; both were inherited from the path of least resistance at ingest.

Three questions, in order

A tier is a commitment about latency and a commitment about cost, and it should be made from the query side rather than the ingest side. Three questions, asked per source rather than per platform, settle most cases.

Who queries this, and under what pressure?

Real-time detection and an analyst mid-incident have a latency requirement measured in seconds. A quarterly audit, a threat-hunting expedition and a model training run do not.

What is the shape of the access?

A narrow lookup on a known key is cheap almost anywhere. A broad scan over ninety days is only affordable in a store designed for it, and the difference is several orders of magnitude.

What obliges us to keep it?

A regulatory retention period and an operational retention period are different numbers, and conflating them is how organisations end up paying hot-tier prices for compliance data nobody has ever read.

Normalise early, decide late

The cheapest architectural property to acquire, and the most commonly skipped, is that normalisation happens before routing. If every record is shaped to a common schema on the way in, then a tier is genuinely a storage decision: the same query can be expressed against the hot tier and the archive, and a stream can be promoted or demoted as its value changes without rewriting the content that depends on it.

Route first — the default when each destination is fed by its own vendor connector — and normalisation happens twice, differently, in two places. The archive then holds data in a shape the detection layer cannot read, which makes the archive an evidence store rather than a queryable asset, and makes tier changes a migration project instead of a configuration change.

A common schema does not have to be a standard one. It has to be written down, versioned, and owned. Where an open model such as OCSF or ECS fits the estate, adopting it saves the effort of inventing field names and buys compatibility with tooling that already speaks it. Where it does not fit, a documented internal schema beats an undocumented external one.

Routing is a security control

The routing layer sees every record before it lands anywhere, which makes it the correct place for several things that are otherwise done badly further downstream.

• Redaction and tokenisation of fields that should never reach a general-purpose analytics store, applied once rather than per destination.
• Residency enforcement, so that a record subject to a jurisdictional constraint cannot be routed to a region that violates it.
• Volume anomaly detection on the pipeline itself — a source that stops arriving is a detection gap, and the pipeline is the only component positioned to notice.
• Provenance stamping, so that a record carries where it came from and what was done to it, which is what makes an archive admissible as evidence rather than merely large.

The last point deserves emphasis. Security data spends its life being transformed — parsed, enriched, deduplicated, summarised. If the transformations are not recorded alongside the record, then a year later nobody can establish whether an absent field was absent at the source or lost in a parser change, and the archive answers questions less reliably the older it gets.

What goes wrong

Three patterns recur, and all three are architectural rather than operational.

1. The tier is chosen by the connector. Whatever the vendor integration writes to becomes the tier, and the platform's cost profile is the sum of decisions nobody made.
2. Cold storage is written but never read. It has no partitioning scheme aligned to how an investigation actually searches, so the first real retrieval attempt reveals that the data is present but not usable within the time available.
3. Retention is set once and never revisited, so the platform carries the regulatory maximum for every source rather than the operational requirement for most of them.

None of these is fixed by changing product. They are fixed by making the tiering decision explicit, assigning it an owner, and reviewing it on a schedule — which is unglamorous work that pays back continuously.