Log Ingestion & SIEM

Security operations teams inherit a platform that collects logs and calls that monitoring. Sources are onboarded because they exist rather than because a detection needs them, each parsed by a bespoke rule written by someone who has left, and the bill is driven by volume nobody has looked at since the pilot. Alerts arrive faster than the team can work them, so they are triaged by title and closed, and the detection content is never tested against known-bad behaviour.

What results is an expensive archive with an alert feed attached. It answers "what happened on this host in March" well, and "are we being attacked now" poorly, because nobody has checked whether the sources that question needs are arriving. A source that stops sending is invisible: the dashboard goes quiet and quiet reads as good news.

The cost conversation then dominates, and the usual response, cutting retention or dropping sources, removes evidence rather than waste, because the volume sits in a few verbose feeds nobody has measured.

Design ingestion backwards from detection. Start with the questions the team must answer, derive the sources those questions need, and measure coverage against that list, not a source count. A source earns its cost only when a detection or investigation depends on it.

Normalise on ingest to a common schema. OCSF is an open, vendor-neutral option, and several platforms ship their own normalisation model. One schema means one set of correlation rules, one set of dashboards and one place to add enrichment, instead of a parser per vendor. Keep the raw event alongside the normalised one for the fields the mapping loses, at least in the hot tier.

Tier retention by purpose: hot for 30 to 90 days supports investigation and hunting, warm for a year supports audit questions, and an archive in object storage covers the regulatory period. Cost falls steeply as the tier cools; the difference is query latency, not existence.

Then treat detection content as code: version controlled, tested against sample events, owned, and measured for false positives.

1. Requirement mapping. The team lists the questions and threats it must detect, and derives the required sources and fields.
2. Collection. Agents, forwarders or provider APIs deliver events, with the method chosen per source and documented.
3. Normalisation. Events are parsed to the common schema and enriched with asset and identity context, with the raw event kept where fidelity matters.
4. Routing. Rules send each event class to the hot, warm or archive tier by detection and retention purpose.
5. Detection. Correlation, threshold and behavioural rules run against the normalised stream, with state for multi-event logic.
6. Alerting and case. A detection becomes an alert with context, then a case with an owner and a timeline.
7. Tuning. Every case feeds rule quality: exclusions, thresholds, new logic, or a decision that the detection is not worth its noise.

• Ingestion as a metered utility, so volume is a budget line with an owner
• Schema-first normalisation (for example OCSF) instead of per-source parsers
• Hot, warm and archive tiers with different cost and latency
• Coverage measured against a required source list, not a licence count
• Correlation across identity, network, endpoint and cloud control planes at query time

• Authenticate every source and encrypt transport; reject unauthenticated ingestion.
• Restrict access by role, with read-only for most users and audit logging of every query.
• Protect the detection repository with review and signed commits; a rule change is a production change.
• Retain audit logs for the regulatory period in an immutable store, separate from the SIEM.
• Alert on ingestion gaps, parser failures and collector restarts.

• Breadth of correlation vs. ingestion is commonly priced by volume, and a handful of verbose sources often carry most of the cost.
• Normalisation enables reuse vs. the mapping loses fields, and the ones it loses are often the ones an investigation needs.
• Long hot retention shortens investigations vs. it is usually the largest line on the bill.
• More detection content catches more vs. every rule adds noise, and a queue nobody trusts is worse than a shorter one that is worked.

• Coverage is a list, not a number. "We ingest 210 sources" says nothing. "We have 27 of the 31 sources that the top 25 detections require" says something an executive can act on.
• Silence is the failure mode. Alert on the absence of expected events per source, with a threshold from normal volume. A source that stops at 3am should page someone.
• Keep the raw event in the hot tier. Schema mappings change, and re-parsing from raw is the only way to answer a question the mapping cannot.
• Measure detection quality, not detection count. For each rule: owner, last review date, false positive rate, and the threat it addresses. Delete rules that fail the test rather than keeping them for the count.
• Threat intelligence is enrichment. Indicator feeds join events at ingest or query time; judge each feed by the detections it improves, as with any other source.

A retail group runs 4,500 endpoints, three cloud accounts and 210 log sources into a SIEM that costs more each quarter. Five analysts work an alert queue averaging 600 a day, most closed as benign within a minute.

The rebuild starts with a required source list derived from 40 detection requirements. Against it, 14 of the 52 required sources are missing entirely, including the Kubernetes audit log and two of the three cloud control planes. Meanwhile 80 sources feed no detection, and one application debug feed is 31 per cent of daily volume. That feed moves to archive, which pays for the two control planes with room to spare.

Normalisation to OCSF takes four months, source class by source class, starting with authentication and network flow. Rules are rewritten against the normalised fields, each with a test event set and an owner in the repository. The queue falls to 90 alerts a day, and the false positive rate on the top ten rules is measured monthly rather than guessed.

Retention is split: 45 days hot, 12 months warm, and longer archive only for the record classes the retention schedule names. The first test is a credential-stuffing campaign against the customer login, caught by a rule correlating identity provider failures with web application firewall blocks, which the schema work made possible.