Tamper-Evident Audit Trail

Investigators and auditors reach for the logs and find they were built for the team that runs the system, not for someone reconstructing what an administrator did. Application logs record errors. Audit logs sit in the same account as the resources they describe, writable by the same administrator, with retention set by whoever owns the storage bill.

Investigations then stall on questions the platform cannot answer: which account read the customer table, who changed the firewall rule. Time is off between hosts, so events cannot be ordered. Logs expire after thirty days because that was the cheap tier, and the incident is discovered on day forty-five. The record exists in fragments, and none of it would survive a challenge.

Define the audit trail by the questions it must answer, then build storage that the audited party cannot alter. Agree the event classes up front: authentication outcomes, privileged and administrative action, access to sensitive data, configuration and policy change, key and secret use, and the audit system's own activity. Each class has a schema, an owner and a retention period derived from legal obligation rather than storage cost.

Send those events to a dedicated logging account, separate from every workload account, with its own identity, its own administrators and a policy that denies deletion to everyone including the platform team. Land the raw events in write-once storage with a defined immutable period, in a lock mode that no administrator can bypass, and keep the searchable copy in the same custody boundary. Keep the logging account out of the organisation's management account: in AWS, for example, organisation policies do not apply there or to service-linked roles. The lock, not the policy, is the control that holds against every administrator, short of closing the account itself, which is the one way AWS documents to remove compliance-mode objects early. Restrict who can close the logging account and alert on any attempt.

Protect integrity with hash chaining: each batch carries the digest of the previous batch, and digests are anchored elsewhere, in a separate account or with an RFC 3161 timestamping service. An investigator can then show that a gap is real rather than a deletion.

1. Control owner. Agrees event classes and retention periods with legal, risk and the regulators who apply.
2. Platform team. Enables native audit logging at the account, cluster and database level, and configures the export before workloads go live.
3. Pipeline. Normalises events into one schema, drops nothing, and attaches a receive timestamp from a trusted clock alongside the event time.
4. Custody store. Writes batches to immutable storage with object lock, and records the batch digest chained to its predecessor.
5. Anchoring. Publishes each digest to a second account or an external timestamping service on a schedule, so a rewrite is detectable.
6. Access control. Grants read-only search to the security and audit functions and, with the lock, denies delete to every principal.
7. Verification. Runs a scheduled job that recomputes the chain and raises an alert on any gap, mismatch or retention change.

• Central custody in a dedicated account whose administrators are not the workload administrators
• Object lock and lifecycle policy as the retention control, not an application setting
• Native control plane audit logs enabled by default and never filtered at source
• Time from a managed, synchronised source rather than host clocks

• Deny delete and retention change through an organisation-level policy, and keep the logging account out of the management account, where such policies may not apply.
• Enable object lock or equivalent write-once storage, in a mode no administrator can bypass short of closing the account, with a defined immutable period.
• Chain batch digests and anchor them outside the custody account.
• Restrict read access to named audit and security roles, with every query itself logged.
• Synchronise time from a managed source and monitor drift.
• Alert on gaps, disabled logging, policy change and retention reduction, and on any attempt to close the logging account, which only named roles may do.
• Test retrieval quarterly, including an export that verifies against the chain.

• Tamper evidence vs. cost: immutable storage and a separate custody copy add materially to the bill for the same volume.
• Immutability vs. mistakes: a bad event written to a locked bucket cannot be deleted, so schema errors persist for the whole retention period.
• Separation of custody vs. operational friction: the platform team loses the ability to debug using audit data and needs a grant process.
• Completeness vs. volume: logging every data read answers the investigator and can produce very large daily volumes.

• Log the decision, not just the action. An allow and a deny are both events. A trail that records only successful operations cannot show an attempted escalation.
• Separate custody is a policy, not a folder. The logging account needs its own administrators, its own break-glass process and an explicit deny on delete for everyone else.
• Check the lock mode. Some object-lock implementations offer a mode that a suitably permissioned user can bypass (governance mode in Amazon S3 Object Lock, for example) and one that no user can override (compliance mode). Only the second holds against your own administrators (short of closing the account; see the Solution).
• Include the audit system itself. Changes to retention, to bucket policy and to the logging pipeline are the events an attacker most wants to make.
• Data access needs an identity. Shared service accounts and brokered sessions without attribution undermine every other control in this pattern.

A health insurer runs forty workloads across three cloud accounts. Audit logs are enabled, but each workload writes to its own bucket and the platform team holds delete rights on all of them. A support engineer leaves, and three weeks later someone notices unusual queries against a claims table. The investigation finds that the database audit log was rotated after seven days, application logs recorded the query text but not the identity, and the engineer's own account had permission to empty the bucket.

The insurer rebuilds the trail. A logging account is created outside the management account, with separate administrators and an organisation policy that denies deletion. Database audit logging is turned on for the claims schema, events are normalised, and batches land in storage under a lock no administrator can lift short of closing the logging account, which only named roles may do, with a seven-year retention for the classes legal names. Each batch is hashed and the digest published to a second account every hour. Support access to production moves behind a brokered session, so the identity in the log is a person rather than a shared role.

The next investigation takes an afternoon. The chain verifies, the query history is complete, and the export is accepted by counsel without argument. That is the difference between a record and a story.