Configuring S3 Object Lock for Lineage Archives

Part of: Object-Storage WORM Retention

When a lineage archive must be provably tamper-proof for its entire retention period, you enable S3 Object Lock and write the object under a compliance-mode retention date plus a legal hold; use this whenever a finalized provenance record needs infrastructure-enforced immutability rather than a policy promise. This how-to is the hands-on companion to the object storage WORM retention overview and shows the exact boto3 calls end to end.

Prerequisites

  • Python 3.10+ with boto3 1.34+ and botocore in an active virtual environment.
  • AWS credentials granting s3:CreateBucket, s3:PutBucketVersioning, s3:PutObject, s3:PutObjectRetention, s3:PutObjectLegalHold, and s3:GetObjectRetention.
  • A region selected via AWS_DEFAULT_REGION or an explicit region_name.
  • A canonical serialization of the lineage document to archive (deterministic JSON is ideal). The append-only record shape from structuring JSON/XML lineage documents works well here.
Order of operations — most of it is not retrofittable Versioning and Object Lock must be enabled when the bucket is created; only the default retention and per-object overrides can be changed later. AT BUCKET CREATION ONLY — cannot be added later · Versioning enabled · Object Lock enabled Getting this wrong means creating a new bucket and copying everything across. CHANGEABLE LATER · Default retention mode and period · Bucket policy · Lifecycle transitions PER OBJECT, AT WRITE TIME · Retain-until date (may exceed the default, never fall below it) · Legal hold on/off

The top band is the reason to get this right in infrastructure-as-code before any data is written. Object Lock cannot be enabled on an existing bucket, and neither can versioning be meaningfully retrofitted for lock purposes, so a bucket created without them is a bucket that must be replaced. Copying an archive across is straightforward for gigabytes and painful for terabytes, and the objects arrive with new creation dates that complicate any retention computed from them.

The bottom band contains the operational subtlety worth testing: a per-object retain-until date may extend beyond the bucket default but may never be shorter. A write attempting a shorter period does not silently fall back to the default — it fails — which is correct behaviour and surprising the first time a script computes a retention from a variable that was briefly zero.

Implementation

The script below creates a fresh Object-Lock-enabled bucket, writes a lineage archive under a five-year COMPLIANCE retention with a legal hold switched on, and returns the version ID. Object Lock can only be enabled at bucket creation, so the workflow starts from a new bucket rather than mutating an existing one.

import hashlib
import json
from datetime import datetime, timedelta, timezone

import boto3

REGION = "us-east-1"
BUCKET = "agency-lineage-worm-archive"
KEY = "lineage/2026/scene-8842-provenance.json"

s3 = boto3.client("s3", region_name=REGION)


def ensure_locked_bucket(bucket: str) -> None:
    # ObjectLockEnabledForBucket implicitly enables versioning; we set it
    # explicitly so the intent is auditable in infrastructure code.
    s3.create_bucket(Bucket=bucket, ObjectLockEnabledForBucket=True)
    s3.put_bucket_versioning(
        Bucket=bucket,
        VersioningConfiguration={"Status": "Enabled"},
    )


def put_locked_archive(bucket: str, key: str, lineage: dict, years: int = 5) -> str:
    # Deterministic bytes => reproducible SHA-256 for chain-of-custody proof.
    body = json.dumps(lineage, sort_keys=True, separators=(",", ":")).encode("utf-8")
    digest = hashlib.sha256(body).hexdigest()

    # UTC-aware arithmetic prevents clock-skew rejection of the retain-until date.
    retain_until = datetime.now(timezone.utc) + timedelta(days=365 * years)

    resp = s3.put_object(
        Bucket=bucket,
        Key=key,
        Body=body,
        ObjectLockMode="COMPLIANCE",              # irreversible WORM for the window
        ObjectLockRetainUntilDate=retain_until,   # no principal can shorten this
        ObjectLockLegalHoldStatus="ON",           # independent, indefinite freeze
        Metadata={"sha256": digest},
    )
    return resp["VersionId"]


if __name__ == "__main__":
    lineage_doc = {
        "dataset_id": "scene-8842",
        "crs": "EPSG:4326",
        "derived_from": ["scene-8840", "scene-8841"],
        "process": "mosaic+radiometric_correction",
        "actor": "svc-etl-prod",
        "committed_at": datetime.now(timezone.utc).isoformat(),
    }
    ensure_locked_bucket(BUCKET)
    version_id = put_locked_archive(BUCKET, KEY, lineage_doc)
    print(f"archived version: {version_id}")

The three ObjectLock* parameters on put_object do the work. ObjectLockMode="COMPLIANCE" means the retain-until date cannot be shortened or bypassed by any principal, including the root account. ObjectLockLegalHoldStatus="ON" layers an independent, open-ended freeze on top, so the object stays immutable even if you later decide to reduce or expire scheduled retention. The SHA-256 digest stored in user metadata binds the archived bytes to a verifiable fingerprint you can re-derive at audit time.

Access Model Around the Archive

Three principals, none of which can shorten retention A write-only ingest principal, a read-only auditor principal, and an administrator explicitly denied the bypass-governance permission. ingest principal s3:PutObject no Get, no Delete auditor principal s3:GetObject, ListBucket no write of any kind administrator explicit Deny on BypassGovernanceRetention The explicit Deny matters even under compliance mode, where no bypass exists. It documents intent, and it survives a later decision to relax the bucket to governance mode. Separating write from read is what makes "the writer could not have read or altered history" true. A single principal holding both is the configuration an assessor will ask you to justify.

Splitting write from read is worth more than it costs. The ingest principal needs only PutObject; granting it read access adds nothing operationally and weakens the story you can tell about what a compromised pipeline credential could do. Likewise the auditor principal should hold no write permission at all — an auditor who can write to the evidence store is an auditor whose findings can be questioned.

The explicit deny on the bypass permission is a small piece of documentation with real value later. Under compliance mode it is redundant, since the permission cannot be exercised regardless; its purpose is to survive the day somebody proposes relaxing the bucket to governance mode for operational convenience. At that point the deny is already in place and the conversation is about removing a control rather than about failing to add one.

Attach a bucket policy rather than relying on identity policies alone. A bucket policy is evaluated for every principal including ones created later, so a new role granted broad object-store access does not silently acquire the ability to write to the archive. Identity policies alone leave that gap open by default.

Verification

Confirm the retention actually took hold by reading it back. get_object_retention returns the mode and retain-until date; get_object_legal_hold returns the hold status.

def verify_lock(bucket: str, key: str, version_id: str) -> None:
    ret = s3.get_object_retention(Bucket=bucket, Key=key, VersionId=version_id)
    hold = s3.get_object_legal_hold(Bucket=bucket, Key=key, VersionId=version_id)
    print("mode:", ret["Retention"]["Mode"])
    print("retain_until:", ret["Retention"]["RetainUntilDate"].isoformat())
    print("legal_hold:", hold["LegalHold"]["Status"])

A correct run prints mode: COMPLIANCE, a retain_until roughly five years out, and legal_hold: ON. As a negative check, attempt a delete of that version and confirm S3 rejects it: s3.delete_object(Bucket=BUCKET, Key=KEY, VersionId=version_id) raises an AccessDenied error while the retention is active, which is exactly the immutability guarantee you want to demonstrate to an auditor.

Writing Into the Archive Safely

Computing and applying retention safely Retention is read from a reviewed configuration, bounds-checked, then applied in the same PutObject request rather than as a follow-up call. retention policy reviewed config, not inline literal bounds check min ≤ period ≤ max refuse anything else PutObject lock params in the SAME request locked on arrival Writing first and locking second leaves a window where the object is deletable. A crash between the two calls produces an unlocked archive object nobody notices. Sweep periodically for objects in the archive prefix with no retention set — there should be none.

Applying the lock in the same request as the write is the detail that makes the guarantee unconditional. Two-step approaches — upload, then set retention — are easier to write and leave a window in which the object exists and is deletable, and a process that dies in that window leaves an unprotected object among protected ones with nothing to distinguish it visually.

The periodic sweep in the caption is cheap insurance and doubles as evidence. Listing the archive prefix and asserting every object has a retain-until date takes one paginated call, catches the two-step remnant, and produces a dated record that the archive was verified — which is precisely the kind of routine check an assessor likes to see evidence of.

Testing the Lock Before You Depend on It

The whole configuration reduces to one claim — that these objects cannot be deleted or altered before their retention expires — and that claim is worth verifying rather than trusting to the console showing the right settings.

Build a throwaway bucket with the same infrastructure code and a deliberately short retention, a minute or two rather than seven years. Write an object, then attempt each of the operations the lock is meant to prevent: delete the version, overwrite it with new content, and reduce the retain-until date. Each must fail with an access-denied or invalid-request error. Assert on the failure rather than merely observing it, so the test can live in CI and catch a future change that relaxes the policy.

Then let the retention expire and confirm deletion succeeds. This second half matters more than it looks: a bucket where deletion fails even after expiry means the lifecycle will never reclaim anything, and the storage cost is not the seven years you budgeted but indefinite. That failure mode usually traces to a bucket policy denying delete outright, which is a reasonable-looking belt-and-braces addition that quietly converts a retention period into a permanent commitment.

Run the same suite against the production bucket’s configuration — not against production data — by pointing the infrastructure test at a parallel bucket built from the identical template. Testing the template rather than the instance is what makes the result meaningful when the production bucket is later recreated in a new account or region.

Finally, verify that the objects you expect to be locked actually are. The settings being correct at bucket level does not prove that every write path applied them, and the sweep described above answers a different question from the configuration review: not “is the bucket set up correctly” but “did everything that was written arrive protected”.

Gotchas & edge cases

  • Object Lock is creation-time only. You cannot enable it on an existing bucket — put_object_lock_configuration returns InvalidBucketState. If you inherited an unlocked bucket, create a new locked bucket and copy the archives across, then verify digests match before decommissioning the old one.
  • Versioning is mandatory. Object Lock retention parameters are rejected unless bucket versioning is Enabled. Enabling Object Lock at creation turns versioning on for you, but never suspend it afterward or new PUTs lose their lock semantics.
  • Compliance mode is a one-way commitment. Once written, a compliance retain-until date cannot be reduced, and the object cannot be deleted until it passes. Validate the lineage document — especially its CRS and derivation edges — before you commit it, because a mistaken archive will occupy locked, billable storage for the full window. Use governance mode in staging if you need a break-glass path during testing.