Capturing Lineage for AWS Location Service Operations

Part of: Structuring JSON/XML Lineage Documents

Geocoding an address, snapping a GPS trace to roads, or calculating a route through AWS Location Service each derives new spatial data from a managed provider, and when that output flows into a downstream dataset you need a record of which operation produced it and against which resource. This how-to wraps boto3 Location Service calls so every place, route, and map request emits a JSON lineage document. It sits under Structuring JSON/XML Lineage Documents, and the emitted documents are designed to land in a write-once archive such as Object Storage WORM Retention.

Prerequisites

  • Python 3.10+ and boto3 1.34+.
  • An IAM principal with geo:SearchPlaceIndexForText, geo:CalculateRoute, and related geo:* permissions for the resources you call.
  • Existing AWS Location resources: a Place Index and/or a Route Calculator, created in the same region as your client.
  • AWS credentials via AWS_PROFILE, environment variables, or an instance role. Set AWS_REGION to match your Location resources.
A managed service is an opaque process step Request and response are fully observable; the provider's algorithm, reference data vintage and internal parameters are not, so the record documents the boundary. Request observable: text, index name, params Managed service algorithm: unknown data vintage: unknown and it changes without notice Response observable: coordinates, relevance, place id Record the exact request and the exact response, plus the API version and the call timestamp. That is genuinely reproducible evidence of what you asked and what you were told. It is NOT reproducible in the sense of re-running: the same call next year may return different coordinates.

The distinction in the last line is the one to be honest about in your records. Provenance for a self-hosted transformation supports reproduction: re-run the code at the pinned version against the same input and get the same output. Provenance for a managed service supports attestation instead — this is what we sent, this is what came back, at this time, from this API version. Both are legitimate, and conflating them leads to a record that implies a reproducibility guarantee nobody can honour.

That asymmetry has a practical consequence: the response must be stored, not merely referenced. A self-hosted step can store parameters and regenerate the output on demand; a geocoding result cannot be regenerated, because the provider’s reference data will have moved on. Persist the full response payload alongside the record, and treat it as the authoritative artefact rather than as a cache.

Implementation

The wrapper records the operation name, the target Location resource, a hashed request payload (so the same query is deduplicated and no raw address is stored in the clear), a summary of the response geometry, and the AWS request id from the response metadata. That request id is the anchor that ties your lineage record back to CloudTrail.

from __future__ import annotations

import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

import boto3


def geocode_with_lineage(
    place_index_name: str,
    text: str,
    lineage_dir: str | Path,
    max_results: int = 1,
) -> dict[str, Any]:
    """Geocode text via AWS Location and emit a JSON lineage document.

    Args:
        place_index_name: Name of the Location Service Place Index resource.
        text: The address or place text to geocode.
        lineage_dir: Directory that receives the .json lineage document.
        max_results: Maximum number of candidate results to request.

    Returns:
        The lineage document written to disk.
    """
    out_dir = Path(lineage_dir)
    out_dir.mkdir(parents=True, exist_ok=True)

    client = boto3.client("location")
    requested_at = datetime.now(timezone.utc)

    response = client.search_place_index_for_text(
        IndexName=place_index_name,
        Text=text,
        MaxResults=max_results,
    )

    # Never persist the raw query text; hash it so records stay privacy-safe.
    query_hash = hashlib.sha256(text.encode("utf-8")).hexdigest()

    results = response.get("Results", [])
    positions: list[list[float]] = [
        r["Place"]["Geometry"]["Point"]
        for r in results
        if r.get("Place", {}).get("Geometry", {}).get("Point")
    ]

    summary = response.get("Summary", {})
    lineage: dict[str, Any] = {
        "event": "location_geocode",
        "operation": "SearchPlaceIndexForText",
        "resource": place_index_name,
        "data_source": summary.get("DataSource", "UNKNOWN"),
        "query_sha256": query_hash,
        "result_count": len(results),
        "positions": positions,  # [lon, lat] pairs, EPSG:4326
        "crs": "EPSG:4326",
        "aws_request_id": response["ResponseMetadata"]["RequestId"],
        "requested_at": requested_at.isoformat(),
    }

    record_name = f"{lineage['operation']}_{query_hash[:16]}.json"
    (out_dir / record_name).write_text(json.dumps(lineage, indent=2), encoding="utf-8")
    return lineage


if __name__ == "__main__":
    doc = geocode_with_lineage(
        place_index_name="agency-places",
        text="1600 Pennsylvania Ave NW, Washington, DC",
        lineage_dir="./lineage",
    )
    print("Captured", doc["operation"], "from", doc["data_source"])

The data_source field from the response Summary (for example Esri or Here) is essential provenance: AWS Location proxies third-party providers whose licensing and accuracy differ, so a record that omits the provider cannot support a downstream accuracy claim.

Capturing at the Right Layer

Three capture points, increasing in what they see CloudTrail records that a call happened; SDK middleware sees request and response; application code additionally knows why the call was made. CloudTrail — the call happened principal, time, API name · no request or response body good for access audit, insufficient for lineage SDK middleware — request and response ← capture here full payloads, API version, retry count one integration point covers every call site Application code — adds intent which dataset, which pipeline step, which batch pass this down as context rather than capturing separately

Middleware is the right layer because it covers every call site without each one remembering to instrument itself, and because it sees the values that actually went over the wire rather than the ones the caller intended. Retries are the clearest illustration: a call that succeeded on its second attempt looks identical from application code and is visibly two requests from middleware, which matters when the first attempt returned a partial result that was then discarded.

CloudTrail remains worth enabling for a different purpose. It answers who called the service and when, independently of your own instrumentation, which makes it a useful cross-check — a period where middleware recorded fewer calls than CloudTrail saw is evidence that some code path bypassed the instrumented client. Reconciling the two counts periodically is cheap and catches exactly the gap that is otherwise invisible.

Pass the pipeline context down rather than trying to infer it. The middleware knows everything about the call and nothing about why it was made, so attach the step identifier and batch identifier to the request context at the call site and let the middleware read them. That keeps one instrumentation point while still producing records that connect to the rest of the lineage graph.

One further consideration applies to any geocoding result specifically: the output is personal data far more often than teams assume. An address resolved to coordinates and stored against a person is exactly the pairing that makes location identifying, and the lineage record — which faithfully retains both the input string and the returned coordinates — becomes a store of personal data in its own right. Classify it accordingly, apply the same retention rules that govern the dataset it feeds, and include it in the scope of any subject-access or erasure procedure. A provenance store that was overlooked during a data-protection review is a common and entirely avoidable finding, and it is usually discovered by a regulator or an external assessor rather than by the team that built it, which makes the conversation considerably harder than it needed to be. Add the lineage store to the data inventory at the same time as the dataset it describes, classify it at the same level, and give it the same retention rule — after which the question of whether it was in scope simply never comes up at all during a review.

Verification

Cross-reference the captured aws_request_id against CloudTrail to prove the call happened as recorded:

import boto3

ct = boto3.client("cloudtrail")
events = ct.lookup_events(
    LookupAttributes=[
        {"AttributeKey": "EventName", "AttributeValue": "SearchPlaceIndexForText"}
    ],
    MaxResults=5,
)
for e in events["Events"]:
    print(e["EventTime"], e["EventId"])

The event time in CloudTrail should fall within seconds of the requested_at timestamp in your JSON document, and the operation name must match. A lineage record whose request id has no CloudTrail counterpart indicates the call was mocked or replayed and should not be trusted as evidence.

Cost, Volume and What to Record Per Call

Three granularities for high-volume service calls Per-call, per-batch and per-job capture compared on record volume and on what question each can still answer. GRANULARITY STILL ANSWERS VOLUME Per call "why is THIS address here?" 1 record per address Per batch + digest "which batch, which API version, and here is the stored response set" 1 record per batch Per job only "geocoding happened" — little else 1 record per run

The middle row is the working compromise for anything at volume. One lineage record per batch, carrying the API version, timestamp, request count and a digest of the stored response set, keeps the record count proportional to jobs rather than to addresses — while the response payload itself, archived once, still answers the per-address question when somebody asks it. The lineage store stays queryable and the detail remains recoverable.

Per-call records are worth it only where an individual result carries consequence on its own: an address that determines a service boundary, a jurisdiction, or an eligibility decision. In that case the granularity is not about provenance completeness but about the specific record being independently defensible, which is a stronger requirement and justifiably more expensive.

Gotchas & edge cases

  • Coordinate order is [longitude, latitude]. AWS Location returns and expects GeoJSON-style [lon, lat] positions, the reverse of the lat, lon order humans write. Store the order explicitly, as above, or a downstream consumer that assumes [lat, lon] will place every point in the wrong hemisphere.
  • Route geometry can be large and unstable. CalculateRoute with IncludeLegGeometry returns a dense polyline that differs run-to-run as the provider updates its road graph. Hash a normalized summary (distance, duration, waypoint order) rather than the full geometry, or identical routing intents will appear as distinct lineage events.
  • Records must be immutable to count as evidence. A lineage document that can be edited after the fact proves nothing. Write each document once and push it to a locked store as described in Object Storage WORM Retention, so the provider, request id, and geometry summary cannot be altered after capture. Keep the schema aligned with Structuring JSON/XML Lineage Documents so Location events query the same way as every other source.