Validating ISO 19115 Lineage with Python

Part of: ISO 19115 Lineage Implementation

Serializing an LI_Lineage structure to XML is only half the job; a record that parses cleanly can still be non-conformant because a mandatory LI_ProcessStep.description is blank or a source citation is missing. This how-to builds a focused lxml validator that checks an ISO 19139 lineage document against required-element and cardinality rules before it reaches a catalog, complementing the assembly workflow in Implementing the ISO 19115-1 Lineage Model in a Spatial Pipeline. Use it as a CI gate so malformed lineage never reaches production metadata stores.

Prerequisites

  • Python 3.10+ with lxml 5.x installed (pip install "lxml>=5.0").
  • An ISO 19139 metadata file using the gmd/gco namespaces (the legacy encoding most validators still accept). The same rule structure ports to the mrl namespace of ISO 19115-3.
  • Familiarity with the element cardinalities in the implementation overview — this validator enforces exactly those rules.
  • Optional: the official ISO codelist catalogue if you extend the rules to check codeListValue membership.
Four validation layers and what each can catch Well-formedness, XSD, profile rules, and semantic rules, with only the last able to catch a lineage chain whose CRS does not join up. 1 · Well-formed catches: truncated files, encoding errors misses: everything about content 2 · XSD valid catches: wrong element order, wrong types misses: an empty lineage block 3 · Profile rules catches: missing mandatory elements, bad vocabularies misses: steps that contradict each other 4 · Semantic rules — yours to write catches: a CRS chain that does not join up, steps out of temporal order, sources not referenced

Layer four is the one nobody ships and the one that catches real defects. No off-the-shelf validator knows that a process step declaring an input CRS of EPSG:4326 should follow a step whose output CRS was EPSG:4326, because that is a domain rule about coordinate systems rather than a rule about XML. Writing it takes a few dozen lines: walk the steps in temporal order, and assert each step’s declared input matches the previous step’s output unless a reprojection is declared between them.

Two more semantic rules earn their keep. Assert that every source cited by a step is either an external source or the output of an earlier step, which catches steps referencing products that do not exist in the record. And assert that step timestamps are non-decreasing, which catches the reversed-history bug that arises whenever steps are read from a log stored newest-first.

Implementation

The validator loads the document, registers namespaces, and applies a small declarative rule set. Each rule is an XPath plus a cardinality expectation; the checker reports the first violation per rule with a human-readable path, so failures point straight at the offending element.

from __future__ import annotations
from dataclasses import dataclass
from lxml import etree

NS = {
    "gmd": "http://www.isotc211.org/2005/gmd",
    "gco": "http://www.isotc211.org/2005/gco",
}

@dataclass
class Rule:
    label: str
    xpath: str          # evaluated relative to each LI_ProcessStep or the root
    min_count: int
    scope: str = "root"  # "root" or "step"

# Rule set: mandatory LI_ProcessStep.description and at least one source citation.
RULES: list[Rule] = [
    Rule("LI_Lineage present", ".//gmd:LI_Lineage", 1, "root"),
    Rule("At least one processStep or statement",
         ".//gmd:LI_ProcessStep | .//gmd:statement", 1, "root"),
    Rule("ProcessStep.description mandatory [1]",
         "gmd:description/gco:CharacterString", 1, "step"),
    Rule("ProcessStep.description non-empty",
         "gmd:description/gco:CharacterString[normalize-space(text())]", 1, "step"),
    Rule("Source citation title present [1..*]",
         ".//gmd:LI_Source/gmd:sourceCitation//gmd:title/gco:CharacterString", 1, "root"),
]

def validate_lineage(xml_path: str) -> list[str]:
    tree = etree.parse(xml_path)
    root = tree.getroot()
    errors: list[str] = []

    for rule in RULES:
        if rule.scope == "root":
            hits = root.xpath(rule.xpath, namespaces=NS)
            if len(hits) < rule.min_count:
                errors.append(f"FAIL [{rule.label}]: found {len(hits)}, "
                              f"need >= {rule.min_count}")
        else:  # per-step evaluation catches an empty description on any one step
            steps = root.xpath(".//gmd:LI_ProcessStep", namespaces=NS)
            for i, step in enumerate(steps):
                hits = step.xpath(rule.xpath, namespaces=NS)
                if len(hits) < rule.min_count:
                    errors.append(f"FAIL [{rule.label}]: LI_ProcessStep[{i}] "
                                  f"found {len(hits)}, need >= {rule.min_count}")
    return errors

if __name__ == "__main__":
    import sys
    problems = validate_lineage(sys.argv[1])
    if problems:
        print(f"INVALID — {len(problems)} violation(s):")
        for p in problems:
            print("  " + p)
        sys.exit(1)
    print("VALID — all lineage rules satisfied")
    sys.exit(0)

The scope="step" rules are evaluated once per LI_ProcessStep, which is what catches the common case where one of several steps has an empty description while the others are fine — a document-level XPath count would miss it because the other steps satisfy the count.

Where Validation Belongs in the Pipeline

Three placements, one correct Validating at emission is too early because the record is incomplete; at publication is correct; after harvest is too late. At emission too early — the record is not finished yet At publication ✓ complete, and still cheap to block Post-harvest too late — already indexed elsewhere Emission-time checks still have a job — they enforce required FIELDS, not document validity. Two checks at two moments, testing different things, is the working arrangement. Run the same suite nightly over already-published records too — profiles change under you.

The nightly re-validation in the caption is worth setting up even though it feels redundant. Records that were valid when published can become non-conformant without changing, because the profile they are assessed against gets revised, a controlled vocabulary retires a term, or a referenced specification is superseded. Discovering that from your own scheduled run gives you a remediation list; discovering it from a harvester’s rejection notice gives you an incident.

Keep the two checks genuinely separate in code as well as in placement. Emission-time validation asks whether a lineage row has the fields policy requires; publication-time validation asks whether the rendered document satisfies the standard. Merging them produces a check that runs at the wrong moment for one of its two jobs.

Verification

Run the validator against a well-formed record and against a deliberately broken one. A conformant document exits 0:

$ python validate.py good_lineage.xml
VALID — all lineage rules satisfied

A record whose second process step has an empty <gmd:description/> and no source citation fails with precise pointers:

$ python validate.py broken_lineage.xml
INVALID — 3 violation(s):
  FAIL [ProcessStep.description mandatory [1]]: LI_ProcessStep[1] found 0, need >= 1
  FAIL [ProcessStep.description non-empty]: LI_ProcessStep[1] found 0, need >= 1
  FAIL [Source citation title present [1..*]]: found 0, need >= 1

Wire the non-zero exit code into your pipeline’s pre-publish stage so a failing record blocks the run, exactly as described for CI gating in the implementation overview. For raster pipelines, run this immediately after the checksum step covered in generating SHA-256 hashes for GeoTIFFs in Python so both integrity and lineage conformance are gated together.

Making the Validator Fail on Purpose

Four fixtures the validator must reject Each fixture isolates one rule; a passing build against any of them identifies exactly which rule is not implemented. FIXTURE MUST BE REJECTED BY Valid XML, empty LI_Lineage profile rule — XSD accepts it Step with no description profile rule CRS chain with an unlogged reprojection semantic rule only Steps in reverse chronological order semantic rule only

Keep all four fixtures in the repository and assert rejection in CI. This is the discipline that distinguishes a validator that works from one that has simply never seen bad input: the bottom two cases pass every standard tool, so without an explicit test there is no evidence the semantic rules are wired in at all.

The third fixture is worth constructing carefully, because it must be otherwise perfect. A document that is schema-valid, profile-conformant, has descriptions on every step, and differs from a good record only in that a reprojection went unlogged is exactly the input a real pipeline produces when a hook is missed — and it is the input that most needs to fail.

Reporting Failures Usefully

A validator that returns a boolean is a validator nobody will act on. What determines whether failures get fixed is how precisely the report identifies the offending element and how clearly it distinguishes a blocking failure from an advisory one.

Report three things per finding: the rule identifier, an XPath to the element that failed, and the value that was found. The XPath matters most — telling an engineer that “a process step is missing a description” in a record with forty steps sends them hunting, while pointing at /…/mrl:processStep[7]/…/mrl:description sends them to the line. Most XML libraries can produce the path cheaply during traversal, and the cost of threading it through the rule functions is repaid the first time somebody debugs a real failure.

Separate severities explicitly and let them mean different things to the build. Schema and profile violations block publication because the record is genuinely non-conformant. Semantic findings — a CRS chain that does not join up, steps out of order — should also block, since they indicate the lineage is wrong rather than merely incomplete. Advisories, such as an empty rationale on a step, belong in the report and should not fail the build; mixing them into the blocking set is how teams end up granting themselves a blanket exemption.

Finally, emit the report as data rather than as log lines. A JSON findings file per validated record lets the same output drive a CI annotation, a coverage dashboard, and the remediation backlog without anyone parsing prose. It also makes the validator’s own behaviour measurable: counting findings by rule over time shows which rules ever fire, and a rule that has never fired across thousands of records is either unnecessary or broken.

Gotchas & edge cases

  • Namespace prefixes are not fixed by the standard. A document may declare gmd as md or use a default namespace with no prefix at all. XPath matches on the namespace URI, not the prefix, so always pass the namespaces=NS mapping and never hard-code a literal prefix into an element test. If a document uses ISO 19115-3, swap the URIs for the mrl/mcc namespaces rather than editing every rule.
  • Codelist values look present but may be empty. CI_RoleCode and MD_ScopeCode carry their value in the codeListValue attribute, not in element text. A rule that only checks for the element’s existence will pass a role of codeListValue="". If you extend the rule set to validate roles, assert @codeListValue is non-empty and, ideally, a member of the official codelist.
  • normalize-space() matters for whitespace-only descriptions. A <gco:CharacterString> </gco:CharacterString> is technically present but semantically empty; the normalize-space(text()) predicate in the rule set is what rejects it. Without it, a step padded with spaces would pass and produce a meaningless audit record.