Capturing Processing Provenance in QGIS with PyQGIS

Part of: Metadata Injection Techniques

Every run of a QGIS Processing algorithm — a buffer, a clip, a reprojection — is a transformation that should leave a provenance trail, but the graphical Processing Toolbox discards its parameters the moment the dialog closes. This how-to shows how to wrap processing.run() in PyQGIS so each execution writes a structured lineage record, and how to reconcile those records against QGIS’s own processing history. It belongs under Metadata Injection Techniques and complements the reusable patterns in Workflow Hooks in Python Pipelines.

Prerequisites

  • QGIS 3.34 LTR or newer, run either from the built-in Python Console or as a standalone PyQGIS script with the QGIS environment initialized.
  • The processing framework must be available. Inside QGIS it is imported directly; standalone scripts call QgsApplication.initQgis() and register Processing.initialize() first.
  • Write access to a directory for the JSON lineage records.
  • Input layers with a defined CRS; algorithms that silently assume a project CRS are the main source of untracked reprojection.
Two capture routes in QGIS, and what each can see Reading the processing history log recovers algorithm and parameters but not digests or intent; wrapping processing.run captures everything but only for calls that go through the wrapper. Route A · read the history log after the fact, newest first Route B · wrap processing.run at call time CAPTURES algorithm id + parameters yes yes input / output digests no yes failed runs no yes work done via the GUI yes no The last row is why most agencies need both: analysts do not run everything through a script.

The bottom row is the one that decides the architecture. A wrapper around processing.run gives you complete, digest-bearing records for scripted work and sees nothing at all of what an analyst does interactively in the Processing Toolbox — which, in most GIS teams, is where a large share of the real transformation happens. The history log is the only route that observes GUI work, and it carries no digests.

Run both and reconcile. The wrapper is the system of record for automated pipelines; a scheduled sweep of the history log identifies interactive operations that produced or altered tracked datasets, which then get enriched with digests computed after the fact. Records from that second path should be marked as reconstructed rather than observed, because the digest was taken later than the operation and cannot prove what the input looked like at the time.

Implementation

The wrapper below captures the algorithm id, the fully resolved parameter dictionary, the input layer sources and their CRS, and the output paths, then hashes the parameter set so identical re-runs are detectable. It writes one JSON record per execution.

from __future__ import annotations

import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

import processing
from qgis.core import QgsProcessingContext, QgsVectorLayer


def run_with_lineage(
    algorithm_id: str,
    params: dict[str, Any],
    lineage_dir: str | Path,
) -> dict[str, Any]:
    """Run a QGIS Processing algorithm and write a lineage record for the run.

    Args:
        algorithm_id: Provider-qualified id, e.g. "native:buffer".
        params: Parameter dictionary passed straight to processing.run().
        lineage_dir: Directory that receives the .json lineage record.

    Returns:
        The lineage record, including the algorithm's output dictionary.
    """
    out_dir = Path(lineage_dir)
    out_dir.mkdir(parents=True, exist_ok=True)

    started = datetime.now(timezone.utc)

    # Describe every input layer so CRS and source path are captured, not just the id.
    inputs: list[dict[str, str]] = []
    for key, value in params.items():
        if isinstance(value, QgsVectorLayer) and value.isValid():
            inputs.append(
                {
                    "param": key,
                    "source": value.source(),
                    "crs": value.crs().authid() or "UNKNOWN",
                    "feature_count": str(value.featureCount()),
                }
            )
        elif isinstance(value, str) and Path(value).exists():
            inputs.append({"param": key, "source": value, "crs": "UNRESOLVED"})

    context = QgsProcessingContext()
    result = processing.run(algorithm_id, params, context=context)

    finished = datetime.now(timezone.utc)

    # Hash the string-normalized parameters so re-runs with identical inputs collide.
    param_repr = json.dumps({k: str(v) for k, v in params.items()}, sort_keys=True)
    param_hash = hashlib.sha256(param_repr.encode("utf-8")).hexdigest()

    record: dict[str, Any] = {
        "event": "processing_run",
        "algorithm": algorithm_id,
        "parameter_hash": param_hash,
        "inputs": inputs,
        "outputs": {k: str(v) for k, v in result.items()},
        "started_at": started.isoformat(),
        "finished_at": finished.isoformat(),
        "duration_s": round((finished - started).total_seconds(), 3),
    }

    record_path = out_dir / f"{param_hash[:16]}.json"
    record_path.write_text(json.dumps(record, indent=2), encoding="utf-8")
    return record


if __name__ == "__main__":
    layer = QgsVectorLayer("/data/wells.gpkg|layername=wells", "wells", "ogr")
    out = run_with_lineage(
        "native:buffer",
        {
            "INPUT": layer,
            "DISTANCE": 500,
            "SEGMENTS": 8,
            "OUTPUT": "/data/out/wells_buffer.gpkg",
        },
        lineage_dir="/data/lineage",
    )
    print("Wrote lineage for", out["algorithm"], out["parameter_hash"][:12])

The single most useful line is the param_repr normalization: Processing parameters mix layer objects, numbers, and enums, so stringifying under sort_keys=True yields a stable hash that is comparable across sessions and machines.

Turning a Model into a Chain of Steps

A model is many steps, not one A single Processing model invocation expands into three algorithm runs; recording only the model loses the intermediate derivations and their parameters. What you called model:floodprep expands to → native:reprojectlayer native:buffer native:dissolve Recording only "model:floodprep ran" loses the CRS change, the buffer distance, and the dissolve. The dissolve is also where feature identity stops being one-to-one — the step that most needs recording.

Models are the common case in QGIS-centred teams and the easiest place to lose resolution. A single processing.run("model:…") call is one invocation to your wrapper and three or thirty transformations in reality, and a record that names only the model cannot answer which CRS the data was reprojected into or what buffer distance was applied. Attach a QgsProcessingFeedback subclass to the run and record each child algorithm as its own step, linked to a parent identifier for the model invocation.

The dissolve in the example illustrates why this is more than completeness pedantry. Dissolving collapses many input features into fewer output features, so feature-level identity ends there — and a lineage record that never mentions the dissolve implies a one-to-one correspondence that does not exist. Any downstream analysis that tries to trace an output polygon back to a specific input parcel will produce a confident, wrong answer.

Reconciling GUI Work Against the Wrapper

Reconciling two capture routes History-log entries and wrapper records are compared; the three resulting classes each require a different response. history log entries wrapper records reconcile in both → nothing to do history only → GUI work enrich, mark reconstructed wrapper only → history off a configuration defect The third class is the surprising one — it means the log you rely on for GUI coverage is not recording.

Run the reconciliation on a schedule rather than on demand, because the value is in noticing the third class early. A wrapper record with no corresponding history entry means logging was disabled or the profile was reset, and until that is fixed your GUI coverage is zero without any visible symptom. Alert on it as a configuration defect rather than filing it as a data discrepancy, and treat the period since the last matched pair as an interval whose interactive work is simply unrecorded.

Verification

QGIS records every Processing run in its own history log. Cross-check that your captured records line up with QGIS’s internal history:

from processing.core.ProcessingLog import ProcessingLog

# QGIS stores its processing history as newest-first log entries.
history = ProcessingLog.getLogEntries()
for entry in history[:5]:
    print(entry.date, entry.text)  # text includes the algorithm id and params

Each captured JSON record should have a corresponding QGIS history line with the same algorithm id and timestamp within the run window. If a record exists with no history entry, the algorithm was invoked outside your wrapper and its provenance is incomplete.

Configuration Reference

Parameter Value Why
output destination a real file path, never memory: Memory layers die with the session and leave an unresolvable source
provider registration explicit, before first processing.run GRASS, GDAL and SAGA algorithms fail silently in standalone scripts otherwise
history read order reverse the log — QGIS stores newest first Reading in file order produces a backwards process sequence
context / feedback a QgsProcessingContext you own The default context makes failures harder to attribute to a run
record source resolved absolute path, not the layer name Layer names are display labels and are not unique
capture on exception wrap in try/finally A failed algorithm is a lineage fact, not an absence

The history-order item catches people repeatedly because a script that reads the log and writes steps in the order it encountered them produces a lineage chain running backwards in time. It validates, it looks plausible, and every ancestry query returns the wrong direction. Reverse explicitly rather than relying on the order you happen to observe.

Resolving outputs to absolute paths matters for a subtler reason than uniqueness. QGIS layer names change when a user renames a layer in the panel, with no effect on the underlying data, so a record keyed on the display name silently detaches from the file it describes the moment somebody tidies their project.

Gotchas & edge cases

  • On-the-fly reprojection hides CRS changes. When INPUT and OUTPUT layers differ in CRS, native:buffer and many geometry algorithms reproject silently using the project’s transform. Record crs().authid() for every input as shown, and add the output CRS after the run, or your lineage will not reveal that a datum shift occurred.
  • Memory-layer outputs vanish. Passing "OUTPUT": "memory:" returns a QgsVectorLayer that is never written to disk, so the source in your record points to a transient id that dies with the session. For durable provenance, always resolve outputs to a file path such as a GeoPackage before logging.
  • Interactive work is invisible to a wrapper. Anything an analyst runs from the Processing Toolbox bypasses your processing.run wrapper entirely. Reconcile against the history log on a schedule, and mark those records as reconstructed.
  • Non-native providers may not be initialized. Algorithms from GRASS, GDAL, or SAGA providers raise QgsProcessingException if their provider was not registered in a standalone script. Register providers before the first processing.run() call, and route the resulting records through your shared workflow hooks so QGIS runs are logged with the same schema as the rest of the pipeline.