Extrinsic Provenance at Entry Tables¶

Overview¶

Rows that enter a pipeline from outside carry a hidden _prov attribute recording where they came from. The framework declares the attribute on Manual tables and fills it on insert; no author writes it.

New in 2.3.4

dj.Entry is available from 2.3.4 as a permanent alias for dj.Manual. This page uses dj.Manual throughout; both names declare the same table.

Motivation¶

Inside a pipeline, provenance is structural. A Computed table's row cannot exist unless its declared upstream exists and is correct, so the foreign-key graph is the lineage. Nothing has to be recorded for that to hold.

At the boundary the structure runs out. Rows arrive in Manual tables from a person, an instrument, or an automated feed, and the graph has nothing to say about where they came from. Before 2.3.4 each pipeline answered that question its own way — a source_file column here, a notes varchar there, an ingestion log somewhere else, or nothing at all.

_prov gives the boundary one shape, so that two pipelines answer "where did this row come from" the same way, and so that "which rows have no recorded origin" is a query rather than an audit.

The Attribute¶

Attribute	Type	Description
`_prov`	`json` (MySQL) / `jsonb` (PostgreSQL)	Extrinsic provenance for a row that entered from outside the pipeline. `NULL` when nothing was recorded.

Hidden attributes are prefixed with _: stored in the database, filtered out of heading.attributes, and excluded from query composition.

Why JSON rather than typed columns. The useful key set is not settled, and deployments add to it. A JSON document absorbs a new key as a configuration change; typed columns would freeze the set at declaration and make every later addition an ALTER across every Manual table in every deployment. Filtering on a field stays portable — see Querying.

Which Tables Carry It¶

Manual tables only.

Tier	Carries `_prov`	Why
`dj.Manual`	Yes	The pipeline's boundary with the outside world
`dj.Lookup`	No	Rows come from the committed `contents`, so the code is the record
`dj.Imported`	No	See below
`dj.Computed`	No	Provenance is entailed by the foreign-key graph
`dj.Part`	No	A part inherits its master's

The slot is granted by matching the Manual tier, not by excluding the other tiers' prefixes, so DataJoint's own system tables — job queues, lineage — never carry it either.

Why not Imported. An Imported table already records agent, time and version through job metadata when config.jobs.add_job_metadata is on, so a _prov there would record the same facts twice. The half that is not covered — which specific file, endpoint, or instrument session its make() read — is known per row inside the make() body, which configuration cannot supply.

In a well-modeled pipeline the external source is registered as a Manual row and the Imported table reaches it through a declared foreign key, which makes that table's provenance structural. An Imported table reading a source no Manual row records is the modeling problem described in Table Declaration; the fix is to register the source, not to add a slot.

Content¶

Three sources fill the attribute, and none of them is the insert call site.

Source	Supplies
Configuration	`config.provenance.source` — the deployment constant naming the external system this process draws from
Ambient connection state	the connecting user, host and database; the insert time; the code version
Ambient execution state	the ingesting table and key, when the insert runs inside a `make()`

A recorded document:

{
  "time": "2026-09-30T14:22:05.481203+00:00",
  "agent": {"user": "ingest_svc", "host": "db.example.org", "database_name": "lab_subjects"},
  "version": "a1b2c3d",
  "source": {"system": "PyRat", "endpoint": "https://pyrat.example.org/api/v2"},
  "context": {"table": "`lab`.`_ingest`", "key": {"file_id": 7}, "version": "a1b2c3d"}
}

Keys are omitted when they have nothing to report: source when none is configured, context outside a make(), version when config.jobs.version_method is disabled. When only time would remain the attribute is left NULL, because a bare timestamp says nothing about origin.

time is the client's UTC clock at insert, not the server's.

The author cannot write it¶

insert() takes no provenance argument, and passing _prov in a row raises KeyError — hidden attributes are not in the heading.

This is the design rather than a limitation. A field an operator can set is weaker evidence than one the system sets, which is what attributable and contemporaneous require of externally-sourced data. It also removes the failure mode a supported-but-optional field would have: there is nothing left for a pipeline to neglect.

Anything an author wants to record deliberately belongs in the data model, as an ordinary visible attribute. Pipeline code can restrict and join on a modeled column; it cannot on a hidden one. The two do different jobs — _prov is the audit record, a modeled column is the domain link. See Fan-Out Ingestion, where both appear side by side.

Configuration¶

Setting	Environment	Default	Description
`provenance.capture`	`DJ_PROVENANCE_CAPTURE`	`False`	Declare `_prov` on Manual tables and fill it on insert
`provenance.source`	`DJ_PROVENANCE_SOURCE`	`{}`	External source identity recorded on every row this process enters

import datajoint as dj

dj.config.provenance.capture          # False until a deployment enables it
dj.config.provenance.source = {"system": "PyRat", "endpoint": "https://pyrat.example.org/api/v2"}

Deployments set these where they set stores and credentials, not in pipeline code:

export DJ_PROVENANCE_SOURCE='{"system": "PyRat", "endpoint": "https://pyrat.example.org/api/v2"}'

{
    "provenance": {
        "source": {"system": "PyRat", "endpoint": "https://pyrat.example.org/api/v2"}
    }
}

Changing the source mid-process is silent

Rows inserted before the change keep what was configured then, and rows after keep the new value. Nothing records that the setting moved. Set it once at start-up.

Capture defaults off¶

Capture changes the DDL of every Manual table declared after it is enabled, adding one hidden nullable column. That is a deployment's decision rather than a library default, so upgrading to 2.3.4 leaves an unchanged schema declaring exactly what it declared under 2.3.3. jobs.add_job_metadata defaults off for the same reason, and does the same kind of thing.

A deployment that turns it on gets the property that makes the slot worth having: across that deployment, "which rows have no recorded origin" is a query rather than an audit. What it cannot assume is that a table declared elsewhere, under someone else's configuration, carries the column — retrofitting is what settles that.

Behavior¶

At declaration¶

With provenance.capture true, _prov is added to the CREATE TABLE of every Manual table that is not a part. Turning capture off later does not remove the column from tables that already have it.

At insert¶

Every insert() and insert1() into a Manual table that carries the column appends the assembled document. A table declared without the column is left alone and the insert succeeds — silently, which is what retrofitting addresses.

Inside `make()`¶

An insert executed inside a make() records the ingesting table and key in context. This is what makes the fan-out ingestion pattern traceable: rows written into Manual tables that carry no foreign key back to the loader still record what wrote them.

Retrofitting Existing Tables¶

Tables declared before 2.3.4, or while capture was off, have no column and record nothing. datajoint.deploy.add_prov_column adds the slot:

from datajoint.deploy import add_prov_column

# Preview
add_prov_column(schema, dry_run=True)["ddl"]

# Apply to every Manual table in a schema
add_prov_column(schema, dry_run=False)

# Or a single table
add_prov_column(Subject, dry_run=False)

It is idempotent — a table that already has the column is reported and left alone — and it lives in datajoint.deploy rather than datajoint.migrate for that reason.

Rows already present keep NULL. Provenance is recorded at insert and is never reconstructed after the fact.

Querying¶

_prov is excluded from heading.attributes, so it does not appear in to_dicts(), in describe(), or in a join.

Restricting on it works, written as a SQL condition string:

# Rows with no recorded origin — portable
Subject & "_prov IS NULL"

Filtering on a field inside the JSON needs backend-specific SQL because the attribute is hidden, not because JSON paths are awkward. On an ordinary JSON attribute the mapping form is portable — DataJoint translates {"data.system": "PyRat"} to json_value() on MySQL and jsonb_extract_path_text() on PostgreSQL. That translation is unavailable here only because the mapping form cannot reach a hidden attribute (below):

# MySQL
Subject & "JSON_VALUE(_prov, '$.source.system') = 'PyRat'"

# PostgreSQL
Subject & "jsonb_extract_path_text(_prov, 'source', 'system') = 'PyRat'"

The mapping form does not reach a hidden attribute

Subject & {"_prov.system": "PyRat"} returns every row. A mapping restriction ignores attributes it cannot match, which is deliberate and useful — it is what lets Session & key work when key carries attributes from a more detailed table. A hidden attribute is invisible to that matching, so the predicate is dropped along with it.

Write the condition as a string, which reaches the column directly — at the cost of portability, since DataJoint's own JSON-path translation is what the mapping form would have given you. Tracked in datajoint-python#1561.

Reading the value back requires SQL until 2.4. There is no public API that returns a hidden attribute — to_arrays('_prov') and proj('_prov') both raise:

# Until 2.4
rows = Subject.connection.query(
    f"SELECT subject_id, _prov FROM {Subject.full_table_name}"
).fetchall()

A supported accessor is planned for 2.4 (datajoint-python#1562), which will cover _prov and the job-metadata attributes together.

Range queries on capture time

Filtering by time goes through JSON extraction, so it does not use an index. If that becomes hot for a deployment, add a generated column over the JSON path and index it; no change to the pipeline or to DataJoint is needed.

Implementation Details¶

Module	Role
`datajoint/provenance.py`	payload assembly and the ingesting context
`datajoint/settings.py`	`ProvenanceSettings`, exposed as `config.provenance`
`datajoint/declare.py`	`PROV_DEFINITION`, and adding it to Manual tables at declaration
`datajoint/user_tables.py`	`is_tier`, the shared tier test
`datajoint/table.py`	appends the value on the insert path
`datajoint/autopopulate.py`	scopes the ingesting context to a `make()` call
`datajoint/deploy.py`	`add_prov_column`

The column is declared the way a user attribute is — _prov = null : json # extrinsic provenance ... — and compiled by the same compile_attribute, so the backend mapping to json or jsonb comes from the adapter's type system rather than from a per-adapter method. No adapter implements anything of its own for it.