What is data lineage and why does the AI Act require it?

What is data lineage

Data lineage is the complete traceability of a data's journey: from its origin in a source system — a transactional database, an external file, an API, a sensor — to its final point of consumption, which could be a dashboard, a business API, a machine learning model or a regulatory report. Along the way, the data undergoes multiple transformations: it is cleaned, aggregated, joined with other data, recalculated, filtered. Lineage records each of these steps.

In practical terms, lineage answers three questions that in many organisations today don't have an immediate answer:

• Where does this data come from? Which system generated it, when and under what conditions.
• What happened to it along the way? What transformations, filters, aggregations or enrichments it underwent before reaching its destination.
• Where is it used? What reports, models, APIs or decisions depend on this data. Essential for assessing the impact of a quality problem.

Types of data lineage: technical, business and operational

Not all lineages are the same. Depending on the level of detail and the context they capture, three types can be distinguished that complement each other:

Technical lineage

Records the physical journey of the data at the system level: which table feeds which table, which pipeline executes which SQL transformation, which Airflow job runs which task on which dataset. It is the lineage that tools like dbt and OpenMetadata generate automatically from code and platform metadata. It is necessary but not sufficient: knowing that table fact_revenue comes from raw_transactions doesn't explain what business rule was applied to calculate it.

Business lineage

Adds business context to the technical journey: what business transformation each step represents, what rule was applied, who defined and approved it, and whether the resulting data is subject to any regulatory or privacy restrictions. This is the lineage that Data Stewards enrich manually on the automatic technical base. It is the one that answers why the data is worth what it is worth, not just how it was calculated.

Operational lineage

Records specific executions: when the pipeline was run, with what input data, what volume it processed, whether there were errors and what outputs it generated. It is the lineage that allows reconstructing exactly what happened in a specific execution, essential for investigating quality incidents or responding to the regulator about a specific decision of an AI system.

Why the AI Act makes lineage an obligation

The AI Act doesn't explicitly mention the word "lineage", but its articles 10 and 12 presume it directly. For high-risk AI systems:

• Article 10 — Data governance: requires documenting training, validation and testing datasets: their origin, the transformations applied, selection and exclusion criteria, and representativeness metrics. Without technical and business lineage, this documentation is not traceable: it's a Word document that someone wrote once and that doesn't reflect the reality of the data.
• Article 12 — Event logging: requires that high-risk systems automatically generate sufficient logs to reconstruct the circumstances of any relevant decision. That is operational lineage applied to the model lifecycle: what input data it received, what output it produced, in what context and with what level of confidence.

The practical consequence is that an organisation deploying a high-risk AI system without documented lineage cannot comply with Article 10 sustainably. It can generate a one-off compliance document, but that document will become obsolete the moment the dataset or pipeline changes. Only automated lineage ensures that documentation stays up to date without manual work.

How data lineage works in practice

Automatic lineage with dbt

dbt is today the de facto standard for data transformation in modern stacks with Snowflake, BigQuery or Databricks. One of its most valuable capabilities for governance is automatic lineage: because dbt knows the dependencies between models — which SQL model depends on which other — it automatically generates a lineage graph showing the complete journey from source tables (source) to final models (mart).

End-to-end lineage with OpenMetadata

OpenMetadata connects the lineages of the different layers of the stack in a single navigable graph. Through its connectors, it imports the technical lineage from dbt, the ingestion lineage from Airflow or Fivetran, and the consumption lineage from Power BI or Tableau. The result is a complete view of the data's journey from the transactional system to the dashboard or AI model, with each step documented and navigable from the catalog interface.

Tools to automate data lineage

How to implement data lineage step by step

Step 1: start with critical domains, not everything

The most common mistake is trying to build lineage for all the organisation's data from day one. The result is a project that never ends. Start with the three to five domains that have the most impact on the business or regulatory compliance: the data that feeds high-risk AI systems, the data that underpins management reports or the data subject to specific regulation.

Step 2: automate technical lineage before documenting business lineage

Configure the lineage tool connectors with the data platforms before writing a single manual description. Technical lineage must flow automatically. If the technical graph depends on manual work, it will be outdated from the first change in code or schema. With dbt and OpenMetadata, this initial configuration can be completed in less than a week on standard stacks.

Step 3: assign Data Stewards to enrich business context

On the automatic technical graph, Data Stewards add business context: what business transformation each step represents, what rule was applied, who approved it and whether there are usage restrictions. This phase is iterative: you don't need to document everything at once. Start with the most consulted nodes — the tables or models that generate the most questions — and expand from there.

Step 4: link lineage with the catalog and access control

Isolated lineage has limited value. Its full potential is activated when it is integrated with the data catalog — where the user can see the lineage of any asset directly from its sheet — and with access control — where the Data Owner can see what data from their domain feeds what downstream systems before approving or denying access.

Step 5: document AI dataset sheets with lineage as the foundation

For each high-risk AI system, create a dataset sheet that uses lineage as the backbone: data origin, transformations applied (with reference to the lineage graph node), selection and exclusion criteria, and owner of each layer. This sheet is the documentary evidence of Article 10 of the AI Act. With automated lineage as the foundation, it stays updated automatically when the pipeline changes; without it, it's a static document that ages from the moment it's created.