Agentic Data & Software Engineering

Data lineage starts in the code that moves the data

By Xavier Franco, CEO de Elternativa · Grupo Zemsania
· 5 min read

Written with the help of AI and reviewed by Xavier Franco

Aerial view of a river delta, with all its branches of water fanning out from a single channel across sandy plains.
In this article

There is a question that sums up the problem of data governance: if you change that table, do you know which report breaks? There is almost never a quick answer. And it is not for lack of catalog tools. It is because data lineage has been treated as a diagram that someone draws, when in fact it is written somewhere else: in the code that moves the data.

What data lineage is

Data lineage is the path a piece of data follows from its origin to where it is used: which system it comes from, which processes transform it, which tables it is stored in, and which reports, metrics or applications read it.

It serves three very concrete purposes:

  • Trusting a figure, because you know where it comes from.
  • Knowing the impact of a change before making it.
  • Proving to audit or compliance how an indicator was calculated.

In practice: pick the most important indicator of your executive committee and ask which tables and which processes it comes from. Measure how long it takes them to answer.

Data and software don’t talk to each other

In many organizations, the data world and the software world run separately. The governance team maintains a catalog with a glossary, owners and, sometimes, lineage. The development team writes the processes that move the data: pipelines, transformations, procedures and loads.

The problem is that the catalog is updated by hand and the code changes every week. A drawn lineage ages from day one. And when you need it most, in a migration or an audit, it no longer matches what is running.

Lineage is written in the code

The good news is that lineage doesn’t have to be invented. It is in the code. Every transformation says what it reads and what it writes. Every report says which table it queries. Every procedure says which columns it touches.

That is why reliable lineage comes from reading the code, not from asking people. And also from capturing it when processes run: the open standard OpenLineage, for example, defines a common model of datasets, jobs and runs, and is designed to record the metadata of a process while it runs.

Reading the code and observing the execution are two sides of the same coin: lineage comes from what really happens, not from what someone believes happens.

In practice: ask your data team whether the lineage in your catalog is generated automatically or maintained by hand. If it is by hand, ask when it was last updated.

What is certain and what is inferred

There is a nuance that I think is important. Not all lineage is equally reliable. A relationship that comes directly from the code is certain. Another one that is inferred, for example from a similar column name, is probable.

A good lineage map has to tell the two apart. And when you click on a relationship, it has to tell you which file it comes from, which method, and with what confidence. Otherwise, the map looks more reliable than it is.

What breaks if you move it

Where the value of lineage shows most is in a migration. Before moving a process, a table or a report, you need to know what depends on it. By following the relationships on the map, you see the impact before touching anything.

And you can narrow it down: start from a screen or a report and see only what it reaches. It is the difference between migrating with a plan and migrating with fear.

In practice: before approving any change to a shared table, ask for the list of what reads it. If it doesn’t exist, the change is a bet.

When a figure doesn’t add up

Lineage is also the most useful tool when something fails. An indicator in a report comes out different from what was expected, and the usual question begins: is it the source data, a transformation or the report itself?

Without lineage, the answer is an investigation that goes through several teams. With lineage drawn from the code, it is a walk: you follow the figure backwards, process by process, until you find where it changes. And when you find it, you also know which other reports read from that same point and may be affected.

It is the difference between saying “we’re looking into it” and saying “it comes from this transformation, and it affects these three reports”.

In practice: the next time a figure doesn’t add up, measure how long your team takes to find the origin. That time is what lineage gives back to you.

New code, with its lineage from day one

All of the above applies to the code you already have. But there is another opportunity: new code. With AI writing more and more pipelines and transformations, lineage can be generated at the same time as the code, not afterwards.

If every process that is written leaves behind its technical lineage and its metadata, and they are published to your governance catalog, the catalog stops lagging behind. Data and software, at last, talk to each other.

Where to start

  1. Pick one critical indicator and rebuild its lineage.
  2. Get the lineage from the code, not from interviews.
  3. Tell the certain from the inferred.
  4. Use the map before every change to see the impact.
  5. Make new code publish its lineage to your catalog.

How we help

At Agentic Data & Software Engineering we orchestrate development AIs with data governance as a cross-cutting layer: the technical lineage and metadata of the generated code are published to your governance catalog, such as Informatica, Collibra or Talend.

And with the Datagentis migration module we draw the map of the system you already have: reports, tables, metrics and screens connected in a single graph, with certain and inferred relationships told apart, the provenance of each piece of data, and what breaks if it is moved.

Frequently asked questions

What is data lineage?

It is the path a piece of data follows from its origin to where it is used: which systems, processes, tables and reports it passes through.

How do you obtain data lineage?

Reliably, by reading the code that moves the data and capturing the metadata of processes when they run, rather than maintaining it by hand.

What is OpenLineage?

An open standard for collecting and analyzing data lineage, with a common model of datasets, jobs and runs.

What is lineage for in a migration?

To know what depends on each table, process or report and to see the impact of moving it before doing so.

Keep reading