Today's Deep-Dive: OpenLineage

Have you ever stopped to really think about the journey your data takes, not just

where

it starts and ends up, but that whole messy path in between the changes, the

systems,

the jobs?

Right.

It's complicated.

Especially now, everything's so connected that data flow can be, well, pretty

chaotic.

It really can.

And when something breaks down or, say, a compliance report gets questioned.

Oh, yeah.

Then data teams are scrambling, trying to figure out where things went wrong across

maybe dozens of different tools.

It's like chasing ghosts.

A real productivity killer, I bet.

Absolutely.

Yeah.

That operational friction is huge.

Well, today we're diving into something designed to bring some order to all that

chaos.

Open lineage and this deep dive.

It's really for you, the learner.

We want to give you a clear starting point on what data lineage actually is, why we

need

an open standard for it, and how open lineage tackles some really big industry

problems.

And before we jump in, we want to thank the supporter of this deep dive.

Safe Server commits to hosting this software and supports you in your digital

transformation.

More information under www.safeserver.de.

So open lineage really comes at a critical time.

The sources we looked at, they call it the foundation for a new generation of

powerful

context-aware data tools.

Context-aware.

Interesting.

Yeah.

It's about collecting metadata consistently.

If you can do that, it fundamentally changes how people understand their data, how

they

trust it, how they see the impact of changes.

We're sort of moving away from siloed guesswork.

Towards a kind of universal data history.

Exactly.

That's the goal.

Okay.

Let's unpack this a bit.

Before we get into all the technical nuts and bolts, maybe we should just quickly

define

data lineage itself.

Good idea.

Keep it simple.

Right.

So stripping away the jargon, it's basically the traceable history of your data.

It tracks metadata about data sets, the jobs that process them, and the specific

times

those jobs ran the runs.

And the real goal there is simple.

If something breaks.

You can find the root cause, like which specific job run messed up the data.

Precisely.

Or, before you deploy a change, you can understand what downstream things might be

affected.

Okay.

So that capability sounds pretty essential, but you mentioned earlier it used to be

a

massive headache.

Oh, absolutely.

Before open lineage, the main pain points were, well, friction and fragility.

Think about a big company with maybe 50 different data projects.

Using all sorts of different tools, I imagine, Spark, Airflow, maybe some custom

stuff.

Exactly.

And they all needed lineage.

But here's the kicker.

Every single project had to build its own way of tracking it.

Wow.

So massive duplication of effort.

Huge duplication.

Teams were constantly writing custom code just to get basic tracking.

It was slow, expensive, and created a ton of technical debt.

I can just picture the maintenance nightmare.

What if they tried using some external tool to just, like, scrape the logs or

something?

Ah, yeah, that led to the other big problem, broken integrations, these external

tools.

They were often really brittle.

How so?

Well, say, Spark released a new version or even just changed how it logged things.

Poof, the external integration breaks.

Instantly.

Often, yeah.

So data teams would spend weeks just trying to catch up, fixing the integration,

which

meant...

They had no lineage data during that time, right, when they might need it most.

Exactly.

The lineage ended up being expensive, unreliable, and almost always incomplete.

It just wasn't working well.

Okay, so open lineage flips that script through collaboration.

That's the core idea.

Yeah.

Instead of 50 projects building 50 systems.

The effort is shared, baked right into the platforms themselves.

Yes, precisely.

By having a standard API, the exact format for the metadata integrations can be

written

once and built directly into tools like Spark or Airflow, like an internal module.

Ah, so in Spark update.

The lineage tracking updates with it.

No more constant catch up.

It dramatically cuts down that maintenance burden.

And I think you mentioned something interesting.

It instruments jobs as they run.

Yeah, that's a key aspect.

It's not trying to piece things together from logs after the fact.

Which means the metadata is captured in real time.

Exactly.

Yeah.

More accurate, more complete, especially for things like stream processing where

stuff is

happening constantly.

Okay, that makes sense.

Let's get into the standard itself then.

The definition is an open standard for metadata and lineage collection, built on a

generic

model.

What are the core pieces of that model we need to understand?

Right, so there are three core entities, the sort of nouns of the model.

They use consistent naming, which is important.

They define the what, the how, and the when.

Okay.

First, the data set.

That's the simplest, just the data being read or written.

It could be a database table, a file, a Kafka topic.

It doesn't matter.

It gets a consistent name.

Got it.

What's next?

Second, the job.

This is the defined task or process itself.

Think of it like the blueprint, like run daily sales report.

That definition.

A reusable blueprint.

Okay.

And the third, the run.

Yes, the run.

And this is crucial.

It separates the blueprint from the actual execution.

So the run is the specific instance when that job executed.

If your daily sales report, job runs every midnight.

The run is Tuesday's execution at 12 00 00 01 AM.

Why is that separation so important?

For debugging.

If a dashboard is wrong, you need to know.

Did the job definition, the code change or did this specific run fail because of

something

else like bad input data?

Yeah.

That distinction is key.

Makes perfect sense.

Okay.

So run job data set.

That gives us the basics, but you said it handles all sorts of environments, right?

ML, SQL.

How does it get that flexibility?

You mentioned something called a facet.

Yes.

Facets.

That's where real power and extensibility comes in.

So what is a facet?

The sources call it an atomic piece of metadata.

Think of it like a standardized label or a little packet of context that you attach

to

one of those core entities, a run, a job, or a data set.

An atomic piece, like a building block.

Exactly.

The core model is simple on purpose, but it's extensible because you can define

these specific

facets to add rich, relevant details.

Okay.

Give me an example.

Let's say a job uses Python.

Right.

You could attach a facet to the run entity that lists the exact Python library

versions

used during that specific run.

Or for a data set.

You could have a schema facet detailing the columns and types.

Or for a job, maybe a source code facet with the git commit hash that defined the

job logic

for that run.

I see.

So you can add very specific context column details, performance stats, code

versions

without bloating the core standard itself.

Precisely.

It allows the standard to adapt to new technologies and specific needs without

breaking that fundamental

model.

And you mentioned it uses OpenAPI.

How does that help?

Well, OpenAPI gives you a structured way to define APIs.

Because the open lineage standard and its facets are defined using OpenAPI, it

makes

it really easy for the community to propose, review, and add new custom facets for

specific

use cases.

Ah.

So even very specialized metadata can fit neatly into the standard structure.

Exactly.

So everything's consistent but also flexible.

So connecting this back to you, the learner, that structure core entities plus extensible

facets is why open lineage is potentially so powerful.

It gives you that deep context when you need it for any kind of data task while

maintaining

that universal language for tracking.

Right.

Structured but flexible.

OK.

So the design sounds solid.

But a standard is only good if people actually use it, right?

Open lineage is an LF, AI, and data foundation graduate project.

What does that mean, and what does real world adoption look like?

The graduate status is basically a seal of approval from the Linux Foundation.

It means the project is mature, well-governed, and has significant community

backing.

It's a big deal.

OK.

And the adoption.

How's the integration matrix looking?

It's actually pretty impressive, which tells you the pain point was real.

People are definitely implementing it.

What kind of details does the matrix track?

Generally, it looks at two main levels of detail.

First, table-level lineage.

So basically, did this job read from or write to this table?

Simple connection.

And the second.

The much more detailed column-level lineage.

Did this job use this specific column in this table to produce that specific column

in another

table?

Ah, that's the really granular stuff you need for compliance or impact analysis.

Exactly.

That's the gold standard.

So walk us through some highlights.

Who's integrated?

What level do they support?

OK.

Let's start with Apache Spark, huge player in big data.

Right.

They use both table-level and column-level lineage, which is great.

But there's always a but, isn't there?

Usually.

There's a key caveat.

Yeah.

It currently doesn't support column-level lineage if you use select queries with gdbc

data sources.

Interesting.

So if you want that detailed column tracking...

You have to explicitly list the columns you're selecting, which honestly is

probably bad

practice anyway.

Good point.

It nudges you towards better coding.

What about workflow tools, like Airflow?

Apache Airflow, and yeah, probably the most popular orchestrator.

It also supports both table and column-level lineage.

It can integrate directly with many SQL operators, so Airflow itself reports what

its tasks

are doing.

Nice.

And newer tools, like dbt.

Yep.

dbt is natively integrated too, supporting both levels.

Crucial for tracking those complex multi-stage data models people build with it.

Okay.

What about real-time stream processing?

Good question.

Flink, a major stream processing engine, currently supports table-level lineage.

But not column-level yet.

Not yet, according to the sources.

It highlights that instrumenting those continuous real-time flows is technically

challenging.

Yep.

But they're starting with the most critical info, which data sources are involved.

So the adoption is growing, covering key platforms.

Definitely.

And this standardization of collecting the metadata is enabling a whole ecosystem

to

build up around it.

Right.

The sources mention related projects.

Tell us about those.

How do they fit in?

Okay.

So the first is Marquez.

Marquez.

Yeah.

If Open Lineage defines the standard for creating the lineage metadata, think of

Marquez as

the reference implementation for using it.

Oh, okay.

So what does it do?

It's another LFAI and data project.

Its job is to collect all that Open Lineage metadata, aggregate it, store it, and

crucially

provide ways to visualize it.

It gives you a UI to actually explore your data ecosystem's lineage.

The display case for the metadata Open Lineage collects.

Nice.

And the other one, Igeria.

Igeria plays a slightly different role.

It's more focused on broader open metadata management and governance across an

entire

enterprise.

How does it relate to Open Lineage?

It acts like a broker or exchange.

Igeria aims to automatically capture and share metadata between all sorts of

different tools

and platforms, whatever the vendor.

So Open Lineage provides the detailed, trusted lineage data, and Igeria helps

integrate that

data into your company-wide governance, discovery, and security tools.

I see.

So Open Lineage focuses on generating the core lineage events.

Marquez provides a way to see and explore that lineage.

And Igeria helps connect that lineage information into the bigger enterprise

picture.

You got it.

Together, they're building this foundational layer for universal data observability.

It's moving things away from being locked into one vendor's proprietary system.

Making lineage more open, more democratized.

Exactly.

Okay.

We've covered a lot of ground.

To summarize for everyone listening, open lineage is this vital open standard.

It shifts that heavy linting of instrumentation away from individual teams.

Shared effort.

And enables consistent detailed tracking of data runs, jobs, and data sets.

And it does this using that flexible model with core entities and those extensible

facets.

Right.

And the adoption is real.

That integration matrix shows it's getting picked up by major platforms.

Yeah.

And so for the learner, the big takeaway is that having this standardized shared

metadata

governed by projects like open lineage, it really changes the game for data trust.

How so?

Well, for security, for audits, for just basic operational efficiency.

Knowing the true history of your data becomes feasible at scale.

It's fundamental for building reliable data systems.

So here's a final thought to leave you with.

Imagine if every single piece of data processing, whether it's a simple query or a

super complex

ML model training.

What if it could automatically report its own complete verifiable history using

open

lineage?

What kinds of new applications could that unlock?

Think about automated governance or real time security checks or maybe even new

forms of

business intelligence based on process history.

Yeah.

If the data reports itself, that opens up a lot of possibilities.

The future of data might just be self-reporting.

A provocative thought indeed.

Thank you for joining us on this deep dive into open lineage.

We hope this helps you see not just the data, but the flow, the connections, the

behind it.

behind it.