Today's Deep-Dive: Open Lineage

If you've ever found yourself staring at a dashboard,

you know, the one showing just garbage data all of a sudden,

you know, that panic, you realize your core data pipeline

just broke somewhere, and then trying to trace back

which transformation job caused it,

it feels like you're navigating this huge maze in the dark.

Yeah, that feeling, being totally exposed.

Right, that's precisely the problem

that's standardizing data observability is trying to solve.

Because as these data systems get bigger and more complex,

tracking the data's whole life cycle,

I mean, where it started,

every single transformation step who used it,

it's not just theoretical anymore.

It's absolutely critical for operations,

for trust, for everything.

Absolutely, and well, that's our mission for today, really.

We're taking a deep dive into open lineage.

It's the open standard design to finally bring some order

to that chaos we were talking about.

We really wanna give you a clear, beginner-friendly way in

to understand how this standard works, why it matters,

and how it changes the game for data integrity.

Yeah, it's foundational stuff.

It really is.

It's kind of the shortcut to understanding

how your data is actually born and transformed.

And we've looked at the core specs, the adoption docs.

The goal is you walk away really understanding

this key piece of the modern data world.

Okay, cool.

Now, before we jump into defining the problem

that Open Lineage actually solves,

just a quick note that this deep dive

is supported by Safe Server.

Ah, good point.

Safe Server focuses on hosting software

and helping out with your digital transformation.

You can find more info and resources

over at www.safeserver.de.

Check them out.

Definitely.

So let's start at the beginning then.

Before Open Lineage, what was the status quo?

I mean, how bad was the pain point that made everyone say,

okay, we need a common standard like right now?

Well, the pain point was pretty simple, actually.

It was duplication, cost, and things breaking all the time.

Every company that needed data lineage,

which is basically everyone, let's be honest,

had to build it themselves from scratch.

So every single project had to instrument,

set up the tracking for all of its jobs individually.

Yeah, so if you had say 10 different data tools,

five orchestration layers,

you were trying to maintain

like 15 separate custom tracking solutions.

That just sounds like a constant grinding drain

on engineering time.

It was, a massive maintenance liability

because those custom tracking things,

they were external to the actual data tools

like Spark or Airflow.

Oh, okay, so not built in.

Not built in.

They relied on specific internal APIs,

often undocumented ones.

So the moment the tool underneath, say Spark,

released a new version, poof,

your custom lineage script just broke.

Nightwear.

Constantly playing catch up, spending thousands,

literally just to keep your basic visibility running,

it was not sustainable.

Okay, so open lineage comes along

and completely flips that script, right?

Instead of every company building

these fragile external scripts,

the effort is shared across the community.

Exactly.

That's the beauty of an open standard.

And the really elegant part.

Now, the integration itself

can be pushed inside each project.

So instead of some external script

trying to like spy on a pipeline,

the pipeline component itself

speaks the lineage language, natively.

So it's embedded.

It's embedded.

The collection is intrinsic.

So you stop worrying about versions

breaking your tracking logic.

It's just there.

It makes so much sense.

And you know, for context,

this isn't some small side project.

Open lineage is an LFAI

and data foundation graduate project.

That means it's recognized, it's battle tested,

it's a proper industry standard.

Got it.

And fundamentally, this data lineage,

it provides the foundation.

It lets you build these powerful context-aware data tools

because it tracks all that metadata

about data sets, jobs, runs.

And at that deeper understanding,

you can pinpoint the root cause

of complex problems much faster.

And crucially, you can understand the impact of changes

before you make them,

before you accidentally break something downstream.

That really highlights the operational win, doesn't it?

Reducing that oops factor.

Exactly.

Fewer oops moments.

Okay, so we know why the standard exists now.

Let's dig into what it actually defines.

If open lineage is creating this shared language

for data tracking, what are the basic words,

the components?

Yeah, good question.

Think of open lineage as defining

a kind of universal data passport.

Okay, I like that, a passport.

Yeah, and this passport dictates the consistent naming,

the structure around three core things,

three core entities that you'll find in any data flow.

Those are the run, the job, and the data set.

Okay, run, job, data set.

So the run, that's like a specific execution

of some process.

Exactly, a single instance of a job running.

And the job is the transformation logic itself,

the code, the definition.

That's right, the definition of the work to be done.

And the data set is, well, the data,

the input or the output asset.

Precisely, the thing being read from or written to,

those three are the absolute foundational building blocks.

Okay, the minimum required pieces.

They are, but the real magic, I think,

the reason the standard can grow and adapt over time

is this concept they call the facet.

Okay, facet.

So the data world is always changing, right?

New regulations pop up, new kinds of tools,

new transformations.

How does a standard like open lineage

avoid becoming obsolete?

Is that where facets come in, like a plugin system?

That's exactly it, you hit it.

Facets are the plugin model.

A facet is defined as an atomic,

sort of self-contained piece of metadata.

Atomic meaning.

Meaning it describes one specific thing,

and you can attach this facet to any of those core entities,

the run, the job, or the data set.

Ah, okay, so if run, job, and data set are the nouns,

facets are like the descriptive adjectives

you can stick onto them.

That's a great way to put it, yes.

And this is absolutely key for things like governance.

How so?

Well, you can attach things like,

say, a regulatory compliance tag,

maybe the GDPR status of a data set.

Or you could attach a detailed schema fingerprint

right onto the data set entity.

Okay.

Or maybe attach quality check results to the run entity,

and because the whole specification is defined

using OpenAPI.

That standard API stuff.

Exactly.

Developers can extend the standard basically endlessly.

They just define their own custom facets

to track whatever proprietary details they need.

This ensures the standard evolves with the industry

and not behind it.

That makes the whole system incredibly flexible.

Yeah.

Kind of future-proof, doesn't it?

That's the goal, yeah.

So we have these core entities, run, job, data set,

and they can be decorated, enriched with these facets.

How does OpenLineage actually capture and send

this information around, especially with all the different

tools people use?

Right.

So OpenLineage also defines a standard API specifically

for capturing these lineage events.

An API call, basically.

Yeah.

So the different pipeline components

think your schedulers like Airflow, your data warehouses,

your analysis tools, SQL engines, whatever.

They use the standard API call.

To report back.

To send data about the runs, the jobs, the data

sets, and any relevant facets.

They package it up according to the standard

and ship this event off to a compatible OpenLineage backend.

Gotcha.

It sounds like the essential plumbing needed

to make all these different tools finally

speak the same lineage language consistently.

That's exactly what it is.

It's the common language.

And I saw the system allows for a configurable backend

so users can choose how those events are sent,

like which protocol.

Yeah, that gives you flexibility in your architecture.

You can choose how you want to receive and process

those events.

Makes sense.

OK, now that we understand the how, let's talk adoption.

Because looking around, this standard

seems to be getting, well, pretty significant traction.

It really is, yeah.

What's the biggest challenge you see folks facing

when they try to implement this?

Is it getting started, doing the initial instrumentation?

Or is it more about handling the sheer volume of metadata

once it's flowing?

That's a good question.

The initial instrumentation effort,

it's actually decreasing pretty rapidly now,

thanks to all the community contributions,

building integrations.

The bigger challenge often is achieving true column level

lineage, especially at scale.

Column level versus table level.

Can you quickly break that down?

Sure.

Table level lineage is basically knowing that, OK, data

move from table A to table B. Useful, but limited.

Column level lineage is knowing that this specific column

in table A was used to calculate that specific column in table

B. It's much more granular.

It's like knowing not just that the package arrived,

but exactly which truck carried the crucial piece of equipment

inside that package.

Ah, OK.

So that lets you trace a specific calculation error

or maybe a compliance issue with PII right back to the source

column.

Exactly.

It's essential for that deep analysis and root cause

finding.

And we see that some heavy hitters, like Apache Spark

and DBT, they're all in. They support both table level

and that more granular column level lineage.

Yeah, that strong adoption by major tools

is absolutely critical for the standard success.

However, there's always a nuance, right?

Listeners should know that column level lineage, while super valuable,

is also inherently complex to capture perfectly in all situations.

So you will find some, let's say, edge cases or specific ways tools

like Spark or Airflow handle certain complex SQL queries

or maybe specific connectors.

Well, like the note says, for Spark, sometimes tracking lineage

through select queries that hit a JDBC source can be tricky.

Or for Airflow, column level might work great for most SQL operators,

but maybe not for a very specific BigQuery operator

doing something complex.

Got it.

So the standard provides the map, but sometimes there

are tricky intersections depending on the specific tool.

That's a good way to put it.

The standard provides the blueprint, yeah.

But the devil can be in the integration details for each tool.

The good news, though, is the community

is super active in identifying and resolving

these tool-specific exceptions.

It's constantly improving.

That's great to hear.

It's really impressive how widely embraced it is becoming.

But open lineage isn't the only name people hear in this space.

How does it fit into the wider data ecosystem?

Are there other key projects that work alongside it

or maybe depend on it?

Yeah, definitely.

It's helpful to think of the ecosystem here.

Open lineage, as we said, defines the standard format,

the language, the electrical plug, if you like,

for data lineage.

OK, the standard plug.

It guarantees the format and structure

of the metadata signal.

Now, the question is, what do you plug

in to that standard outlet?

Right, so tell us about Marquez.

That name comes up a lot.

Marquez is essentially the reference implementation

of the open lineage API.

Think of it as the back end service and the UI

you plug into the wall.

OK.

It focuses on collecting all those open lineage events,

aggregating them, storing the history,

and then visualizing the metadata.

It gives you that dashboard view of your lineage.

So open lineage provides the raw data feed.

Marquez helps you actually see it and explore the history.

Exactly.

Open lineage is the language.

Marquez helps you read the story told in that language.

Got it.

And then there's another project, Egeria.

Where does that fit in?

Is it similar to Marquez?

Egeria is a bit different.

Think bigger picture.

If Marquez is the visualizer plugged into the open lineage

outlet, Egeria is more like the central switchboard

for your entire enterprise's metadata.

The switchboard.

OK.

It offers open metadata and governance capabilities

across the whole organization.

It's designed to automatically capture, manage,

and importantly, exchange metadata

between lots of different tools and platforms,

regardless of the vendor.

So it connects things.

Yeah.

So open lineage collects the raw standardized lineage data.

Marquez can visualize that specific data.

Egeria can take that traceable lineage data

from open lineage and other metadata sources

and share it intelligently across your entire governance

system, maybe feeding it to risk management tools or data

catalogs or data science platforms.

It helps integrate lineage into broader processes.

OK, that makes sense.

Open lineage for the standard feed,

Marquez for visualization and history,

Egeria for broader enterprise metadata management

and governance integration.

You got it.

They complement each other nicely.

And it's clearly a vibrant ecosystem

developing around this.

I mean, looking at the community stats for open lineage itself,

it's not just a theoretical paper, right?

People are actively using this.

2.1 thousand stars on GitHub, nearly 400 forks.

That's real activity.

That activity is crucial, absolutely.

It shows it's solving real problems.

And look at the primary languages being used

for the core project, Java at over 60%, Python around 25%.

That mix perfectly mirrors the modern data stack, doesn't it?

You often have execution engines like Spark running on the JVM,

Java, and then orchestration layers like Airflow

heavily using Python.

So that language mix ensures it fits naturally

into the places where lineage actually

needs to be generated, broad applicability.

Fantastic.

OK, so to kind of bring this all back home,

the core takeaway here seems to be that open lineage is really

this essential open standard.

It's transforming data tracking from this fragmented, custom

built mess.

Yeah, the old way.

Into a consistent, shared framework

for collecting lineage.

It's letting all the different systems in your stack

finally communicate lineage effectively,

and hopefully killing off those maintenance

nightmares of the past.

That's the promise, and increasingly the reality.

And if we connect this to the bigger picture for a second.

Please do.

What open lineage really means is that data observability,

it's no longer this specialized, separate tool

that you have to somehow bolt onto the outside

of your systems.

It's becoming a native function, an inherent capability that's

getting embedded directly into every key piece of your data

infrastructure, from Spark to Airflow to DBT.

It's built in.

That feels like a fundamental shift.

It is, and it raises a really important question

for the future, I think.

When every data movement, every transformation

is inherently traceable, because the tools speak

open lineage natively, how will that standardized,

built-in lineage fundamentally change things like compliance

or automated data governance?

What becomes possible then?

That is a compelling thought.

What happens when observability is just part of the fabric?

Something to definitely chew on as you're

planning your next data modernization project.

Well, thank you so much for walking us

through all that.

Really helpful.

My pleasure.

Thanks for having me.

And finally, a big thank you once again to SafeServer

for supporting this deep dive into open lineage.

SafeServer supports your hosting needs

and digital transformation.

We'll catch you on the next deep dive.

We'll catch you on the next deep dive.