If you've ever found yourself staring at a dashboard,
you know, the one showing just garbage data all of a sudden,
you know, that panic, you realize your core data pipeline
just broke somewhere, and then trying to trace back
which transformation job caused it,
it feels like you're navigating this huge maze in the dark.
Yeah, that feeling, being totally exposed.
Right, that's precisely the problem
that's standardizing data observability is trying to solve.
Because as these data systems get bigger and more complex,
tracking the data's whole life cycle,
I mean, where it started,
every single transformation step who used it,
it's not just theoretical anymore.
It's absolutely critical for operations,
for trust, for everything.
Absolutely, and well, that's our mission for today, really.
We're taking a deep dive into open lineage.
It's the open standard design to finally bring some order
to that chaos we were talking about.
We really wanna give you a clear, beginner-friendly way in
to understand how this standard works, why it matters,
and how it changes the game for data integrity.
Yeah, it's foundational stuff.
It really is.
It's kind of the shortcut to understanding
how your data is actually born and transformed.
And we've looked at the core specs, the adoption docs.
The goal is you walk away really understanding
this key piece of the modern data world.
Okay, cool.
Now, before we jump into defining the problem
that Open Lineage actually solves,
just a quick note that this deep dive
is supported by Safe Server.
Ah, good point.
Safe Server focuses on hosting software
and helping out with your digital transformation.
You can find more info and resources
over at www.safeserver.de.
Check them out.
Definitely.
So let's start at the beginning then.
Before Open Lineage, what was the status quo?
I mean, how bad was the pain point that made everyone say,
okay, we need a common standard like right now?
Well, the pain point was pretty simple, actually.
It was duplication, cost, and things breaking all the time.
Every company that needed data lineage,
which is basically everyone, let's be honest,
had to build it themselves from scratch.
So every single project had to instrument,
set up the tracking for all of its jobs individually.
Yeah, so if you had say 10 different data tools,
five orchestration layers,
you were trying to maintain
like 15 separate custom tracking solutions.
That just sounds like a constant grinding drain
on engineering time.
It was, a massive maintenance liability
because those custom tracking things,
they were external to the actual data tools
like Spark or Airflow.
Oh, okay, so not built in.
Not built in.
They relied on specific internal APIs,
often undocumented ones.
So the moment the tool underneath, say Spark,
released a new version, poof,
your custom lineage script just broke.
Nightwear.
Constantly playing catch up, spending thousands,
literally just to keep your basic visibility running,
it was not sustainable.
Okay, so open lineage comes along
and completely flips that script, right?
Instead of every company building
these fragile external scripts,
the effort is shared across the community.
Exactly.
That's the beauty of an open standard.
And the really elegant part.
Now, the integration itself
can be pushed inside each project.
So instead of some external script
trying to like spy on a pipeline,
the pipeline component itself
speaks the lineage language, natively.
So it's embedded.
It's embedded.
The collection is intrinsic.
So you stop worrying about versions
breaking your tracking logic.
It's just there.
It makes so much sense.
And you know, for context,
this isn't some small side project.
Open lineage is an LFAI
and data foundation graduate project.
That means it's recognized, it's battle tested,
it's a proper industry standard.
Got it.
And fundamentally, this data lineage,
it provides the foundation.
It lets you build these powerful context-aware data tools
because it tracks all that metadata
about data sets, jobs, runs.
And at that deeper understanding,
you can pinpoint the root cause
of complex problems much faster.
And crucially, you can understand the impact of changes
before you make them,
before you accidentally break something downstream.
That really highlights the operational win, doesn't it?
Reducing that oops factor.
Exactly.
Fewer oops moments.
Okay, so we know why the standard exists now.
Let's dig into what it actually defines.
If open lineage is creating this shared language
for data tracking, what are the basic words,
the components?
Yeah, good question.
Think of open lineage as defining
a kind of universal data passport.
Okay, I like that, a passport.
Yeah, and this passport dictates the consistent naming,
the structure around three core things,
three core entities that you'll find in any data flow.
Those are the run, the job, and the data set.
Okay, run, job, data set.
So the run, that's like a specific execution
of some process.
Exactly, a single instance of a job running.
And the job is the transformation logic itself,
the code, the definition.
That's right, the definition of the work to be done.
And the data set is, well, the data,
the input or the output asset.
Precisely, the thing being read from or written to,
those three are the absolute foundational building blocks.
Okay, the minimum required pieces.
They are, but the real magic, I think,
the reason the standard can grow and adapt over time
is this concept they call the facet.
Okay, facet.
So the data world is always changing, right?
New regulations pop up, new kinds of tools,
new transformations.
How does a standard like open lineage
avoid becoming obsolete?
Is that where facets come in, like a plugin system?
That's exactly it, you hit it.
Facets are the plugin model.
A facet is defined as an atomic,
sort of self-contained piece of metadata.
Atomic meaning.
Meaning it describes one specific thing,
and you can attach this facet to any of those core entities,
the run, the job, or the data set.
Ah, okay, so if run, job, and data set are the nouns,
facets are like the descriptive adjectives
you can stick onto them.
That's a great way to put it, yes.
And this is absolutely key for things like governance.
How so?
Well, you can attach things like,
say, a regulatory compliance tag,
maybe the GDPR status of a data set.
Or you could attach a detailed schema fingerprint
right onto the data set entity.
Okay.
Or maybe attach quality check results to the run entity,
and because the whole specification is defined
using OpenAPI.
That standard API stuff.
Exactly.
Developers can extend the standard basically endlessly.
They just define their own custom facets
to track whatever proprietary details they need.
This ensures the standard evolves with the industry
and not behind it.
That makes the whole system incredibly flexible.
Yeah.
Kind of future-proof, doesn't it?
That's the goal, yeah.
So we have these core entities, run, job, data set,
and they can be decorated, enriched with these facets.
How does OpenLineage actually capture and send
this information around, especially with all the different
tools people use?
Right.
So OpenLineage also defines a standard API specifically
for capturing these lineage events.
An API call, basically.
Yeah.
So the different pipeline components
think your schedulers like Airflow, your data warehouses,
your analysis tools, SQL engines, whatever.
They use the standard API call.
To report back.
To send data about the runs, the jobs, the data
sets, and any relevant facets.
They package it up according to the standard
and ship this event off to a compatible OpenLineage backend.
Gotcha.
It sounds like the essential plumbing needed
to make all these different tools finally
speak the same lineage language consistently.
That's exactly what it is.
It's the common language.
And I saw the system allows for a configurable backend
so users can choose how those events are sent,
like which protocol.
Yeah, that gives you flexibility in your architecture.
You can choose how you want to receive and process
those events.
Makes sense.
OK, now that we understand the how, let's talk adoption.
Because looking around, this standard
seems to be getting, well, pretty significant traction.
It really is, yeah.
What's the biggest challenge you see folks facing
when they try to implement this?
Is it getting started, doing the initial instrumentation?
Or is it more about handling the sheer volume of metadata
once it's flowing?
That's a good question.
The initial instrumentation effort,
it's actually decreasing pretty rapidly now,
thanks to all the community contributions,
building integrations.
The bigger challenge often is achieving true column level
lineage, especially at scale.
Column level versus table level.
Can you quickly break that down?
Sure.
Table level lineage is basically knowing that, OK, data
move from table A to table B. Useful, but limited.
Column level lineage is knowing that this specific column
in table A was used to calculate that specific column in table
B. It's much more granular.
It's like knowing not just that the package arrived,
but exactly which truck carried the crucial piece of equipment
inside that package.
Ah, OK.
So that lets you trace a specific calculation error
or maybe a compliance issue with PII right back to the source
column.
Exactly.
It's essential for that deep analysis and root cause
finding.
And we see that some heavy hitters, like Apache Spark
and DBT, they're all in. They support both table level
and that more granular column level lineage.
Yeah, that strong adoption by major tools
is absolutely critical for the standard success.
However, there's always a nuance, right?
Listeners should know that column level lineage, while super valuable,
is also inherently complex to capture perfectly in all situations.
So you will find some, let's say, edge cases or specific ways tools
like Spark or Airflow handle certain complex SQL queries
or maybe specific connectors.
Well, like the note says, for Spark, sometimes tracking lineage
through select queries that hit a JDBC source can be tricky.
Or for Airflow, column level might work great for most SQL operators,
but maybe not for a very specific BigQuery operator
doing something complex.
Got it.
So the standard provides the map, but sometimes there
are tricky intersections depending on the specific tool.
That's a good way to put it.
The standard provides the blueprint, yeah.
But the devil can be in the integration details for each tool.
The good news, though, is the community
is super active in identifying and resolving
these tool-specific exceptions.
It's constantly improving.
That's great to hear.
It's really impressive how widely embraced it is becoming.
But open lineage isn't the only name people hear in this space.
How does it fit into the wider data ecosystem?
Are there other key projects that work alongside it
or maybe depend on it?
Yeah, definitely.
It's helpful to think of the ecosystem here.
Open lineage, as we said, defines the standard format,
the language, the electrical plug, if you like,
for data lineage.
OK, the standard plug.
It guarantees the format and structure
of the metadata signal.
Now, the question is, what do you plug
in to that standard outlet?
Right, so tell us about Marquez.
That name comes up a lot.
Marquez is essentially the reference implementation
of the open lineage API.
Think of it as the back end service and the UI
you plug into the wall.
OK.
It focuses on collecting all those open lineage events,
aggregating them, storing the history,
and then visualizing the metadata.
It gives you that dashboard view of your lineage.
So open lineage provides the raw data feed.
Marquez helps you actually see it and explore the history.
Exactly.
Open lineage is the language.
Marquez helps you read the story told in that language.
Got it.
And then there's another project, Egeria.
Where does that fit in?
Is it similar to Marquez?
Egeria is a bit different.
Think bigger picture.
If Marquez is the visualizer plugged into the open lineage
outlet, Egeria is more like the central switchboard
for your entire enterprise's metadata.
The switchboard.
OK.
It offers open metadata and governance capabilities
across the whole organization.
It's designed to automatically capture, manage,
and importantly, exchange metadata
between lots of different tools and platforms,
regardless of the vendor.
So it connects things.
Yeah.
So open lineage collects the raw standardized lineage data.
Marquez can visualize that specific data.
Egeria can take that traceable lineage data
from open lineage and other metadata sources
and share it intelligently across your entire governance
system, maybe feeding it to risk management tools or data
catalogs or data science platforms.
It helps integrate lineage into broader processes.
OK, that makes sense.
Open lineage for the standard feed,
Marquez for visualization and history,
Egeria for broader enterprise metadata management
and governance integration.
You got it.
They complement each other nicely.
And it's clearly a vibrant ecosystem
developing around this.
I mean, looking at the community stats for open lineage itself,
it's not just a theoretical paper, right?
People are actively using this.
2.1 thousand stars on GitHub, nearly 400 forks.
That's real activity.
That activity is crucial, absolutely.
It shows it's solving real problems.
And look at the primary languages being used
for the core project, Java at over 60%, Python around 25%.
That mix perfectly mirrors the modern data stack, doesn't it?
You often have execution engines like Spark running on the JVM,
Java, and then orchestration layers like Airflow
heavily using Python.
So that language mix ensures it fits naturally
into the places where lineage actually
needs to be generated, broad applicability.
Fantastic.
OK, so to kind of bring this all back home,
the core takeaway here seems to be that open lineage is really
this essential open standard.
It's transforming data tracking from this fragmented, custom
built mess.
Yeah, the old way.
Into a consistent, shared framework
for collecting lineage.
It's letting all the different systems in your stack
finally communicate lineage effectively,
and hopefully killing off those maintenance
nightmares of the past.
That's the promise, and increasingly the reality.
And if we connect this to the bigger picture for a second.
Please do.
What open lineage really means is that data observability,
it's no longer this specialized, separate tool
that you have to somehow bolt onto the outside
of your systems.
It's becoming a native function, an inherent capability that's
getting embedded directly into every key piece of your data
infrastructure, from Spark to Airflow to DBT.
It's built in.
That feels like a fundamental shift.
It is, and it raises a really important question
for the future, I think.
When every data movement, every transformation
is inherently traceable, because the tools speak
open lineage natively, how will that standardized,
built-in lineage fundamentally change things like compliance
or automated data governance?
What becomes possible then?
That is a compelling thought.
What happens when observability is just part of the fabric?
Something to definitely chew on as you're
planning your next data modernization project.
Well, thank you so much for walking us
through all that.
Really helpful.
My pleasure.
Thanks for having me.
And finally, a big thank you once again to SafeServer
for supporting this deep dive into open lineage.
SafeServer supports your hosting needs
and digital transformation.
You can find more info at www.safeserver.de.
We'll catch you on the next deep dive.