Today's Deep-Dive: Kestra

Welcome back to the deep dive.

This is your shortcut, really, to understanding complex topics

without getting completely lost in all the details.

Now, before we jump into today's subject, which

I think is fascinating, we want to give a huge shout out

to Safe Server.

They actually take care of the hosting for the software

we're talking about today.

They provide really robust infrastructure and great support

for digital transformation.

You can find out more about what they do at www.safeserver.de.

All right, let's dive right in.

So we live in this hybrid-connected world, right?

And we're just bombarded, I mean,

constantly with this massive amount of information,

trying to stay genuinely well-informed about the latest

tech or how industries are changing.

Well, without getting totally overwhelmed,

it feels like a full-time job.

And that's exactly why we do the deep dive

to cut through all that noise for you.

We pull out the most important bits, the key insights,

so you can grasp these tricky subjects quickly and clearly.

Today, we're zoning in on Kistra.

It's an open source, workful automation platform.

Our mission today, we're going to unpack Kistra,

break it down so that really anyone,

whether you're a seasoned developer

or maybe just starting to explore automation,

can get what it is, why it's actually so powerful,

and crucially, how it's making complex operations

much simpler for teams all over the world.

Yeah, and what's really interesting about Kistra, I think,

is how it fundamentally rethinks the whole orchestration

and scheduling of code.

It's really designed to bring some order to that chaos

we often see, streamlining automation

across loads of different systems.

And as we'll get into, its declarative approach,

combined with this, well,

remarkably intuitive visual interface,

it's really key to making this powerful tech

genuinely accessible, much more so than

many traditional tools.

Okay, let's unpack that right from the start.

Workflow automation, orchestration,

I mean, these terms can sound a bit jargony, can't they?

But let's think about a common pain point,

that feeling, you know, when a critical report

depends on like five different systems,

or a key business process means someone has to

manually kick off scripts, check logs,

and just sort of hope nothing breaks.

We've all been there, right?

Absolutely.

So what is this workflow automation exactly?

And why can't we just string things together manually

or use simpler tools?

What's the chaos that needs something like Kestra?

That's a really good way to put it,

that chaos you mentioned.

That's precisely what Kestra was built to tame.

See, historically, when you needed to automate

a sequence of tasks, let's say,

pulling data from a database,

doing some transformation,

then loading it somewhere else, like an analytics tool,

we'd often end up stitching together custom scripts,

maybe using cron jobs for scheduling,

or perhaps older, more complex tools like Apache Airflow.

Or even simpler, sometimes quite limited services

like Zapier or NNN for lighter tasks.

The problem is, as soon as things get even slightly complex,

multiple dependencies, proper error handling, retries,

different systems need to talk, version control

may be needing real-time responses.

Those approaches, they become brittle, fast,

they're hard to monitor, a nightmare to scale reliably.

Kestrel comes in as an event-driven

declarative orchestration platform

to basically solve that headache.

Its core purpose is making both scheduled workflows,

like that daily report you mentioned,

and event-driven workflows, like reacting instantly

when a new customer signs up,

making those easy and, really importantly, resilient.

It sort of abstracts away a lot of the tricky underlying stuff,

letting you define what needs to happen,

rather than having to manually script

how to handle every single failure or dependency.

Okay, that makes a lot of sense,

bringing order to that potential spaghetti of tasks.

And here's where I think it gets really interesting,

especially for someone who maybe doesn't live

and breathe code day in, day out.

Kestrel has this core idea.

Everything as code and from the UI.

Now, that sounds almost contradictory.

Like, on one hand, you've got the rigor of code

for developers, but then you're saying non-coders

can use the UI.

How does that actually bridge the gap?

Is there a tension there or is it seamless?

It's actually a really brilliant fusion

and it pretty much eliminates that tension you might expect.

So when we talk about a declarative YAML interface,

think of it like writing a simple recipe.

You're describing what you want the final dish to be,

not giving super detailed step-by-step instructions

for every single chop and stir.

You say, I want to extract data here,

transform it like this, then load it there.

Kestra handles the how,

including the tricky bits like error handling, retries,

managing the state of the workflow.

In this declarative way, it's not just simpler,

it's actually way more robust.

Your workflow is defined precisely

in a format humans can read,

which makes it easier to version control, test,

ensure it runs the same way everywhere.

It's like a blueprint.

You write these workflows using Kestra's built-in code editor

right there for the YAML.

But, and this is the magic bit that bridges the gap,

you can also build and visualize the exact same workflow

directly from the UI.

It's got drag and drop,

it has auto-completion to help you,

real-time checks to see if it makes sense.

And the truly seamless part,

when you drag a new task in the UI

or change a setting visually,

the underlying YAML code adjusts automatically, instantly.

So the core logic, the orchestration,

it's always managed declaratively in code,

even if you built it all by clicking and dragging.

This means, say, a data engineer could tweak some YAML,

and then maybe a business analyst

could visually check out, the flow makes sense.

It makes collaboration way easier.

Wow, okay.

And, yeah, to keep things proper,

it even has built-in Git integration.

So your UI built workflows,

still versioned, still managed,

like any other piece of code.

That really is a powerful combination.

It sounds like it makes creating things much easier,

but you still get all the benefits of managing it as code.

So, okay, if I'm someone looking

to start automating something, what are the basic terms?

The building blocks I need to get my head around in Kestra

to actually start building a workflow.

Give us the basics.

Absolutely, let's break down the main pieces

you'll work with, the fundamental thing,

the whole process of the recipe you wanna run,

that's called a flow.

A flow, got it.

Think of it as the complete automated sequence,

an entire data pipeline, maybe,

or a multi-step approval process.

Now, within that flow, each individual action,

each step, like run this Python script,

move that file, send an email, call an API, that's a task.

Okay, tasks are the steps in the flow.

Exactly, they're the specific ingredients

and instructions in your recipe.

Then you have triggers, they decide how your flow starts.

You've got scheduled triggers, pretty simple,

like setting an alarm clock, run this flow every hour

or every Tuesday at 3 p.m., whatever you need.

And crucially, Keystra is really strong

with event-driven triggers.

This lets your flow react instantly to something happening.

Maybe a new file appears in cloud storage

or a message pops up in a Kafka topic

or even a specific API call comes in.

Reactive, okay.

Yeah.

And for keeping things organized,

especially as you build more flows, you use namespaces.

They're basically like folders,

just ways to group related flows together,

keep things tidy.

And inputs and variables add flexibility.

They let you customize flows and tasks on the fly

so you can reuse the same flow

for slightly different situations

without copying everything.

Makes sense.

And finally, to see all this,

that intuitive UI we mentioned

gives you a live topology view.

Technically, it's a directed acyclic graph.

A DAG.

A DAG.

Think of it like a flow chart

or a roadmap for your workflow.

Each task is a box.

Arrows show how data moves and what depends on what.

It's super important because it ensures tasks

run on the right order,

avoids messy loops,

and lets you see dependencies at a glance.

Makes even really complex stuff

much easier to understand visually.

That DAG view sounds incredibly useful,

especially when things get complicated.

Now you mentioned one of Kestra's standout features

is its rich plugin ecosystem.

That sounds powerful.

When you say plugin ecosystem,

what kind of things can these plugins actually do?

How comprehensive is it?

Can it really connect to almost anything?

Oh, it really is comprehensive.

This is where Kestra's flexibility just shines.

It basically future-proofs your automation.

The plugin system means you can, first off,

run scripts in pretty much any language you like.

Python, Node.js, R, Go, Shell, Java, even just SQL,

whatever your team knows or what's best for the job.

And you can run them anywhere.

On the machine, Kestra is running on, sure,

but also on remote servers using SSH,

or scale out using containers with built-in Docker

and Kubernetes support.

Super versatile.

But the real power is connecting to other systems.

For instance, plugins let you extract and load data

from a huge range of sources.

We're talking traditional databases, Postgresql, MySlick,

Oracle NoSQL ones like MongoDB or Redis,

cloud storage like AWS S3, Google Cloud Storage,

Azure Blob Storage, countless APIs.

Wow.

So you're pulling data from some old legacy system

and mixing it with data from a modern cloud service.

Kestra handles that.

You can also set up flows to react to real-time events

from message systems like Kafka, RabbitMQ, Redis Streams,

or the big cloud ones, AWS SQS, Google PubSub,

Azure Event Hubs.

Imagine kicking off a fraud check instantly

when a new transaction message appears.

Yeah, powerful.

And it deeply integrates with the major clouds,

AWS, Google Cloud, Azure for everything,

managing servers, triggering serverless functions,

orchestrating data warehouses like Snowflake or Databricks,

and of course, keeping people informed.

There are tons of plugins for notifications,

sending messages to Slack, Teams, email,

even triggering alerts in PagerDuty

so the right people know if something needs attention.

Currently, there are over 600-plus plugins available,

ready to go.

Plus, users can build their own

or contribute new ones back to the community.

So yeah, the goal is really to connect

to virtually anything you're already using.

You're not locked in the one way of doing things.

That level of connection,

tying everything together like that,

it sounds incredibly valuable.

But okay, beyond just the tech specs,

what does this all actually mean for the teams,

the people using Kestra day-to-day?

What's the real impact they're seeing?

And who really benefits most?

Yeah, it translates into some really significant,

tangible benefits for a surprisingly wide range of people.

Kestra positions itself as being for every developer.

And that seems true.

Data engineers building pipelines,

software engineers orchestrating services or CICD,

even marketing ops folks automating campaigns,

platform engineers automating infrastructure.

It really does help unify teams,

breaks down those silos you sometimes see.

What really scans out in feedback is how often users

praise its simplicity and user experience.

You hear things like beginner-friendly, so easy to use,

fantastic web UI, extremely approachable and clear.

People often say it's much simpler to get going with

compared to alternatives like, say, Airflow or Prefect.

Right.

And that simplicity, it's not just nice to have,

it directly leads to faster time to value.

We see reports from companies saying they deliver in days

what once took weeks,

achieving things like 5x deployment speed.

That points to less time fighting the tool,

more time building the actual solution.

Definitely a plus.

And you see big jumps in productivity and efficiency.

Things like 10x faster workflow completion,

an 80% boost in team efficiency,

even a 75% reduction in operational cost

by automating stuff that was manual before.

It's also built for scalability.

It's proven to handle millions of workflows

in production environments

with over 400 million plus workflows executed

across its user base so it can handle serious load.

Plus, that flexibility and no vendor lock-in is key.

Write your logic in any language,

deploy Kestra wherever makes sense for you,

on-prem, hybrid cloud, any major cloud provider you choose.

That's important. Yeah, and to make starting even easier,

there are over 250 plus blueprints.

These are like pre-built templates for common tasks,

pulling data from Salesforce to Snowflake, for example.

There's likely a blueprint that gets you

like 90% of the way there straight away.

Okay, those blueprints sound like a great starting point.

Well, that paints a really comprehensive picture

from taming that initial chaos of complex processes

to enabling really powerful, flexible,

but also it seems surprisingly easy to manage automation.

Kestra really does seem to cover a lot of ground.

That unique mix of everything is code and from the UI,

the huge plugin ecosystem,

and just that focus on making orchestration accessible,

those are clearly its big strengths.

Sounds like a potential game changer

for teams maybe struggling with their current tools.

Absolutely, and this kind of leads to an important question

maybe for you, our listener, to think about.

What repetitive task, what complex data flow,

or maybe what integration challenge

exists in your world right now

that could potentially be simplified,

made more reliable, using a tool like Kestra.

Is there something you do manually every week?

Data stuck in silos that needs connecting.

If you are curious and you want to see it in action,

you can actually try their live demo.

There's no installation needed,

you just play with it in your browser.

Or if you want to get hands on,

check out the quick start installation guide.

It literally takes about five minutes

to get it running locally with Docker.

You'll have the Kestra UI up at http.localhost.8080.

And definitely check out their community on Slack and YouTube.

Lots of help, lots of examples there too.

Great resources.

And remember, this deep dive was made possible

thanks to the support of Safe Server.

They're your partner for robust hosting

and really seamless digital transformation solutions.

Do visit www.safeserver.de to learn more about them.

Thank you so much for joining us

We look forward to exploring another fascinating topic

We look forward to exploring another fascinating topic