Today's Deep-Dive: Apache Druid

Welcome back to the deep dive. This is where we take a piece of complex tech

We peel back all that jargon and we just give you the essentials and today we are

plunging right into the core of big data

Focusing on something that's just critical for handling these massive volumes of

data that are arriving, you know right now

We're talking about Apache druids

So if you're out there dealing with just mountains of streaming data or maybe you're

trying to power a dashboard that has to answer questions

Instantly, I mean not in minutes but in milliseconds then this deep dive is for you.

Our mission is pretty simple

we're going to break down Apache drew this powerful real-time analytics database

and just give you a really clear entry point a way to

Quickly grasp what it is how it works and why it's becoming so important in the big

data world

We've been digging through the core descriptions the feature lists all the

community docs and we're ready to well distill that for you

But first I just want to mention that this deep dive is supported by safe server

safe server handles the hosting for this type

Of software and supports you in your digital transformation. You can find out more

at www.safeserver.de

That's www.safeserver.de. Right. So let's just start with the fundamental

definition because Apache drew it isn't a general-purpose tool

It's very very specific. It's a high-performance

real-time analytics database and its design goal is incredibly focused

It's all about delivering sub-second queries on just immense amounts of data sub-second

Okay. Yeah, and that's on both streaming data that just arrived and you know

Pedabytes of historical data all at scale and under huge load. Think of it this way

It's an open-source alternative to your traditional data warehouse

But it's been optimized specifically for that high-speed interactive analysis

Oh LAP right online analytical processing. Exactly. It's built to answer those

complex analytical questions instantly

Okay, let's unpack that OLAP thing immediately because I mean there are countless

databases out there

You've got relational no sequel data warehouses. Why do we need a different kind of

database like druid just for analytics?

What's the specific problem here that those other tools can't solve?

The problem is what you could call the time to insight or you know time to action

Traditional systems, especially the ones built for transactions OLTP. They're great

for say recording a single sale

Right one record at a time exactly and even traditional data warehouses, which are

good for big complex reports

They're often optimized for batch loading. That means query times can be slow and

you're asking ad hoc real-time questions

Druid is designed for that exact moment when you need a live answer

Right now so for instance you run a gaming platform and you need to know the total

spend of users in Berlin in the last minute

Filtered by their device type Wow. Okay, that's specific it is and it requires

instantaneous analysis across

Maybe billions of events druid steps in to make those ultra-fast highly dimensional

workflows possible

So give me the wow metric here if I'm used to my reports taking I don't know 30

seconds to run

What kind of speed are we really talking about with druid? We are talking about

true millisecond response times millisecond the documentation

It consistently highlights its ability to run these really complex high-dimensional

OLP queries in

Milliseconds and that's even against datasets that range from billions to get this

trillions of rows trillions

Okay, and critically it does this without you having to pre-aggregate the data or

pre-define the exact query

You're gonna ask you can just ask anything instantly that speed is impressive

But that usually implies a single highly optimized query what happens when?

Say a thousand users hit that same data all at once

Does the performance just tank and that's where its concurrency power is?

So essential speed is pretty meaningless if the system just buckles under load

Druids architecture as well. It's fundamentally built to handle that kind of

immense concurrent load

We're talking about supporting anywhere from hundreds to even a hundred thousands

of queries per second

QPS a hundred thousand queries per second maintaining that consistent low latency

performance

This is precisely why it's used to power consumer facing ui's or internal dashboards

that you know

Hundreds of employees are using all day every click is a new query every single

click on a filter a

Visualization that requires a fresh lightning fast query

Druid is built for that exact operational load

Okay, but achieving that kind of performance usually means you have to throw

massive expensive hardware at the problem

Does druid manage to be cost-efficient? It absolutely does and it addresses that

cost question mainly through resource efficiency

Because the architecture is so optimized and we'll get into the compression and

indexing in a bit. It just

Significantly reduces the total hardware footprint you need compared to other

solutions. Yeah compared to many competing data warehouses

it's really designed for massive scale, but with

Resource conservation built right into its DNA. It makes the total cost of

ownership much much lower over time

Okay, so this is where it gets really interesting for me the real-time aspect

We're in a streaming world now data from logs sensors clicks. It never stops

So how does druid deal with data that showed up one second ago? This is its core

differentiator

True stream ingestion druid is you could say stream native if you've ever worked

with traditional ETL processes

You know, there's always a delay the data lands. It gets transformed then it's

loaded that can take minutes

Sometimes hours the dreaded batch window right druid just eliminates that wait time

It has native baked in integration with the big streaming platforms like Apache

Kafka

Amazon kinesis this allows it to support query on arrival it ingests and

Immediately makes millions of events per second available for query immediately

with guaranteed consistency and ultra low latency

So if a critical event happens like a security alert or a sudden drop in sales

I can query that event and act on it within a second not wait for some nightly

batch process

That is exactly the use case

You are querying the live stream at the same time as all the historical data

You can instantly combine the hot fresh data

What happened the last minute with a massive cold data from the last five years all

in a single unified?

Subsecond query and that gives you the complete picture a truly complete up to the

second operational picture

It's essential for things like monitoring and alerting apps. Okay, we've

established. It's ridiculously fast. It's scalable

It's stream made of now. I want to look under the hood. How how does it sustain?

millisecond speed on

trillions of rows

What's the secret sauce in the architecture?

Well, the speeds no accident. It starts the very moment the data is ingested the

second data touches druid

It's automatically transformed into what they call an optimized data format

It's a multi-layered process, but we can simplify it down to say three key

principles

Okay, first the data is columnarized. This is a huge departure from traditional row

based databases

Imagine your data is a physical library a row based database stores every books

info author title date subject all

Bundled together on the shelf. That's a row

So if I just want to know all the subjects I have to pull every single book record

off the shelf precisely

You have to read through everything it takes time at waste operation

Columnar storage is like taking that library and putting all the authors in one

aisle all the titles in another and all the subjects in

A third so if you only need to run a query on the city column

You only read the city aisle you skip all the other massive amounts of unrelated

data

It just dramatically cuts down the IO. Okay, so columnar storage is step one for

efficiency

What's next step two is that the data is profoundly optimized for time series

analysis?

Specifically through time indexing since most data in druid is time series data

events happening over time

It immediately indexes everything based on time. It's like installing a master

clock in your data

So you can jump to a specific time range instantly instantly the last week the

third quarter of 2021

Whatever without scanning anything irrelevant and the third principle is all about

compression and encoding

This is the digital shorthand the data gets highly optimized using techniques like

dictionary encoding and bitmap indexing

Okay, that sounds like serious jargon. Can you make that a bit more accessible?

What's dictionary encoding actually doing for me?

Of course think of dictionary encoding is creating a little lookup table for any

repetitive values

So if you have a column with a million rows that only contain five city names

New York London Paris, Tokyo Sydney druid

Doesn't store the full-text New York a million times. That would be crazy

It would instead it assigns it a tiny numerical code like one it stores that

dictionary mapping just once and then stores

Millions of tiny ones in the actual data segment

Ah, so you save a massive amount of storage space and because the index is so much

smaller the query engine can search it way faster

It's looking for tiny numbers not long strings. Exactly. It's an ultra-efficient

data reduction that directly boosts speed

The result of all this is that every piece of data is basically a pre-optimized

ready-to-run package from the second

It's ingested. Okay, so that's the data structure

How does the query engine actually use that to get to sub second speeds?

So the engine uses what's called a scatter gatherer approach

Yeah, and the core philosophy is don't move the data don't want the data data

movement is slow. It's expensive

So druid make sure those optimized data segments are loaded directly into memory or

local

SSDs on the specific nodes where they live when you run a query the system

Intelligently figures out which nodes hold the data you need it scatters the work

to those nodes

They process it locally using their own CPU and memory and then they only gather

the small result sets back

That dramatically cuts down on network latency

It does it avoids reading anything extra and keeps the network traffic to an

absolute minimum

So instead of hauling the entire library to a central desk for processing

You just send little instruction cards to the specific aisles and they send back

only the chapter summaries you asked for that's a perfect analogy

And that localized processing is just crucial for maintaining performance under all

that concurrency

We talked about speaking of concurrency

Applications scale and you know, they often scale unpredictably if my user base triples

tomorrow

Can I scale the system quickly and reliably that brings us to its elastic

architecture?

Druid was designed from the ground up to be distributed all its components ingestion

queries orchestration deep storage

They're all loosely coupled and why does loose coupling matter?

It means you can scale the query processing nodes completely independently from the

ingestion nodes or from the storage layer

So if you suddenly need to handle ten times the query traffic

You could just provision and add ten times the query nodes in minutes

You never have to interrupt the system or stop data ingestion. It's incredibly

flexible scaling both vertically and horizontally

Yes, and what about when things go wrong? If a node goes down does my whole

dashboard crash? Absolutely not

Reliability is a cornerstone druid has non-stop reliability features built in

things like continuous backup to deep storage like s3

Or HDFS automated recovery and multi node replication of data segments

So there's always a copy always if a query node fails

Another node that already holds a copy of that data just automatically steps in it

ensures high availability and durability

This is all very powerful

But for the user who needs to adopt this the learning curve can be steep

How accessible is druid for developers or analysts who are used to more traditional

tools?

Accessibility has been a huge focus in its recent development

I mean for one developers and analysts can use the familiar SQL API for everything

for everything not just querying for all end-to-end

Data operations you can use standard SQL for querying sure

But also for defining ingestion tasks and performing data transformations

If you know sequel you have a massive head start that lowers the barrier to entry

significantly

You don't have to learn some proprietary query language

What about defining the schema the structure of the incoming data is that a rigid

manual process?

That's another big accessibility when schema auto discovery gives you the ease of a

schemal a system

but with the performance benefits of a strict schema as

Data streams in druid can automatically detect define and update column names and

their data types

You don't have to stop everything and manually define a hundred columns before you

can core your data

It just handles that on the fly. So if I were a new user trying to load my first

data stream

What's that actual user experience like am I writing yaml scripts from day one? Not

at all

There's a really practical built-in web console. It's designed for easy interaction

Through that console you get a point-and-click wizard for setting up ingestion

whether you're loading a huge historical batch file or

Configuring a continuous stream from Kafka. So it guides you through it

It does and you can also manage the entire cluster from that console viewing your

data sources monitoring tasks

Checking the health of all your services. So management and prototyping are all in

one place

Exactly, and the console also has a query workbench. You can prototype and refine

your druid SQL queries or native queries

Interactively, it's the perfect little sandbox to see how your data will perform

before you push those queries into your actual application

This has been a fascinating deep dive

so to summarize the key takeaways for you listening Apache druid delivers high

performance real-time analytics by combining two key things first an

Optimized data storage. It's always columnar. It's time indexed and it's highly

compressed

Second it has a stream native elastic architecture that enables that scatter gather

query approach and the result of that combination is just

Massive concurrency and sub second query performance on enormous data sets

and if we connect this back to the bigger picture the whole architecture of druid

prioritizing query on arrival and

millisecond response times it really raises an important question in a world is

demanding instantaneous data high concurrency

Dashboards, how much longer will traditional database architectures the ones that

rely on lengthy ETL processes or pre caching?

How much longer will they remain competitive for operational apps and dynamic ui's

so the very definition of fast enough is changing

It's changing very rapidly a truly provocative thought to mull over if you want to

explore this any further

You can check out the quick start the FAQ and all the documentation through the

Apache druid project

We really encourage you to dive deeper. Thank you for joining us for this deep dive

and once again a huge

Thanks to our sponsor safe server

Remember safe server cares for the hosting of this type of software and supports

you in your digital transformation

www.safeserver.de

www.safeserver.de