Welcome back to the Deep Dive. If you need to get up speed quickly on complex
research,
technical stuff, or what's moving in an industry, you're definitely in the right
place.
We go deep into the source material so you get the essential informed perspective.
Now, before we unpack today's sources, we really want to give a huge thank you to
our supporter,
SafeServer. SafeServer handles the hosting for exactly this kind of, you know,
crucial distributed
software. They're dedicated to supporting you on your digital transformation
journey.
You can find out more info and resources over at www.SafeServer.de.
So today, we're focusing on something that sounds pretty technical, but it's
actually
quite democratizing in the end. Distributed object storage. We're taking a deep
dive into
a specific open source project called Garage. Yeah, Garage. It basically aims to
take the kind
of resilience you see with, like, the massive cloud giants. Like Amazon, Google.
Exactly.
Yeah. And put that power directly into the hands of, you know, self-hosted users,
small businesses,
that kind of thing. Okay. So our sources really lay out a fascinating path for how
you can get
world-class data redundancy, but without needing a world-class budget. Right. The
challenge we're
sort of tackling for you, the listener, is this. How do you get data storage that's,
well, practically
indestructible, even if the servers you're running it on are a bit flaky or old or
even spread out
across totally different physical locations? That's the mission, then. To demystify
this
geo-distributed storage idea, look at the pretty cutting-edge theory Garage is
built on. And show
how this specific solution is really tailored for the, let's say, small to medium
operator,
not the huge enterprises. Got it. So the key takeaway right from the start. The
core nugget
you need is that Garage is open source. It's S3 compatible. We'll get into why that
matters.
And it's an object store specifically designed for geo-distributed setups. Right.
It's fundamentally
about bringing that sort of enterprise level availability down to environments
where, frankly,
failure is kind of expected sometimes, not this rare anomaly. Okay, let's start
with the basics
then. Because object storage itself, it's a term people hear but maybe don't fully
get compared to
you know, the normal file storage on their computer, or maybe block storage they've
heard about.
Yeah, good place to start. If I'm just saving a document or maybe doing a backup,
why would I even think about object storage? Exactly. Well, it really comes down to
scale
and flexibility. Think of your traditional file storage like a very rigid filing
cabinet. You've
got folders inside folders, a strict hierarchy. Okay. Block storage is even lower
level, dealing
with the actual chunks on a hard drive. Yeah. Object storage though, it's totally
different.
It's flat. A flat. Yeah, every piece of data could be a video file, a log entry, a
database backup,
whatever. It's treated as its own distinct self-contained object. Ah, okay. And
each object
gets tagged with rich metadata information about the object, and it gets a unique
ID. This flat
structure makes it almost infinitely scalable. Right. Which is why it's the main
method used by
companies like, say, Netflix for streaming video or Amazon S3 itself for just
massive archives of
data. And Garage ties into that directly. You said its identity is a distributed
object storage
service that emphasizes S3 compatibility. Why is being S3 compatible so critical?
Oh, it's huge
because Amazon's simple storage service, S3, is basically the universal language,
the lingua franca
of cloud storage. Right. Everyone uses it or integrates with it. Exactly. Yeah. So
by
implementing the exact same API, the way software talks to the storage garage just
wipes out the
biggest hurdle to using it. How so? Well, if you're already using some software for
say backups or
monitoring or maybe hosting website assets, and that software talks to AWS S3.
Which a lot of
software does. Precisely. You can literally just point that same software at your
own garage cluster
instead. No code changes needed. It just works. Wow. Okay. That makes it incredibly
attractive
then, especially for sysadmins who don't want to rebuild their entire tool chain.
Absolutely. Now,
this wasn't built by some corporate giant you mentioned who's actually behind
garage. And who
are they building it for? Yeah, it came from a group called Doofsler. They're
actually an experimental
small-scale self-hosted service provider themselves. So they built it out of their
own need.
Pretty much. They built garage because they needed it and they've been using it in
production since
back in 2020. Yeah. So it was really built by people running these kinds of small
distributed
setups for people doing the same. Makes sense. And importantly, it's entirely free
software.
It's released under the AGPL v3 license. Hold on. AGPL v3. That's a specific choice,
isn't it?
What does that license mean for someone thinking about using garage or maybe
building on it?
Yeah. It's an important detail. The Afro general public license AGPL is one of the
stronger copy
left licenses. Basically, if you modify the garage software and then run that
modified version as a
network service, like people connect to it over the internet, then you must make
the source code
of your modified version available to those users. Ah, so it prevents someone from
taking the open
source code, improving it, and then locking those improvements up in their own
private cloud service.
Exactly. It's designed to make sure that improvements made to free software,
especially when it's run as a service, flow back to the community. It keeps the
technology free and
open. That fits perfectly with this whole mission of democratizing resilient
infrastructure.
Okay, let's move on to the driving goals. Section two, I mean, why bother with all
this complexity?
If I can just plug a huge external hard drive into my server, why build distributed
storage across
multiple locations? You basically hit the nail on the head there. The core
motivations are resilience
and geographical distribution. Right. That single giant hard drive that's a single
point of failure,
power supply dies, fire, flood, whatever, your data's gone. Okay, yeah. Garage tackles
this head-on.
It's explicitly designed for clusters where the nodes, the servers are running in
different
physical locations. Could be different racks, different buildings, different cities
even.
So the idea is if one data center goes dark, maybe a power outage,
a network cut, the data is still safe and accessible somewhere else.
Precisely. High availability through geographic redundancy.
But wait, moving data across the public internet, that sounds like a recipe for
terrible latency consistency problems, doesn't it? Is dealing with that really
worth the
complexity for a small operator? That's the critical question, right?
Yeah. And it's actually what defines
Garage's whole engineering philosophy. Most big enterprise storage software assumes
you
have these perfect low latency, high bandwidth, dedicated network links between
data centers.
Which small operators usually don't have. Exactly. Garage's developers,
being sysadmins themselves running stuff over the regular internet,
just accepted that high latency, maybe up to 200 milliseconds even,
and flaky connections were just reality. So they built around that reality.
Yes. They focused the design on handling those network issues gracefully.
The goal isn't just copying data. It's keeping the service available and ensuring
the data
becomes eventually consistent, even if entire locations are temporarily offline.
So they democratized resilience by not requiring perfect, expensive networking.
That's it. They built the software to deal with the messy reality of non-enterprise
networks.
Making it actually usable. Right. They aimed to make it
operationally manageable, lightweight, prioritizing that availability, that
survival,
above almost everything else. Okay. This is where it gets really
interesting for me. Section three, the how. How does Garage actually achieve this
high resilience?
How does it handle data integrity when you've got machines failing, disks dying,
networks dropping packets, the whole potential mess?
Yeah. The secret sauce. They rely on, well, structured redundancy and concepts like
Quorum.
Garage is built to be highly resilient against pretty much everything.
Disk errors, yes, but also whole network partitions separating geographic locations.
And they mentioned something interesting. Sysadmin failures.
Ah, yeah. They specifically considered that. Because let's be honest, sometimes the
biggest
cause of an outage is human error during maintenance or configuration. The design
tries to minimize the impact of those two. Smart. Okay. Tell us about the data
replication.
How many copies of my data exist and where do they live in this system?
The magic number they landed on is three. Every chunk of data you upload gets
replicated,
stored in three distinct zones. Zones. What's a zone?
Think of a zone as a logical grouping that usually maps to a physical failure
domain.
So it can be a specific rack, a specific building, or most powerfully a specific
data center or geographic location. And importantly, a zone itself usually
consists of multiple servers for redundancy within the zone.
So three copies spread across three potentially separate geographical areas.
Correct. And the logic is classic quorum systems. To survive N1 failures, you need
N copies. Here,
N3. So you can lose up to two entire zones simultaneously.
Two whole data centers could go offline.
Could be, yeah. One is down for planned maintenance and another has a sudden
network
failure. Your data remains fully accessible and readable from that third surviving
zone.
Wow. That level of fault tolerance with potentially just three locations, that's
pretty remarkable for something aimed at smaller scale.
It really is. But achieving that reliability isn't trivial, theoretically speaking.
The sources mention garages standing on the shoulders of giants drawing from
decades of distributed systems research. Right. Let's unpack those giants. They
mention influences like Amazon's Dynamo. What key ideas did Garage borrow from
massive systems like that? This is really the core of it, I think.
The Garage team didn't necessarily invent brand new distributed systems math.
Instead, they very cleverly applied existing, proven, but often complex,
academic research to their specific scale and problem set.
Okay, so Dynamo first. What's the key principle there?
Dynamo, Amazon's highly available key value store.
Its core idea was prioritizing availability over perfect, immediate consistency.
Meaning?
Meaning if there's a network issue, like a flow link or a partition, Dynamo won't
just block the
user and say, try again later. It will likely accept a write operation, let the
user think
it succeeded, and then promise to sort out any potential conflicts later once
communication is
restored.
Uh, so it keeps working even when parts of the system can't talk to each other
properly.
Exactly. It favors being available for reads and writes over ensuring every node
has the
absolute latest data right now. This leads to the concept of eventual consistency.
Okay, eventual consistency. But how do you make sure things do eventually become
consistent
without losing data or getting corrupted when servers have seen different things?
Great question. And that brings us to the second major influence, conflict-free
replicated data
types, or CRDTs.
CRDTs? Sounds complicated.
The concept is brilliant, actually. CRDTs are special data structures designed
specifically
for this eventual consistency model. They allow multiple servers to update the same
piece of data
independently, even while disconnected from each other.
Okay.
And they have mathematical properties that guarantee, when those servers do
eventually
reconnect and share their updates, their states will merge together correctly and
automatically
without losing information and without needing slow, complex locking mechanisms to
coordinate.
Whoa. So CRDTs are like the algorithmic magic that makes eventual consistency
safe and reliable, especially over flaky internet links.
It stops data getting messed up when different nodes are out of sync for a while.
You got it. It elegantly solves the coordination headache that plagues
many traditional distributed databases when dealing with network partitions or high
latency.
It's a huge enabler for systems like Garage.
Fascinating. And there was a third influence mentioned, Maglev.
Right. Maglev. That's Google's high-performance software network load balancer.
Mentioning this shows they weren't just thinking about data storage theory,
but also the practicalities of efficiently routing requests and managing
connections within the
cluster. So handling traffic effectively to make sure requests go to the right
place quickly,
even under load. Exactly. Making sure data requests are
steered efficiently to the nearest or the healthiest node that holds the data.
It's really impressive how they've taken these heavyweight architectural ideas born
from giants
like Amazon and Google and managed to translate them into something lightweight
enough for modest
infrastructure. That's the key achievement, I think. Which brings us nicely to the
practicalities.
Section four, the low barrier to entry. If I actually want to set up this geo-distributed,
super resilient cluster, do I need to go out and buy three identical brand new
server racks?
That's probably the most compelling part for the self-hosting community or small
businesses.
The answer is a definite no. An explicit design goal was keeping the barrier to
entry low.
They actively encourage using existing or even older machines. You don't need a
supercomputer cluster, then? Not at all. The minimum requirements are honestly
quite minimal.
Like what? Per node, they suggest just one gigabyte of RAM.
One gig? Seriously? Yep. And at least 16 gigabytes of disk space.
For the CPU, basically any BI8664 processor from the last decade or so, or an ARMv7
or ARMv8 chip,
think Raspberry Pi level or similar is sufficient. That is incredibly low overhead
for a system
promising this kind of resilience. It really suggests, like you said, if you've got
a few
old office PCs lying around or maybe some cheap virtual servers scattered in
different regions.
You could genuinely start building a garage cluster. That's the core economic
appeal.
And crucially, they build it specifically to allow mixing and matching different
types of hardware.
Ah, so heterogeneous hardware is supported. Explicitly. You can combine servers
with
different CPUs, different amounts of RAM, different disk sizes within the same
cluster.
That massively simplifies things because you don't need to source expensive
identical machines. Use
what you have or what you can get cheap. That's huge for operational reality.
And the deployment is super simple, too. It ships as a single self-contained binary
file,
no complex dependencies to install. It just runs on pretty much any modern Linux
distribution.
Just copy the file and run it, basically. Pretty much. It really emphasizes that
focus on ease of operation for the sysadmin. And just circling back quickly to a
tech detail,
the main language used is Rust, right? About 95% of the code?
Correct. And Rust is known for its performance, efficiency, and especially memory
safety.
That choice directly contributes to those low resource requirements and overall
stability.
That technical excellence seems linked to its sustainability, too. You mentioned it's
not
corporate-backed, but projects like this need ongoing work. How is Garage supported?
Has it managed to find funding? Yes. And that's another critical
point. In a world often dominated by venture capital, Garage has actually secured
significant
public funding, which really signals confidence in it as a public good for the
internet.
That's really interesting. Where did this public funding come from?
It's primarily come via the European Commission's Next Generation Internet,
or NGI initiative. It's had several grants. Can you detail those?
Sure. Back in 2021-2022, the NGI Pointer Fund supported three full-time employees
working on
Garage for a whole year. Wow, three people full-time.
Yeah. Then more recently, from 2023 to 2024, the NLNet Foundation through the NGIU
and Trust Fund
supported one full-time employee. Okay.
And it's ongoing. Looking ahead to 2025, the NLNet NGI Year Commons Fund is
providing support
for the equivalent of 1.5 full-time employees. So there's a steady stream of grant
funding,
keeping this critical piece of decentralized infrastructure alive and evolving,
driven by
community needs, not profit. That seems to be the model, yes.
It keeps it open source, keeps it free, and aligned with that original mission.
Okay. So let's try and summarize the core takeaway for you, the listener. What
Garage seems to have
done is successfully bridge this gap, right, between really advanced complex
distributed
systems theory like Dynamo, CRDTs, and the practical, often hardware-constrained
reality
of self-hosters and small organizations. Exactly. It essentially offers smaller
players
the kind of power, geo-distributed, highly resilient storage that used to be pretty
much exclusively the domain of tech giants with massive budgets.
It levels the playing field in a way. It really does. And this whole project,
this whole approach, it raises a bigger, quite provocative question, I think.
When you have open-source projects, especially ones backed by public funds like NGI,
actively working to democratize access to this kind of complex, resilient
infrastructure,
what does that really mean for the future? For decentralized data ownership, for
control,
it kind of challenges the default assumption that only huge centralized
corporations
can truly guarantee the safety and availability of our important data.
As the cloud continues to consolidate around a few big players,
projects like Garage offer a different path. It's definitely something worth
thinking about.
An excellent point to end on. A powerful thought to mull over, indeed.
Hopefully, you feel much better informed now about the potential of geo-distributed
object storage,
and specifically, the Garage project. We want to once again thank our sponsor,
Safe Server, for supporting this deep dive. They support the hosting of this very
type of software,
helping with digital transformation. You can find out more about how they can
support your
infrastructure needs at www.safeserver.de. Thanks for joining us for the deep dive.
We'll catch you