Today's Deep-Dive: Garage

Welcome back to the Deep Dive. If you need to get up speed quickly on complex

research,

technical stuff, or what's moving in an industry, you're definitely in the right

place.

We go deep into the source material so you get the essential informed perspective.

Now, before we unpack today's sources, we really want to give a huge thank you to

our supporter,

SafeServer. SafeServer handles the hosting for exactly this kind of, you know,

crucial distributed

software. They're dedicated to supporting you on your digital transformation

journey.

You can find out more info and resources over at www.SafeServer.de.

So today, we're focusing on something that sounds pretty technical, but it's

actually

quite democratizing in the end. Distributed object storage. We're taking a deep

dive into

a specific open source project called Garage. Yeah, Garage. It basically aims to

take the kind

of resilience you see with, like, the massive cloud giants. Like Amazon, Google.

Exactly.

Yeah. And put that power directly into the hands of, you know, self-hosted users,

small businesses,

that kind of thing. Okay. So our sources really lay out a fascinating path for how

you can get

world-class data redundancy, but without needing a world-class budget. Right. The

challenge we're

sort of tackling for you, the listener, is this. How do you get data storage that's,

well, practically

indestructible, even if the servers you're running it on are a bit flaky or old or

even spread out

across totally different physical locations? That's the mission, then. To demystify

this

geo-distributed storage idea, look at the pretty cutting-edge theory Garage is

built on. And show

how this specific solution is really tailored for the, let's say, small to medium

operator,

not the huge enterprises. Got it. So the key takeaway right from the start. The

core nugget

you need is that Garage is open source. It's S3 compatible. We'll get into why that

matters.

And it's an object store specifically designed for geo-distributed setups. Right.

It's fundamentally

about bringing that sort of enterprise level availability down to environments

where, frankly,

failure is kind of expected sometimes, not this rare anomaly. Okay, let's start

with the basics

then. Because object storage itself, it's a term people hear but maybe don't fully

get compared to

you know, the normal file storage on their computer, or maybe block storage they've

heard about.

Yeah, good place to start. If I'm just saving a document or maybe doing a backup,

why would I even think about object storage? Exactly. Well, it really comes down to

scale

and flexibility. Think of your traditional file storage like a very rigid filing

cabinet. You've

got folders inside folders, a strict hierarchy. Okay. Block storage is even lower

level, dealing

with the actual chunks on a hard drive. Yeah. Object storage though, it's totally

different.

It's flat. A flat. Yeah, every piece of data could be a video file, a log entry, a

database backup,

whatever. It's treated as its own distinct self-contained object. Ah, okay. And

each object

gets tagged with rich metadata information about the object, and it gets a unique

ID. This flat

structure makes it almost infinitely scalable. Right. Which is why it's the main

method used by

companies like, say, Netflix for streaming video or Amazon S3 itself for just

massive archives of

data. And Garage ties into that directly. You said its identity is a distributed

object storage

service that emphasizes S3 compatibility. Why is being S3 compatible so critical?

Oh, it's huge

because Amazon's simple storage service, S3, is basically the universal language,

the lingua franca

of cloud storage. Right. Everyone uses it or integrates with it. Exactly. Yeah. So

by

implementing the exact same API, the way software talks to the storage garage just

wipes out the

biggest hurdle to using it. How so? Well, if you're already using some software for

say backups or

monitoring or maybe hosting website assets, and that software talks to AWS S3.

Which a lot of

software does. Precisely. You can literally just point that same software at your

own garage cluster

instead. No code changes needed. It just works. Wow. Okay. That makes it incredibly

attractive

then, especially for sysadmins who don't want to rebuild their entire tool chain.

Absolutely. Now,

this wasn't built by some corporate giant you mentioned who's actually behind

garage. And who

are they building it for? Yeah, it came from a group called Doofsler. They're

actually an experimental

small-scale self-hosted service provider themselves. So they built it out of their

own need.

Pretty much. They built garage because they needed it and they've been using it in

production since

back in 2020. Yeah. So it was really built by people running these kinds of small

distributed

setups for people doing the same. Makes sense. And importantly, it's entirely free

software.

It's released under the AGPL v3 license. Hold on. AGPL v3. That's a specific choice,

isn't it?

What does that license mean for someone thinking about using garage or maybe

building on it?

Yeah. It's an important detail. The Afro general public license AGPL is one of the

stronger copy

left licenses. Basically, if you modify the garage software and then run that

modified version as a

network service, like people connect to it over the internet, then you must make

the source code

of your modified version available to those users. Ah, so it prevents someone from

taking the open

source code, improving it, and then locking those improvements up in their own

private cloud service.

Exactly. It's designed to make sure that improvements made to free software,

especially when it's run as a service, flow back to the community. It keeps the

technology free and

open. That fits perfectly with this whole mission of democratizing resilient

infrastructure.

Okay, let's move on to the driving goals. Section two, I mean, why bother with all

this complexity?

If I can just plug a huge external hard drive into my server, why build distributed

storage across

multiple locations? You basically hit the nail on the head there. The core

motivations are resilience

and geographical distribution. Right. That single giant hard drive that's a single

point of failure,

power supply dies, fire, flood, whatever, your data's gone. Okay, yeah. Garage tackles

this head-on.

It's explicitly designed for clusters where the nodes, the servers are running in

different

physical locations. Could be different racks, different buildings, different cities

even.

So the idea is if one data center goes dark, maybe a power outage,

a network cut, the data is still safe and accessible somewhere else.

Precisely. High availability through geographic redundancy.

But wait, moving data across the public internet, that sounds like a recipe for

terrible latency consistency problems, doesn't it? Is dealing with that really

worth the

complexity for a small operator? That's the critical question, right?

Yeah. And it's actually what defines

Garage's whole engineering philosophy. Most big enterprise storage software assumes

you

have these perfect low latency, high bandwidth, dedicated network links between

data centers.

Which small operators usually don't have. Exactly. Garage's developers,

being sysadmins themselves running stuff over the regular internet,

just accepted that high latency, maybe up to 200 milliseconds even,

and flaky connections were just reality. So they built around that reality.

Yes. They focused the design on handling those network issues gracefully.

The goal isn't just copying data. It's keeping the service available and ensuring

the data

becomes eventually consistent, even if entire locations are temporarily offline.

So they democratized resilience by not requiring perfect, expensive networking.

That's it. They built the software to deal with the messy reality of non-enterprise

networks.

Making it actually usable. Right. They aimed to make it

operationally manageable, lightweight, prioritizing that availability, that

survival,

above almost everything else. Okay. This is where it gets really

interesting for me. Section three, the how. How does Garage actually achieve this

high resilience?

How does it handle data integrity when you've got machines failing, disks dying,

networks dropping packets, the whole potential mess?

Yeah. The secret sauce. They rely on, well, structured redundancy and concepts like

Quorum.

Garage is built to be highly resilient against pretty much everything.

Disk errors, yes, but also whole network partitions separating geographic locations.

And they mentioned something interesting. Sysadmin failures.

Ah, yeah. They specifically considered that. Because let's be honest, sometimes the

biggest

cause of an outage is human error during maintenance or configuration. The design

tries to minimize the impact of those two. Smart. Okay. Tell us about the data

replication.

How many copies of my data exist and where do they live in this system?

The magic number they landed on is three. Every chunk of data you upload gets

replicated,

stored in three distinct zones. Zones. What's a zone?

Think of a zone as a logical grouping that usually maps to a physical failure

domain.

So it can be a specific rack, a specific building, or most powerfully a specific

data center or geographic location. And importantly, a zone itself usually

consists of multiple servers for redundancy within the zone.

So three copies spread across three potentially separate geographical areas.

Correct. And the logic is classic quorum systems. To survive N1 failures, you need

N copies. Here,

N3. So you can lose up to two entire zones simultaneously.

Two whole data centers could go offline.

Could be, yeah. One is down for planned maintenance and another has a sudden

network

failure. Your data remains fully accessible and readable from that third surviving

zone.

Wow. That level of fault tolerance with potentially just three locations, that's

pretty remarkable for something aimed at smaller scale.

It really is. But achieving that reliability isn't trivial, theoretically speaking.

The sources mention garages standing on the shoulders of giants drawing from

decades of distributed systems research. Right. Let's unpack those giants. They

mention influences like Amazon's Dynamo. What key ideas did Garage borrow from

massive systems like that? This is really the core of it, I think.

The Garage team didn't necessarily invent brand new distributed systems math.

Instead, they very cleverly applied existing, proven, but often complex,

academic research to their specific scale and problem set.

Okay, so Dynamo first. What's the key principle there?

Dynamo, Amazon's highly available key value store.

Its core idea was prioritizing availability over perfect, immediate consistency.

Meaning?

Meaning if there's a network issue, like a flow link or a partition, Dynamo won't

just block the

user and say, try again later. It will likely accept a write operation, let the

user think

it succeeded, and then promise to sort out any potential conflicts later once

communication is

restored.

Uh, so it keeps working even when parts of the system can't talk to each other

properly.

Exactly. It favors being available for reads and writes over ensuring every node

has the

absolute latest data right now. This leads to the concept of eventual consistency.

Okay, eventual consistency. But how do you make sure things do eventually become

consistent

without losing data or getting corrupted when servers have seen different things?

Great question. And that brings us to the second major influence, conflict-free

replicated data

types, or CRDTs.

CRDTs? Sounds complicated.

The concept is brilliant, actually. CRDTs are special data structures designed

specifically

for this eventual consistency model. They allow multiple servers to update the same

piece of data

independently, even while disconnected from each other.

Okay.

And they have mathematical properties that guarantee, when those servers do

eventually

reconnect and share their updates, their states will merge together correctly and

automatically

without losing information and without needing slow, complex locking mechanisms to

coordinate.

Whoa. So CRDTs are like the algorithmic magic that makes eventual consistency

safe and reliable, especially over flaky internet links.

It stops data getting messed up when different nodes are out of sync for a while.

You got it. It elegantly solves the coordination headache that plagues

many traditional distributed databases when dealing with network partitions or high

latency.

It's a huge enabler for systems like Garage.

Fascinating. And there was a third influence mentioned, Maglev.

Right. Maglev. That's Google's high-performance software network load balancer.

Mentioning this shows they weren't just thinking about data storage theory,

but also the practicalities of efficiently routing requests and managing

connections within the

cluster. So handling traffic effectively to make sure requests go to the right

place quickly,

even under load. Exactly. Making sure data requests are

steered efficiently to the nearest or the healthiest node that holds the data.

It's really impressive how they've taken these heavyweight architectural ideas born

from giants

like Amazon and Google and managed to translate them into something lightweight

enough for modest

infrastructure. That's the key achievement, I think. Which brings us nicely to the

practicalities.

Section four, the low barrier to entry. If I actually want to set up this geo-distributed,

super resilient cluster, do I need to go out and buy three identical brand new

server racks?

That's probably the most compelling part for the self-hosting community or small

businesses.

The answer is a definite no. An explicit design goal was keeping the barrier to

entry low.

They actively encourage using existing or even older machines. You don't need a

supercomputer cluster, then? Not at all. The minimum requirements are honestly

quite minimal.

Like what? Per node, they suggest just one gigabyte of RAM.

One gig? Seriously? Yep. And at least 16 gigabytes of disk space.

For the CPU, basically any BI8664 processor from the last decade or so, or an ARMv7

or ARMv8 chip,

think Raspberry Pi level or similar is sufficient. That is incredibly low overhead

for a system

promising this kind of resilience. It really suggests, like you said, if you've got

a few

old office PCs lying around or maybe some cheap virtual servers scattered in

different regions.

You could genuinely start building a garage cluster. That's the core economic

appeal.

And crucially, they build it specifically to allow mixing and matching different

types of hardware.

Ah, so heterogeneous hardware is supported. Explicitly. You can combine servers

with

different CPUs, different amounts of RAM, different disk sizes within the same

cluster.

That massively simplifies things because you don't need to source expensive

identical machines. Use

what you have or what you can get cheap. That's huge for operational reality.

And the deployment is super simple, too. It ships as a single self-contained binary

file,

no complex dependencies to install. It just runs on pretty much any modern Linux

distribution.

Just copy the file and run it, basically. Pretty much. It really emphasizes that

focus on ease of operation for the sysadmin. And just circling back quickly to a

tech detail,

the main language used is Rust, right? About 95% of the code?

Correct. And Rust is known for its performance, efficiency, and especially memory

safety.

That choice directly contributes to those low resource requirements and overall

stability.

That technical excellence seems linked to its sustainability, too. You mentioned it's

not

corporate-backed, but projects like this need ongoing work. How is Garage supported?

Has it managed to find funding? Yes. And that's another critical

point. In a world often dominated by venture capital, Garage has actually secured

significant

public funding, which really signals confidence in it as a public good for the

internet.

That's really interesting. Where did this public funding come from?

It's primarily come via the European Commission's Next Generation Internet,

or NGI initiative. It's had several grants. Can you detail those?

Sure. Back in 2021-2022, the NGI Pointer Fund supported three full-time employees

working on

Garage for a whole year. Wow, three people full-time.

Yeah. Then more recently, from 2023 to 2024, the NLNet Foundation through the NGIU

and Trust Fund

supported one full-time employee. Okay.

And it's ongoing. Looking ahead to 2025, the NLNet NGI Year Commons Fund is

providing support

for the equivalent of 1.5 full-time employees. So there's a steady stream of grant

funding,

keeping this critical piece of decentralized infrastructure alive and evolving,

driven by

community needs, not profit. That seems to be the model, yes.

It keeps it open source, keeps it free, and aligned with that original mission.

Okay. So let's try and summarize the core takeaway for you, the listener. What

Garage seems to have

done is successfully bridge this gap, right, between really advanced complex

distributed

systems theory like Dynamo, CRDTs, and the practical, often hardware-constrained

reality

of self-hosters and small organizations. Exactly. It essentially offers smaller

players

the kind of power, geo-distributed, highly resilient storage that used to be pretty

much exclusively the domain of tech giants with massive budgets.

It levels the playing field in a way. It really does. And this whole project,

this whole approach, it raises a bigger, quite provocative question, I think.

When you have open-source projects, especially ones backed by public funds like NGI,

actively working to democratize access to this kind of complex, resilient

infrastructure,

what does that really mean for the future? For decentralized data ownership, for

control,

it kind of challenges the default assumption that only huge centralized

corporations

can truly guarantee the safety and availability of our important data.

As the cloud continues to consolidate around a few big players,

projects like Garage offer a different path. It's definitely something worth

thinking about.

An excellent point to end on. A powerful thought to mull over, indeed.

Hopefully, you feel much better informed now about the potential of geo-distributed

object storage,

and specifically, the Garage project. We want to once again thank our sponsor,

Safe Server, for supporting this deep dive. They support the hosting of this very

type of software,

helping with digital transformation. You can find out more about how they can

support your

We'll catch you

We'll catch you