Okay, so get this. Imagine you're so deep into Kubernetes.
Like you're giving a KubeCon talk about your setup.
You've got to handling millions of users.
But then you get it.
That's what we're diving into today.
Why Gitpod decided Kubernetes,
specifically for their developer environments,
wasn't working.
Yeah, it's really interesting
because they're not saying Kubernetes is bad, right?
They're saying it's not the right tool
when it comes to developer environment.
Exactly, we're looking at their blog post
from October 31st, 2024.
It's this like six year saga of them trying to make it work,
hitting all these roadblocks.
They came up with some pretty interesting workarounds.
Oh yeah.
You almost feel bad for them,
but you learn a lot whether you're deep into Kubernetes
or just curious about developer tools.
Yeah, and it just shows that even teams
with tons of experience, even huge teams,
sometimes have to take a step back and look at their tools.
Right.
You've got to pick the right tool for the job.
Right.
It doesn't have to be the popular one.
Yeah, okay.
So Gitpod's main argument is running applications
in production, that's where Kubernetes shines.
Yeah.
But developer environments, that's a whole different beast.
Totally.
And the blog post breaks down why they highlight
these four characteristics of developer environments.
The first being that they're super stateful
and interactive.
So you've got gigabytes of source code,
you've got build caches, you've got containers running.
All that is constantly changing.
It's not like a stateless app.
Your developer environment is basically an extension of you.
Yeah, it's like the difference between a pristine server room
and your desk.
Your desk has projects everywhere and coffee mugs.
Exactly.
And that mess is really valuable to developers.
So you can imagine it's a huge pain
if they lose changes or get interrupted.
And that leads us to the second characteristic,
unpredictable resource usage.
So you might be coding along, and suddenly, bam,
you need tons of CPU for compilation.
Or memory usage might spike.
Yeah, and Kubernetes isn't really
known for loving surprises.
Not really, no.
Gitpod talks about all the struggles
they had with CPU throttling.
Your terminal's lagging because your IDE is fighting
some random process for resources.
They did all kinds of stuff.
Custom controllers, messing with process priorities,
even tweaking Cgroups V2.
Yeah, and for those who don't know,
Cgroups V2 is how the Linux kernel organizes processes
into these hierarchical groups.
It's for controlling and monitoring
things like CPU and memory and disk I.O.
It's very fine-grained control, but it's complex.
Yeah, it sounds like they went really deep.
Deep down the rabbit hole.
And remember, this is all happening
inside a single container because that's
the way Kubernetes works.
So all these processes crammed together,
it just makes resource usage a total guessing game.
Right.
OK, so then there's memory management.
Apparently, until SwapSpace was available in Kubernetes
version 1.22, overbooking memory was a pretty big risk.
Like, you could end up killing a central processor,
you imagine.
Developer rage.
Yeah.
I mean, this just shows that even mature technologies
like Kubernetes can have limitations, especially
for specific use cases, right?
It's really important to evaluate
whether a tool's strengths really fit what you need it for.
They must've been, I mean, can you imagine?
Pulling their hair out.
Yeah.
OK, so then we have storage performance.
Gitpod really hammers on about how much this matters,
not just for how fast your environment starts up,
but your whole experience inside the environment.
Yeah, because if you're waiting for files to load
or for builds to finish, it just kills your flow.
Totally.
And they tried everything.
SSD, rate zero for speed, a little risky.
Then block storage for availability,
but they hit a wall with persistent volume claims, or PVCs.
For those who aren't deep into Kubernetes,
explain why PVCs were such a pain.
Sure.
So PVCs, it's like this abstraction layer
that lets you use storage.
You don't have to worry about the underlying hardware,
so it's flexible.
But in practice, when these PVCs would attach or detach,
it was unpredictable, and that messed with their attempts
to make workspace startups super fast.
They also ran into some reliability issues,
especially on Google Cloud.
So you're a developer, you're ready to code,
and your whole environment just crashes.
Yeah.
Not a good look.
Talk about a buzzkill.
And then there's backing up and restoring these environments.
They can get huge, right?
Right.
So moving them around became this balancing act
of I-O, network bandwidth, and CPU.
Wow.
They even had to use sick group-based I-O limiters
to prevent one workspace from hogging all the resources
and then starving the others.
It's crazy how these things that sound simple get so complex.
Totally.
Speaking of complex, another challenge?
Autoscaling and startup time.
Yeah.
They were obsessed with minimizing
that initial wait time.
Of course, yeah.
But that clashed with their desire
to use their machines as efficiently as possible.
Yeah, I mean Kubernetes by design
has this inherent lower limit on startup time, right?
Right.
Because of all the steps involved,
moving content around, spinning up containers.
So they started off thinking, let's just
run multiple workspaces on one node
to leverage shared caches.
But that didn't really work out.
Didn't quite work out, no.
So they tried some creative solutions.
They tried something they called ghost workspaces.
Ghost workspaces.
Yeah.
So these were preemptible pods that would just
sit there to hold space so they could scale in advance.
They're like phantom developers taking up space.
That's a good way to put it.
Clever, but too slow and unreliable.
Then they tried ballast pods.
So these were entire nodes filled with dummy pods
just to ensure that they had enough capacity.
Kind of like renting out an empty apartment building
just in case you might need it later.
Pretty much not efficient.
Finally, they landed on cluster autoscaler plugins,
which is a much more elegant solution.
But it took a while to get there.
They even implemented proportional autoscaling,
which basically controls the rate of scale up.
It's based on how quickly devs are starting new environments.
So if there's a sudden rush, they
can add capacity quickly without overshooting.
It's all about finding that balance
between being responsive and making
the most of your resources.
My brain's hurting.
Anyone else?
OK.
Image polls, another headache.
Workspace container images can be huge.
We're talking like 10 gigabytes or more.
And that impacts performance when you have to download and extract
that much data for every workspace.
Yeah, it's like downloading the entire Library of Congress
every time you want to read a book.
Right.
So they tried pre-pulling images with demon sets, which
are basically agents on every node making sure the images are ready.
Then they tried building their own custom images
to maximize layer reuse, even baking images directly
into the node disk image.
Yeah, each of those came with their own trade-offs, right?
Increased complexity, higher costs, limits
on what images devs could use.
Again, another example of how something seemingly simple
can get really complicated at scale.
Yeah, and they even built their own registry facade.
They integrated it with IPFS, the Interplanetary File System,
that decentralized way to store and share files.
They were so proud of it.
They gave a whole KubeCon talk about it.
But in the end, the best solution
was just encouraging everyone to use similar base images,
making caching a lot more effective.
Sometimes the simplest answer really is the best one.
But getting there takes some effort.
OK, buckle up.
We're going into the world of networking in Kubernetes.
And this is where it gets a little technical.
This is where the conflict between what Kubernetes assumes
and what developer environments need becomes really clear.
Yeah, you've got the issue of access control.
You want each environment to be its own little island.
So walled gardens for every developer.
Exactly.
So no peeking at your neighbor's code.
And you need to control who can access what.
Kubernetes has these things called network policies.
They're for defining fine-grained rules
about what traffic can flow within the cluster.
Sounds great, but even those cause headaches for Gitpod.
Of course they did.
So what was their initial approach?
So they started using Kubernetes services and an ingress proxy.
It's to manage access to individual environment ports.
Think your IDE or services running within the workspace.
But as they scaled, this approach became unreliable.
Because more users equals more complexity equals more things
that can go wrong.
Exactly.
With thousands of environments running simultaneously,
name resolution started failing.
Sometimes, it even crashed entire workspaces.
Even established Kubernetes features
have their limits when you push them to the extreme.
It's a good reminder that scaling isn't just
about making things bigger.
No.
It's about making sure they can handle all the complexity that
comes with size.
OK, so resource constraints, another area
where Gitpod face challenges, network bandwidth sharing.
It's like having multiple apartments sharing
the same internet connection, and everyone
wants to stream movies at the same time.
Yeah, just like CPU and memory, you've
got multiple workspaces on a node, all competing
for that same network pipe.
Some container network interfaces, or CNIs,
have features for network shaping,
but that adds even more complexity.
And then there's the question of fairness.
How do you divide up that bandwidth
so everyone gets a decent slice?
It's a never-ending battle.
Balancing performance, security, making
the most of your resources.
And that brings us to, I think, one of the hairiest topics.
Security.
Specifically in the context of developer environments.
How do you give developers the freedom
they need without creating a security nightmare?
This is where the tension between flexibility and control
really comes in.
It gets complicated.
So they start by highlighting this naive approach.
Just give everyone root access to their containers.
Seems simple, right?
Yeah, just give everyone the keys to the kingdom.
What could go wrong?
Well, aside from being a security disaster waiting
to happen, giving users root in their containers
basically gives them root on the node itself.
That means they can potentially swoop around
in other environments that are running on the same node.
They could mess with the infrastructure.
Yeah, not good.
Not exactly what you want.
Not stable.
So they needed something more sophisticated.
Enter user namespaces.
So this is a Linux kernel feature
that lets you map user and group IDs inside containers.
So you can basically make a user feel
like they have root privileges within their environment,
but without actually giving them control over the host system.
OK, that sounds clever, but I bet it wasn't easy to set up.
You bet it wasn't.
Kubernetes did eventually add support for user namespaces
in version 1.25, but Gitpod had already
started their own implementation with version 1.22.
And let me tell you, their solution
involves some serious technical gymnastics.
Give us the highlights.
What kind of gymnastics?
Well, for starters, they had to implement something
called file system UID shifting.
This ensures that files that are created inside the container
are mapped correctly to user IDs on the host system.
So it prevents any security bypasses.
They tried a bunch of different approaches,
like shifts, fuse overlays, even id mapped mounts.
Each of those had their own quirks
in terms of performance and compatibility.
It sounds like they were really pushing the limits of what
Kubernetes could do, trying to fit a square peg
into a round hole.
Exactly.
And then there was a challenge of mounting
what they call a masked proc file system.
So usually when a container starts up, it mounts proc.
This gives it access to information
about the host system.
But for Gitpod's security model, proc
had to be hidden to prevent vulnerabilities.
So they had to create this custom masked proc
and then carefully move it into the right mount
namespace for each container.
And they did this using seccomp notify,
which is like a super low level way to intercept and modify
system calls.
Pretty hardcore stuff.
Wow, it's like they're doing brain surgery on Kubernetes
to make it work.
Pretty much.
But wait, there's more.
They also needed to add support for FUSE file
system in user space.
Yeah.
A lot of developer tools rely on that.
So this involved messing with the container's EBPF device
filter, another low level tweak.
And then there's the issue of network capabilities.
Right.
So as root, you have these powerful capabilities
like KAPNA TADBEN and KAPNA TRAW.
They let you control networking.
Right.
So giving those to a container would totally
break their security model.
Yeah.
So how did they get around that?
Well, they ended up creating another network namespace,
but this time inside the Kubernetes container.
Initially, they used sloop fornets.
And then they switched to veth pairs and custom NF tables
rules.
It's like they were building a secure little networking
sandbox within another sandbox.
It's amazing how much work they put into making this all work.
It really is.
But all this complexity comes with a price, right?
You've got performance hits, especially
with the earlier solutions.
You've got compatibility issues with certain tools.
And then the never-ending struggle
to keep up with Kubernetes updates.
So you can see why they started looking for alternatives.
And that's where their exploration of micro VMs comes
in.
But we're going to save that for part two.
Stay tuned, folks.
Things get really interesting.
Welcome back.
If you're just tuning in, we're talking
about Gitpod's journey, how they went from Kubernetes fans
to creating their own system for developer environments.
Yeah, it got to the point where they were willing to try
anything, even something completely
different from Kubernetes.
Right.
So that's where micro VMs come in.
Now, for those of us who aren't living
in the infrastructure world, can you give us a micro VMs 101?
What are they?
And why was Gitpod so interested?
Sure.
So think of micro VMs like tiny specialized virtual machines,
right?
Strip down to just the essentials.
They boot up super fast, small footprint,
and security is kind of baked into their design.
Gitpod was looking at technologies
like Firecracker, Cloud Hypervisor, QEMU.
So what was it about micro VMs that they were so excited about?
What problems were they hoping to solve that Kubernetes just
wasn't cutting it for?
Well, first and foremost, better resource isolation.
Unlike containers, which share the host's kernel, micro VMs,
they get their own dedicated kernel.
So that means less chance of one environment interfering
with another, more predictable performance overall.
So no more laggy terminal, because your IDE is fighting
some compiler process for CPU.
Exactly.
Another big plus, memory snapshots, near instant resume.
With something like Firecracker, you
can take a snapshot of the entire VM's memory state,
and that includes everything that's running.
You can restore it in an instant.
Wait, so you're saying you could literally
pause your whole developer environment, mid-debug
session, coffee break, whatever, and come back to it
exactly as you left it.
That's the power of micro VMs.
Imagine the productivity boost, especially
for large projects, complex projects,
where restarting everything can take forever.
Yeah, that's a feature I think a lot of developers would love.
For sure.
But I'm guessing there were some downsides, right?
Otherwise, Gitpod would have just switched over
and called it a day.
Of course, no technology is perfect.
One challenge was overhead.
Even though micro VMs are lightweight
compared to like traditional VMs,
they still add more overhead than containers.
And that impacts performance, resource utilization,
which for a platform like Gitpod is a huge deal.
Right, because they're running thousands, if not millions,
of these environments.
Exactly.
Every little bit of efficiency matters.
Another hurdle was image conversion.
Most developer tools, they come packaged as container images
using the OCI standard.
Kubernetes loves that.
But to use those images in a micro VM,
you have to convert them to a format
that the micro VM understands, that adds complexity
and slows down startup.
Right, so it's not just as simple
as swapping out Kubernetes and plugging in micro VMs.
No, it's a whole translation process.
And then there are some limitations
that are specific to micro VM technologies themselves.
For example, Firecracker, which is known for its speed
and its snapshotting.
Well, at the time, it didn't support
GPUs, which is a deal breaker if you're
working on graphics intensive applications.
OK, so even cutting edge technology
has its limitations.
What else did they run into?
Well, data movement became a much bigger problem.
With micro VMs, you're dealing with whole VM images,
including those memory snapshots, which
can be pretty large.
Moving them around, whether it's for backups or scheduling,
gets more complex and it takes more time.
And I bet storage, which was already a pain point,
became even more of a headache.
You got it.
They tried attaching EBS volumes,
that's elastic block storage, from AWS to their micro VMs,
thinking that they could improve startup times
and reduce network strain by keeping the workspace
data local.
But then you run into all these performance quotas, latency
issues, and just the challenge of scaling that approach
across a huge platform.
So kind of swapping one set of problems for another.
In a way.
But the micro VM detour, it wasn't a dead end at all.
It was really a turning point in their thinking.
First, it really solidified their commitment
to things like full workspace backup
and being able to suspend and resume environments.
So that became a must have.
Exactly.
It was non-negotiable.
But maybe more importantly, this experiment
made them really consider moving away from Kubernetes.
Trying to shoehorn these micro VMs into the Kubernetes world
made them realize that there might be a better way.
A way where they weren't constantly fighting
the limitations of the platform.
So it's like, those micro VMs were the gateway drug
to their Kubernetes exodus.
I like that analogy.
It's perfect.
They got a taste of something different.
And they realized maybe they didn't need Kubernetes
after all.
OK, so after all that experimenting,
what was their final move?
Did they find the solution they were searching for?
They did.
They built their own system called Gitpod Flex.
It's designed from the ground up to be like the perfect home
for developer environments.
Taking the best of what they learned
and leaving the Kubernetes baggage behind.
All right, so this is where it gets really interesting.
Tell me more about Gitpod Flex.
What makes it so special?
Well, it's not a complete rejection of Kubernetes, right?
They kept some of the core principles.
For example, declarative APIs are still
a core part of Gitpod Flex.
Remember all those YAML files in Kubernetes?
Yeah.
Defining your infrastructure as code.
Well, that's still there.
OK.
But in a more streamlined and targeted way.
So you still get those benefits of infrastructure as code
without all the complexity.
Right.
And they also kept the use of control theory
for resource management.
This basically means they're using fancy algorithms
to automatically adjust resource allocation based on what's
happening in real time.
OK.
Kind of like Kubernetes auto scaling,
but tailored for how developer environments actually behave.
Right.
So even though it sounds complex under the hood,
what does this mean for developers
who are using Gitpod Flex?
What's the experience like?
Well, one big plus is the seamless integration
with dev containers.
These are like pre-configured, self-contained developer
environments, all the tools, libraries, dependencies,
all bundled up for specific projects.
So it's like a recipe for your perfect developer environment,
just had code.
Exactly.
And Gitpod Flex makes it super easy to spin those up.
They've also really doubled down on self-hosting.
So remember, Gitpod used to offer a cloud and a self-managed
version.
And they said that the self-managed version, which
was heavily Kubernetes-based, was a real pain to support.
Right.
Well, with Gitpod Flex, self-hosting is super easy.
You can have it up and running in less than three minutes
on pretty much any infrastructure.
Three minutes?
That's faster than it takes to order a pizza.
It really is.
And that opens up a lot of possibilities.
Companies can now run their developer environments closer
to their data, even on premises if they need to.
Gives them more control over security, compliance, all
that stuff.
So flexibility and control are really key here.
But what about performance?
All those Kubernetes headaches, the CPU throttling, storage
bottlenecks, all those things.
Have they managed to get rid of those with Gitpod Flex?
And that was one of their main goals.
And from what they've said, it seems
like they made a lot of progress.
By moving away from that shared kernel model of containers
and giving each environment its own dedicated resources,
they've managed to smooth out a lot of those performance
hiccups.
So each environment gets its own slice of the pie.
Exactly.
Now what about that memory snapshot feature
that they were so keen on with micro VMs?
Did that make it into Gitpod Flex?
So they haven't specifically said,
but knowing how much they care about making developer
environments stateful friendly, I
wouldn't be surprised if they're working on it.
Fingers crossed.
Right, because it fits perfectly with their vision.
OK, let's talk about security.
We know they put a ton of effort into securing
their Kubernetes setup.
Oh, yeah.
But it always felt like they were swimming upstream.
Right.
What's the story with Gitpod Flex?
Did they manage to make it simpler but also more secure?
Well, security is kind of baked into Gitpod Flex
from the very beginning.
They went all in on a zero trust architecture.
That basically means no user, no device, no request
is automatically trusted.
Everything has to be authenticated, authorized,
every step of the way.
Fort Knox for code.
Exactly.
This approach kind of avoids a lot of the vulnerabilities
they were dealing with in Kubernetes.
Right.
No more messing around with user namespaces or containers
breaking out of their isolation.
So more secure A and D, easier to manage.
That's the goal.
That's the dream.
Right.
And they've also made it much easier for companies
to apply their own security policies within Gitpod Flex.
So they can hook it into their existing identity management
systems.
They can really control who has access to what.
And they can monitor everything.
So they really put security front and center
from the beginning.
They did.
And it just shows how Gitpod Flex is really built for this.
It's not just about running code.
It's about creating this space where developers
can be productive, collaborative, and secure.
So after this whole journey, what's
the big takeaway here?
What can we learn from their experience?
Welcome back to the Deep Dive.
We've been talking all about Gitpod's journey,
from Kubernetes lovers to creating Gitpod Flex,
their own custom system.
Yeah, it shows that sometimes the most popular solution
isn't always the right one.
They realized Kubernetes just wasn't the right tool
for what they needed.
And they had the guts to go and do their own thing.
Exactly.
So in this final part, let's kind
of dig into what makes Gitpod Flex tick.
What were some of the architectural decisions
they made?
What are the features that really set it apart?
So one of the first things to understand
is that it's not a total rejection of Kubernetes.
They kept some of the core principles.
For example, declarative APIs are still
a big part of Gitpod Flex.
Remember all that YAML configuration
we talked about in Kubernetes?
That approach is still there, but it's a lot more streamlined,
more focused.
So you're still defining your infrastructure as code
without all that Kubernetes baggage.
Exactly.
And they also kept the use of control theory
for resource management.
Basically, this means that they're using these smart
algorithms to automatically adjust resource allocation
based on what's needed in real time,
kind of like Kubernetes auto-scaling, but again,
tailored for developer environments.
Right.
So even though it might sound kind of complex under the hood,
what does it mean for developers who are actually using Gitpod
Flex?
Well, one big benefit is the seamless integration
with dev containers.
These are basically like pre-configured, self-contained
developer environments.
You've got all your tools, libraries, dependencies,
all bundled together for specific projects.
So it's like a recipe for your perfect developer environment.
You just add code.
Exactly.
And Gitpod Flex makes it super easy to just spin those up.
And remember how they were struggling
with self-hosting their platform on Kubernetes?
Yeah.
With Gitpod Flex, self-hosting is incredibly easy.
You can have it up and running in under three minutes
on pretty much any infrastructure.
Three minutes.
That's faster than making a cup of coffee.
Pretty much.
And that opens up a lot of possibilities.
Companies can run their developer environments
closer to their data, even on premises, if they need to.
Gives them more control over security, compliance,
all that good stuff.
So flexibility and control are key here.
What about performance?
They had all those struggles with Kubernetes, CPU
throttling, storage bottlenecks, all those things.
Did they manage to fix those with Gitpod Flex?
That was definitely a top priority for them.
And it seems like they've made some major progress.
By ditching the whole shared kernel model of containers
and giving each environment its own dedicated resources,
they've managed to smooth out a lot of those performance issues.
So no more fighting over resources.
Right.
Every environment gets its own slice of the pie.
Now, what about that memory snapshot feature
that they were so excited about during the micro VM phase?
You know, the one where you could just pause and resume
your entire environment in a snap?
Did that make it into Gitpod Flex?
They haven't explicitly said, but I
wouldn't be surprised if they found a way to make it work.
It really aligns with their goal of making a system that's
truly developer friendly.
Fingers crossed.
OK, let's talk about security.
We know that they put a ton of effort
into securing their Kubernetes setup,
but it felt like they were constantly
fighting an uphill battle.
What's the security story with Gitpod Flex?
Well, security is a core part of Gitpod Flex.
They decided to go all in on a zero trust architecture, which
means that nothing is automatically trusted.
Every user, every device, every request
has to be authenticated and authorized
every step of the way.
So it's like Fort Knox for your code.
Exactly.
And this approach kind of eliminates
a lot of those vulnerabilities that they were always
struggling with in Kubernetes.
No more complex user namespaces or containers breaking out
of their isolation.
So more secure and easier to manage.
It sounds almost too good to be true.
Well, it shows what's possible when
you build a system that's designed for these requirements
from the ground up.
They've also made it a lot easier for companies
to integrate their own security policies into Gitpod Flex,
connecting it with their existing identity management
systems, setting fine grained access controls,
monitoring everything in real time.
So they're giving companies the tools
they need to make sure that everything's locked down.
Exactly.
And this really highlights what Gitpod Flex is all about.
It's not just a platform to run code.
It's an environment that's built to support developers.
A place where they can be productive,
they can be collaborative, and most importantly, secure.
So after this whole journey, what's the big takeaway?
What can we learn from their experience?
I think it's a reminder that sometimes you
have to go against the grain.
The most popular solution isn't always the best, right?
It's about understanding what you need, what your goals are,
and then finding the tools that fit,
even if it means building something yourself.
It's a story about challenging assumptions
and being willing to experiment and having the courage
to try something new when the old way just isn't working.
It really is.
And it makes you wonder, in our own work,
are we forcing tools into roles they weren't meant for?
Are there other systems out there
that could benefit from a similar rethink,
like what Gitpod did?
That's a great question for all of us to think about.
This has been a really interesting deep dive exploring
developer environments and how Gitpod
built this innovative solution.
In this world of technology that's always changing,
being willing to adapt, to experiment,
to break away from the norm, well,
that can lead to some amazing breakthroughs.
It's been a great discussion.
Thanks for joining us on the deep dive.