Welcome back to the deep dive.
If you've been watching fields like computer vision
or multimodal machine learning grow,
you quickly realize something.
It's often not the algorithms holding things back.
It's the data, like managing it, cleaning it up.
Yeah, and figuring out what's actually worth labeling.
That's a huge one.
Exactly.
So today, we're doing a deep dive into a platform built
specifically for that problem, Lightly Studio.
We've gathered a bunch of sources
to really unpack this open source tool.
Our goal here is pretty simple.
Break down what Lightly Studio is, who it's for,
and walk through the key ideas that, well,
aim to turn messy data wrangling into something more automated.
Try to make it approachable, even if you're just
getting started in this space.
But before we jump into the data pipelines themselves,
we really want to thank the supporter of this deep dive,
SafeServer.
SafeServer handles hosting for exactly this kind
of specialized software, and they support
your digital transformation.
So when you need to deploy tools like Lightly Studio,
they provide that crucial infrastructure.
You can find out more and start your own journey
at www.safeserver.de.
And that infrastructure piece is really key,
because scaling these modern ML projects,
it often hits a wall when the data just gets out of control.
You've got to curate it, label it right,
keep track of every version, every change.
It's a lot.
Lightly Studio really pitches itself
as that unified data platform for multimodal ML.
The idea is consolidating all those tricky, separate steps
into one place, make it manageable.
OK, so let's start right there.
What is this tool, fundamentally?
We know it's from Lightly, it's open source.
But what are those main tasks it's trying to unify?
Right, it's about unifying that whole workflow, curation,
annotation, and management.
It was really built by ML engineers
for ML engineers and organizations
that are trying to scale up computer vision work.
They recognize that, well, speed and flexibility are paramount.
At speed, you mentioned, how does it achieve that?
Well, the sources point to a pretty key architectural choice.
It's built using Rust.
Rust, OK.
Yeah, and Rust is known for performance, right?
And memory safety.
So what that means for someone using it is efficiency.
You can actually handle massive data sets like Kokyo or ImageNet
Scale and do some pretty heavy processing,
even on, say, standard hardware.
Not necessarily a giant server farm.
Like what kind of standard hardware?
I think like a decent laptop, an M1 MacBook Pro with maybe 16 gigs
of RAM, that kind of thing.
OK, that's impressive.
So it's got this powerful engine.
What are the actual functions?
What does the platform do day to day?
Yeah, there are really four core functions
that kind of cover the whole data lifecycle representing
its main value.
First up, you've got label and QA,
so built-in tools for annotating images and videos.
Pretty essential.
Second, it helps you understand and visualize your data.
So you can filter it, automatically find exact duplicates,
which honestly saves so much time,
spot those really important edge cases,
and also catch data drift
as your real-world conditions change.
Okay, makes sense.
Third, it lets you intelligently curate data.
This means automatically selecting the samples
that are actually the most valuable for training
or fine-tuning your model.
We'll probably dig into that more later.
Yeah, definitely want to circle back to that.
And the last one.
And finally, after you've done all that work,
you need to export and deploy that curated data set.
And it lets you do that whether you're running Lightly Studio
on your own machines, on-prem, using a hybrid cloud setup,
or fully in the cloud.
Gotcha, flexible deployment.
Exactly, and you know, we keep saying multimodal ML.
We should probably clarify that a bit.
Good point.
It means the platform isn't just for like standard photos.
It really supports a wide range of data types.
Images, sure, but also video clips, audio files, text,
and importantly, even specialized formats like DICOM data.
Oh, the medical imaging format.
That's the one, yeah, for X-rays, MRIs, that kind of thing.
So having that breadth really makes it a genuinely unified
hub for different data types.
OK, so it handles a lot of data types.
Let's unpack who actually uses this.
Who benefits from this unification?
Yeah.
Because the source has mentioned like two main groups that
don't always talk to each other.
Yeah, that's a good way to put it.
It tries to bridge that gap.
So on one side, you've got the ML engineers, data scientists,
the infrastructure folks.
Yeah.
What's in it for them specifically?
Right.
For the engineers, it's really all
about integration and automation.
It's built with SDKs and API support, all based
on open source standards.
So the idea is it slots into their existing ML stacks
pretty easily without needing a total rewrite of everything.
Less disruption.
Exactly.
And automation is key, right?
That's handled mainly through the Python SDK.
So they can script everything, importing data, managing it.
They can pull data straight from where they usually
keep it, like local folders or cloud storage, like S3 or GCS.
There's big ones from Amazon and Google Cloud, right?
Yeah, the standard object storage.
And crucially, once data is in, it's
not like it's locked forever.
You can keep adding new data to existing data
sets as you get it, which is, well,
vital for any kind of research or iterative development.
Absolutely.
OK, so that's the engineers.
Then on the other side, you mentioned labelers and project
managers.
These might be less technical users.
Often, yeah.
They're focused on the quality assurance, managing
large teams, logistics.
For them, the platform emphasizes
more intuitive workflows.
So a user-friendly GUI, tools for collaboration,
and really critical features, like data set versioning.
Oh, versioning is huge.
Anyone who's tried to reproduce an old result knows that pain.
Totally.
Knowing exactly which version of the labels
you used six months ago, yeah, it's crucial.
Plus, things like role-based permissions
to manage who can do what in the annotation team
make sense for project managers.
So it's bridging these two worlds.
And interestingly, it seems very focused
on making it easy to switch to Lightly Studio.
The sources actually call out that it simplifies migrating
data from other tools.
Oh, like competitors.
Yeah, they mention names like Encore, Voxel 51, Ultralytics,
V7 Labs, Roboflow, popular tools in the space.
It seems like they actively want to be that central data hub,
reducing the friction if the team decides, OK,
we need to consolidate onto one platform.
OK, that makes strategic sense.
Now, here's where I think it gets really interesting,
especially for folks wanting to automate things.
The Python interface.
We don't need to become Python experts here.
But understanding the basic concepts
seems key to unlocking that automation power.
What are the main building blocks for a beginner?
Yeah, definitely.
You can think of it like setting up
a big physical filing system makes it easier to grasp.
So the first main concept is the data set.
If you're using the Python interface,
this is sort of your top level thing.
Think of the data set like your main binder
or the whole filing cabinet.
You use it to set up your data, connect to the database file
where everything's stored, kick off the visual interface
if you need it.
And critically, it's what you use
to run your queries and selections
to find specific data.
OK, so data set is the container.
Then what's inside?
Inside, you have the sample.
So if the data set is the binder,
the sample is like a single page or a single file inside it.
It's just one data instance.
Could be one image, one audio clip, whatever.
And what does it know about itself?
It holds all the key info, the unique ID,
like a serial number, the file name
where it lives on the disk, the absolute path,
and importantly, a list of descriptive tags.
Tags could be anything like reviewed,
needs labeling, nighttime, simple labels you attach.
And it also gives you access to the metadata.
That's all the other descriptive stuff about the file image
resolution when it was captured.
Maybe GPS coordinates, depends on the data.
Dataset holds samples, samples hold info and tags.
What's the third piece?
The third piece is the real power move, data set queries.
This is how you find very specific subsets of your data
without looking through potentially millions of files
manually.
Queries let you combine filtering, sorting, slicing,
using standard logic, those Boolean expressions,
and D, or not Dame.
So is this just like filtering columns in a spreadsheet,
or is it more powerful with this kind of data?
It's way more powerful because you
can query based on the tags and the metadata at the same time.
So for instance, you could build a query like,
find all samples that are tagged, needs labeling, or are.
Find samples where the image width is less than 500 pixels
in D, they have not been tagged as reviewed yet.
OK, so you can get really specific to find
potential problems or gaps.
Exactly.
It finds that precise set of data
that maybe slipped through your initial checks
or needs special attention.
That feels like the big win right there,
turning data prep from this manual slog
into something you can script.
Precisely.
And once you run that query, the really useful part
is you can then immediately do something with that subset,
like apply a new tag to all of them, say, needs review.
Then later in the visual interface,
they're super easy to find and work on just
by filtering for that tag.
Nice.
And for beginners, getting data in
seems pretty straightforward, too.
The sources mention easy ways to load data
from common formats, like YOLO.
That's for object detection, right, bounding boxes.
Yeah.
And Cocoa, which is often used for instant segmentation
or image captions, you just use simple Python functions,
like add samples from Milo or add samples from Cocoa.
Keeping the barrier to entry low for common formats.
Good.
Now, this feels like a good transition
to that feature you mentioned earlier,
the one that really shows off the platform's advanced side.
Selection.
You said this is where it saves real money and time.
Yeah.
This is arguably the core IP, the really smart bit.
Selection is basically automated data selection.
And the purpose is simple, but huge.
Save potentially massive labeling costs
and cut down training time while actually improving
your final model quality.
How does that work?
If I have, say, a million images,
but only budget to label a few hundred,
how does it pick the best hundred?
That sounds tricky.
It is tricky.
That's why automation helps.
It avoids human bias and, frankly,
the tediousness of trying to ensure variety manually.
The mechanism works by automatically picking
the samples considered most useful.
And it does this by balancing two key factors that models
need to be robust.
OK, what are the two factors?
First, you need representative samples.
This is your core data, the typical stuff
your model will see 95% of the time, the normal cases.
Right, the bread and butter.
But if you only train on that, your model
falls apart the moment something slightly unusual happens.
So second, you need diverse samples.
These are the crucial edge cases,
the novel or rare examples, the stuff the model hasn't really
seen before but needs to handle.
OK, so it's like, if I'm training a self-driving car
model, I need lots of pictures of normal daytime driving.
That's your representative data.
But I also absolutely need examples
of driving in heavy rain at night, maybe with a weird object
on the road.
Those are your diverse edge case samples.
Exactly.
Selection aims to pick a subset that
intelligently balances both.
So if I just labeled 100 pictures
of the same boring highway on a sunny day,
I've kind of wasted 99 labels.
Selection stops that.
That's the idea.
It forces variety into your labeled set.
And you, the user, get to control this balance.
You can use different strategies.
For example, a metadata weighting strategy.
Maybe you tell it to prioritize samples tag, nighttime,
or rainy, because you know those are hard cases for your model.
OK, using the tags we talked about earlier.
Right.
Or you could use something like an embedding diversity
strategy.
This is more AI driven.
It looks at the actual visual content
using embeddings, numerical representations of the images,
and picks samples that are mathematically distant
or different from what's already been selected,
even if a human didn't explicitly tag it as diverse.
Wow, OK, that sounds powerful.
It leads to some pretty impressive results,
according to the sources.
They cite things like up to an 80% cut in annotation costs.
80%.
Yeah.
And model iteration cycles getting three times faster.
Plus, actual accuracy bumps, sometimes 10%.
Sometimes even sizes as high as 36%.
That's significant.
It really underlines that focusing on data quality
over just raw quantity pays off.
Absolutely.
The return on investment for curating your data smartly
is undeniable.
So just to wrap up the main idea for the listener,
Lightly Studio positions itself as that central data hub.
It brings together the management, the labeling,
and this intelligent automated curation using selection.
And it also links up with their other tools,
like LightlyTrain for pre-training models
and LightlyEdge for optimizing data collection out
in the field.
So you get the open source flexibility and cost benefits,
but paired with enterprise level security rigor.
They mentioned being ISO 27001 certified,
which is important for businesses.
All right, that security aspect is key for adoption.
Okay, that level of control over the data
brings us to our final thought for you,
the listener, to maybe chew on.
We tend to get really fixated on picking
the perfect model architecture,
tweaking the training parameters,
but maybe the real leverage, the real power,
actually lies in data curation.
So here's the question.
If you could only afford to label, say,
100 images for your next big computer vision project,
how confident are you right now
that those specific 100 images would be the absolute best,
most useful, most diverse set possible
to train your model effectively?
Tools like Lightly Studio are fundamentally designed
to take the guesswork out of that question
and give you some actual measurable certainty,
something to think about.
Well, thank you for joining us for this deep dive
into data management for multimodal ML.
And just one final reminder
that this deep dive was supported by Safe Server.
Safe Server supports your digital transformation
and handles hosting for this kind of software.
You can find out more at www.safeserver.de.
We'll see you next time on the deep dive.