Today's Deep-Dive: Lightly Studio

Welcome back to the deep dive.

If you've been watching fields like computer vision

or multimodal machine learning grow,

you quickly realize something.

It's often not the algorithms holding things back.

It's the data, like managing it, cleaning it up.

Yeah, and figuring out what's actually worth labeling.

That's a huge one.

Exactly.

So today, we're doing a deep dive into a platform built

specifically for that problem, Lightly Studio.

We've gathered a bunch of sources

to really unpack this open source tool.

Our goal here is pretty simple.

Break down what Lightly Studio is, who it's for,

and walk through the key ideas that, well,

aim to turn messy data wrangling into something more automated.

Try to make it approachable, even if you're just

getting started in this space.

But before we jump into the data pipelines themselves,

we really want to thank the supporter of this deep dive,

SafeServer.

SafeServer handles hosting for exactly this kind

of specialized software, and they support

your digital transformation.

So when you need to deploy tools like Lightly Studio,

they provide that crucial infrastructure.

You can find out more and start your own journey

at www.safeserver.de.

And that infrastructure piece is really key,

because scaling these modern ML projects,

it often hits a wall when the data just gets out of control.

You've got to curate it, label it right,

keep track of every version, every change.

It's a lot.

Lightly Studio really pitches itself

as that unified data platform for multimodal ML.

The idea is consolidating all those tricky, separate steps

into one place, make it manageable.

OK, so let's start right there.

What is this tool, fundamentally?

We know it's from Lightly, it's open source.

But what are those main tasks it's trying to unify?

Right, it's about unifying that whole workflow, curation,

annotation, and management.

It was really built by ML engineers

for ML engineers and organizations

that are trying to scale up computer vision work.

They recognize that, well, speed and flexibility are paramount.

At speed, you mentioned, how does it achieve that?

Well, the sources point to a pretty key architectural choice.

It's built using Rust.

Rust, OK.

Yeah, and Rust is known for performance, right?

And memory safety.

So what that means for someone using it is efficiency.

You can actually handle massive data sets like Kokyo or ImageNet

Scale and do some pretty heavy processing,

even on, say, standard hardware.

Not necessarily a giant server farm.

Like what kind of standard hardware?

I think like a decent laptop, an M1 MacBook Pro with maybe 16 gigs

of RAM, that kind of thing.

OK, that's impressive.

So it's got this powerful engine.

What are the actual functions?

What does the platform do day to day?

Yeah, there are really four core functions

that kind of cover the whole data lifecycle representing

its main value.

First up, you've got label and QA,

so built-in tools for annotating images and videos.

Pretty essential.

Second, it helps you understand and visualize your data.

So you can filter it, automatically find exact duplicates,

which honestly saves so much time,

spot those really important edge cases,

and also catch data drift

as your real-world conditions change.

Okay, makes sense.

Third, it lets you intelligently curate data.

This means automatically selecting the samples

that are actually the most valuable for training

or fine-tuning your model.

We'll probably dig into that more later.

Yeah, definitely want to circle back to that.

And the last one.

And finally, after you've done all that work,

you need to export and deploy that curated data set.

And it lets you do that whether you're running Lightly Studio

on your own machines, on-prem, using a hybrid cloud setup,

or fully in the cloud.

Gotcha, flexible deployment.

Exactly, and you know, we keep saying multimodal ML.

We should probably clarify that a bit.

Good point.

It means the platform isn't just for like standard photos.

It really supports a wide range of data types.

Images, sure, but also video clips, audio files, text,

and importantly, even specialized formats like DICOM data.

Oh, the medical imaging format.

That's the one, yeah, for X-rays, MRIs, that kind of thing.

So having that breadth really makes it a genuinely unified

hub for different data types.

OK, so it handles a lot of data types.

Let's unpack who actually uses this.

Who benefits from this unification?

Yeah.

Because the source has mentioned like two main groups that

don't always talk to each other.

Yeah, that's a good way to put it.

It tries to bridge that gap.

So on one side, you've got the ML engineers, data scientists,

the infrastructure folks.

Yeah.

What's in it for them specifically?

Right.

For the engineers, it's really all

about integration and automation.

It's built with SDKs and API support, all based

on open source standards.

So the idea is it slots into their existing ML stacks

pretty easily without needing a total rewrite of everything.

Less disruption.

Exactly.

And automation is key, right?

That's handled mainly through the Python SDK.

So they can script everything, importing data, managing it.

They can pull data straight from where they usually

keep it, like local folders or cloud storage, like S3 or GCS.

There's big ones from Amazon and Google Cloud, right?

Yeah, the standard object storage.

And crucially, once data is in, it's

not like it's locked forever.

You can keep adding new data to existing data

sets as you get it, which is, well,

vital for any kind of research or iterative development.

Absolutely.

OK, so that's the engineers.

Then on the other side, you mentioned labelers and project

managers.

These might be less technical users.

Often, yeah.

They're focused on the quality assurance, managing

large teams, logistics.

For them, the platform emphasizes

more intuitive workflows.

So a user-friendly GUI, tools for collaboration,

and really critical features, like data set versioning.

Oh, versioning is huge.

Anyone who's tried to reproduce an old result knows that pain.

Totally.

Knowing exactly which version of the labels

you used six months ago, yeah, it's crucial.

Plus, things like role-based permissions

to manage who can do what in the annotation team

make sense for project managers.

So it's bridging these two worlds.

And interestingly, it seems very focused

on making it easy to switch to Lightly Studio.

The sources actually call out that it simplifies migrating

data from other tools.

Oh, like competitors.

Yeah, they mention names like Encore, Voxel 51, Ultralytics,

V7 Labs, Roboflow, popular tools in the space.

It seems like they actively want to be that central data hub,

reducing the friction if the team decides, OK,

we need to consolidate onto one platform.

OK, that makes strategic sense.

Now, here's where I think it gets really interesting,

especially for folks wanting to automate things.

The Python interface.

We don't need to become Python experts here.

But understanding the basic concepts

seems key to unlocking that automation power.

What are the main building blocks for a beginner?

Yeah, definitely.

You can think of it like setting up

a big physical filing system makes it easier to grasp.

So the first main concept is the data set.

If you're using the Python interface,

this is sort of your top level thing.

Think of the data set like your main binder

or the whole filing cabinet.

You use it to set up your data, connect to the database file

where everything's stored, kick off the visual interface

if you need it.

And critically, it's what you use

to run your queries and selections

to find specific data.

OK, so data set is the container.

Then what's inside?

Inside, you have the sample.

So if the data set is the binder,

the sample is like a single page or a single file inside it.

It's just one data instance.

Could be one image, one audio clip, whatever.

And what does it know about itself?

It holds all the key info, the unique ID,

like a serial number, the file name

where it lives on the disk, the absolute path,

and importantly, a list of descriptive tags.

Tags could be anything like reviewed,

needs labeling, nighttime, simple labels you attach.

And it also gives you access to the metadata.

That's all the other descriptive stuff about the file image

resolution when it was captured.

Maybe GPS coordinates, depends on the data.

Dataset holds samples, samples hold info and tags.

What's the third piece?

The third piece is the real power move, data set queries.

This is how you find very specific subsets of your data

without looking through potentially millions of files

manually.

Queries let you combine filtering, sorting, slicing,

using standard logic, those Boolean expressions,

and D, or not Dame.

So is this just like filtering columns in a spreadsheet,

or is it more powerful with this kind of data?

It's way more powerful because you

can query based on the tags and the metadata at the same time.

So for instance, you could build a query like,

find all samples that are tagged, needs labeling, or are.

Find samples where the image width is less than 500 pixels

in D, they have not been tagged as reviewed yet.

OK, so you can get really specific to find

potential problems or gaps.

Exactly.

It finds that precise set of data

that maybe slipped through your initial checks

or needs special attention.

That feels like the big win right there,

turning data prep from this manual slog

into something you can script.

Precisely.

And once you run that query, the really useful part

is you can then immediately do something with that subset,

like apply a new tag to all of them, say, needs review.

Then later in the visual interface,

they're super easy to find and work on just

by filtering for that tag.

Nice.

And for beginners, getting data in

seems pretty straightforward, too.

The sources mention easy ways to load data

from common formats, like YOLO.

That's for object detection, right, bounding boxes.

Yeah.

And Cocoa, which is often used for instant segmentation

or image captions, you just use simple Python functions,

like add samples from Milo or add samples from Cocoa.

Keeping the barrier to entry low for common formats.

Good.

Now, this feels like a good transition

to that feature you mentioned earlier,

the one that really shows off the platform's advanced side.

Selection.

You said this is where it saves real money and time.

Yeah.

This is arguably the core IP, the really smart bit.

Selection is basically automated data selection.

And the purpose is simple, but huge.

Save potentially massive labeling costs

and cut down training time while actually improving

your final model quality.

How does that work?

If I have, say, a million images,

but only budget to label a few hundred,

how does it pick the best hundred?

That sounds tricky.

It is tricky.

That's why automation helps.

It avoids human bias and, frankly,

the tediousness of trying to ensure variety manually.

The mechanism works by automatically picking

the samples considered most useful.

And it does this by balancing two key factors that models

need to be robust.

OK, what are the two factors?

First, you need representative samples.

This is your core data, the typical stuff

your model will see 95% of the time, the normal cases.

Right, the bread and butter.

But if you only train on that, your model

falls apart the moment something slightly unusual happens.

So second, you need diverse samples.

These are the crucial edge cases,

the novel or rare examples, the stuff the model hasn't really

seen before but needs to handle.

OK, so it's like, if I'm training a self-driving car

model, I need lots of pictures of normal daytime driving.

That's your representative data.

But I also absolutely need examples

of driving in heavy rain at night, maybe with a weird object

on the road.

Those are your diverse edge case samples.

Exactly.

Selection aims to pick a subset that

intelligently balances both.

So if I just labeled 100 pictures

of the same boring highway on a sunny day,

I've kind of wasted 99 labels.

Selection stops that.

That's the idea.

It forces variety into your labeled set.

And you, the user, get to control this balance.

You can use different strategies.

For example, a metadata weighting strategy.

Maybe you tell it to prioritize samples tag, nighttime,

or rainy, because you know those are hard cases for your model.

OK, using the tags we talked about earlier.

Right.

Or you could use something like an embedding diversity

strategy.

This is more AI driven.

It looks at the actual visual content

using embeddings, numerical representations of the images,

and picks samples that are mathematically distant

or different from what's already been selected,

even if a human didn't explicitly tag it as diverse.

Wow, OK, that sounds powerful.

It leads to some pretty impressive results,

according to the sources.

They cite things like up to an 80% cut in annotation costs.

80%.

Yeah.

And model iteration cycles getting three times faster.

Plus, actual accuracy bumps, sometimes 10%.

Sometimes even sizes as high as 36%.

That's significant.

It really underlines that focusing on data quality

over just raw quantity pays off.

Absolutely.

The return on investment for curating your data smartly

is undeniable.

So just to wrap up the main idea for the listener,

Lightly Studio positions itself as that central data hub.

It brings together the management, the labeling,

and this intelligent automated curation using selection.

And it also links up with their other tools,

like LightlyTrain for pre-training models

and LightlyEdge for optimizing data collection out

in the field.

So you get the open source flexibility and cost benefits,

but paired with enterprise level security rigor.

They mentioned being ISO 27001 certified,

which is important for businesses.

All right, that security aspect is key for adoption.

Okay, that level of control over the data

brings us to our final thought for you,

the listener, to maybe chew on.

We tend to get really fixated on picking

the perfect model architecture,

tweaking the training parameters,

but maybe the real leverage, the real power,

actually lies in data curation.

So here's the question.

If you could only afford to label, say,

100 images for your next big computer vision project,

how confident are you right now

that those specific 100 images would be the absolute best,

most useful, most diverse set possible

to train your model effectively?

Tools like Lightly Studio are fundamentally designed

to take the guesswork out of that question

and give you some actual measurable certainty,

something to think about.

Well, thank you for joining us for this deep dive

into data management for multimodal ML.

And just one final reminder

that this deep dive was supported by Safe Server.

Safe Server supports your digital transformation

and handles hosting for this kind of software.

We'll see you next time on the deep dive.

We'll see you next time on the deep dive.