Today's Deep-Dive: CollectiveAccess

Welcome back to The Deep Dive.

Today we're tackling something huge

and often totally invisible.

Right.

How do the world's biggest cultural institutions,

we're talking major museums, huge archives, research

universities, how do they manage these just

colossal collections?

And we're not talking about a few spreadsheets here.

These collections have everything.

Digitized texts, paintings, high res video, audio histories,

and now even complex 3D scans of objects.

That's an incredible mix of data.

Exactly.

And handling all that, keeping it safe,

and making it accessible to people,

that needs a really industrial strength solution.

And that's our focus today.

We're looking at the open source software

that hundreds of these institutions

trust to solve this exact problem, collective access.

We're going to get into its core architecture, which

has this powerful duality providence in Pawtucket 2,

and we'll look especially at the big modernizing updates

in the new version 2.0.

So we've been unpacking the official documentation,

deep diving into the GitHub repos,

and really trying to understand the system.

And our mission here is to give you

a clear structural understanding of it.

We want you to see past the pretty website

and get how the engine behind modern archives actually works.

OK, so let's start at the very beginning, the why.

Collective access, or CA.

It started back in 2003.

Right, and it wasn't born to be a commercial product.

It was a direct response to a massive gap in the market.

What was the gap?

The tools that existed were just prohibitively expensive.

They were proprietary, they were rigid.

Institutions had these really complex needs,

but tight budgets.

So collective access came in as a free, open source alternative.

Exactly.

It's under the GPL 3.0 license.

That meant anyone, a small local history group

or a huge national gallery, could just download it and use it.

No massive licensing fees.

And that accessibility is what let

it grow into what it is today.

Which brings us to, I think, the most fundamental thing

you have to understand about it.

The core duality.

The duality, yeah.

I like to think of it like a restaurant.

You have the kitchen in the back, super organized complex

where all the prep work happens.

And then you have the dining room out front.

Beautiful, simplified, where the guests actually

interact with the final product.

You need that separation.

So the kitchen, the back end, where the hard work is done,

that's called Providence.

Yes.

That's the data management and cataloging application.

That's where the curators and archivists spend their days,

meticulously documenting every single item.

And the dining room, the front end, is Pawtucket 2.

Pawtucket 2, that's the public interface.

It's the website layer that takes all that organized data

from Providence and makes it look good,

makes it interactive for researchers, and well, for you.

And separating them isn't just for convenience, is it?

It's a really core architectural choice.

Why is that so important for an archive?

It's about sanity, really, and security.

If you try to manage everything in one single application,

you risk compromising your complex internal data standards

just to make a slick website.

Ah, I see.

This duality separates the incredibly detailed

internal organization from the user-friendly public discovery.

It means a web designer can go and update Pawtucket 2

without ever, ever risking the archival integrity of the data

that's locked down in Providence.

That makes perfect sense.

So before we dive deeper into Providence,

let's just take a moment to thank the supporter of this deep dive.

Good idea.

SafeServer handles the hosting of exactly this kind

of complex, high-demand software,

and they support institutions with their digital transformation.

So if you're looking for reliable hosting

for something powerful like collective access,

you can find more info at www.safeserver.de.

OK, let's go into that engine room.

Providence, this is the app that's

built to solve what I call the archivist's nightmare.

Handling extreme data diversity.

Exactly.

Yeah.

The number one feature of Providence

is just how configurable it is.

Unlike a lot of off-the-shelf software,

it's not built for just one type of collection.

It supports museums, archives, and research contexts

all at once.

When you say configurable, what does that actually

mean for the person cataloging something?

It means they are forced to sort of shoehorn their collection

into someone else's idea of what a field should be.

Providence lets you build catalogs

that conform to recognized standards, like Dublin Core,

or you can customize totally new fields.

And you can do that without a programmer?

Without writing custom code, yes.

The system adapts to the institution,

not the other way around, which is vital,

because a history museum's needs are just

wildly different from a natural science museum's.

And the media handles.

It's really impressive.

I mean, documents, images, audio, video, that's expected.

But it also handles 3D models.

And that detail tells you how modern the system is.

That's the key technical achievement, really.

How so?

Because managing a huge 3D scan of a fragile artifact, which

is basically a cloud of data points,

the same system as a simple PDF of a letter from 1910,

that requires immense flexibility on the back end.

It treats them both as collection items.

So it's not just about the variety of data.

It's also got to handle the day-to-day work

of running an institution.

What kind of workflow features are built in?

It's all about bulk processing.

An archive might acquire 10,000 records at once.

Providence supports batch importing, vexporting.

You can bring in massive data sets, clean them up,

and then quickly output reports, like inventory lists

as PDFs or spreadsheets.

It's built to scale.

So Providence is the highly granular professional tool

for data integrity.

Right.

But the public never sees any of that.

And all that effort is useless if people

can't find the stuff.

Absolutely.

So now we shift from the structured back end

to the elegant front end, Pawtucket II.

This is where the collection comes alive.

OK, so Pawtucket II takes that data

and focuses on the user experience.

You can style it to match a museum's brand.

But the real power is in the discovery tools, right?

That's right.

It goes way beyond a simple keyword search.

Pawtucket II lets you use customizable facets

and filters.

A facet is what for a non-technical user?

It's basically a preset category.

So instead of you having to search

for blue dress Victorian, you can just

filter by object type, dress, then era, Victorian,

and then color blue.

It lets you explore the collection,

even if you don't know exactly what you're looking for.

And it can turn static data into something interactive.

I've seen examples with maps and timelines.

How does it do that?

It leverages the structured data from Providence.

So if an archivist entered a date and a location

for an object, Pawtucket II can visualize it.

A timeline takes that static date field

and turns it into a navigable journey.

A map takes a GPS coordinate and shows you

where an object is from.

It completely changes the user's relationship

with the collection.

You're not just viewing, you're exploring.

And there are also features for public engagement, things

that let the community contribute.

Precisely.

The institution can decide to turn on public commenting

or tagging or even rating items.

For something like a local history archive,

that's invaluable.

You can have the community help identify people in old photos.

It turns the public into research assistants.

In a way, yes.

And this is not theoretical software.

It's running some really high-profile digital exhibits.

Oh, absolutely.

We're talking about projects like the Crest Digital

Archive at the National Gallery of Art or the Chicago Film

Archive.

They're all using this framework.

OK, so we've established the core duality, Providence,

the rock solid engine, and Pawtucket II,

the engaging public face.

Now let's get to the new stuff.

It was a long wait since version 1.7.

A very long wait.

But Collective Access version 2.0 brings

some huge leaps forward.

The focus seems to be on stability, connectivity,

and this is the big one, AI integration.

Yeah, version 2.0 is basically the developers saying, OK,

we're ready for the next decade.

First, they just future-proof the platform.

It's now compatible with modern server tech,

like PHP 8.2 and 8.3.

Not a flashy feature, but essential for security

and performance.

Vital.

But the real user benefits are somewhere else.

One key improvement is in historical tracking.

You mean provenance, knowing where an object has been.

Exactly.

V2.0 has a new, much more flexible system

for tracking change over time.

It's not just the creation date anymore.

It's about granular history, location changes,

ownership shifts, you name it.

OK, so better accuracy.

But the features that got everyone talking

are about automation, the AI stuff.

I was genuinely surprised when I read

about the transcription feature.

It's a massive leap in efficiency.

V2.0 uses machine learning in two huge areas.

First, automated translation.

The system supports services like DeepL and Google

Translate, so staff using Providence

can work in their own language.

But that immediately lowers the barrier

to entry for global institutions.

It does.

But the truly transformative feature

is the automated transcription.

It uses models like OpenAI Whisper

to automatically transcribe audio and video materials

right inside the workflow.

Just think about that.

An archivist gets 50 years of oral history interviews,

manually transcribing that could take months, maybe years.

And this integration automates it.

It turns all that spoken word into searchable text

almost instantly.

It redefines what's possible for searching media.

So much of that content was basically locked away

from keyword searches before.

This unlocks it.

Wow.

So that's efficiency through AI.

What about connectivity?

How does v2.0 talk to other systems?

They completely modernized it.

They introduced a new GraphQL-based API.

This is a huge deal for any developer trying

to build custom tools that need to talk to the Providence data.

For listeners who aren't developers,

why is switching to GraphQL so important?

Well, a traditional API often makes

you download a big packet of data,

even if you only need one tiny piece of it.

GraphQL is about precision.

A developer can ask for only the specific fields they need.

So it's faster, less resource intensive.

Much faster.

For custom visualizations, mobile apps,

any kind of integration, it's a major upgrade.

And there's also a new support for external media, right?

Yes, another efficiency thing.

V2.0 can now reference content that

lives somewhere else on YouTube, Vimeo, the internet archive.

So the museum doesn't have to eat up all its server space

storing a huge video file if it's already

hosted somewhere else.

Exactly, they just catalog the link.

They keep the metadata integrity and providence,

but save a fortune on storage costs.

And finally, digital preservation.

Long term integrity is everything.

Right, and V2.0 has a new export system

that uses something called configurable Bag IT packages.

Bag IT packages.

Think of them like standardized tamper-proof digital shipping

containers.

They bundle up all the data and the metadata

in a way that can be validated for integrity checks years

or decades down the line.

It's the gold standard.

So it's really clear.

Version 2.0 is all about automation, next gen

connectivity, and making these massive data

sets manageable and secure.

Absolutely, collective access is now, I think,

firmly positioned as a cutting edge open source powerhouse.

You have the surgical precision of Providence for management

and the dynamic presentation of Pawtucket 2.0 for the public.

These updates, especially the AI and the new API,

they just solidify its role for the future.

So the practical takeaway for you, listening,

is that you now understand that two-part architecture.

When you see a beautiful online museum

collection with interactive maps and timelines,

you know there's a good chance software like this

has the engine behind it all, letting

them present their world without compromising their data.

But this does raise an important question, something

for you to mull over as these tools become standard.

Given the new support for automated transcription

and translation, what are the ethical or contextual challenges

that come up?

When we use machine learning to interpret or tag

historical records, can an algorithm

truly capture the necessary historical nuance or context?

Can a machine understand what's not being said in an interview?

Exactly.

That's something every institution

using these amazing new tools is going to have to grapple with.

A fascinating and really crucial point to end on.

Thank you for joining us for this deep dive

into collective access.

And remember, this deep dive was supported by Safe Server.

If you are looking for reliable hosting for complex software

or support for your digital transformation projects,

you can find more information at www.safeserver.de.

Until then, keep digging.

Until then, keep digging.