Welcome back to The Deep Dive.
Today we're tackling something huge
and often totally invisible.
Right.
How do the world's biggest cultural institutions,
we're talking major museums, huge archives, research
universities, how do they manage these just
colossal collections?
And we're not talking about a few spreadsheets here.
These collections have everything.
Digitized texts, paintings, high res video, audio histories,
and now even complex 3D scans of objects.
That's an incredible mix of data.
Exactly.
And handling all that, keeping it safe,
and making it accessible to people,
that needs a really industrial strength solution.
And that's our focus today.
We're looking at the open source software
that hundreds of these institutions
trust to solve this exact problem, collective access.
We're going to get into its core architecture, which
has this powerful duality providence in Pawtucket 2,
and we'll look especially at the big modernizing updates
in the new version 2.0.
So we've been unpacking the official documentation,
deep diving into the GitHub repos,
and really trying to understand the system.
And our mission here is to give you
a clear structural understanding of it.
We want you to see past the pretty website
and get how the engine behind modern archives actually works.
OK, so let's start at the very beginning, the why.
Collective access, or CA.
It started back in 2003.
Right, and it wasn't born to be a commercial product.
It was a direct response to a massive gap in the market.
What was the gap?
The tools that existed were just prohibitively expensive.
They were proprietary, they were rigid.
Institutions had these really complex needs,
but tight budgets.
So collective access came in as a free, open source alternative.
Exactly.
It's under the GPL 3.0 license.
That meant anyone, a small local history group
or a huge national gallery, could just download it and use it.
No massive licensing fees.
And that accessibility is what let
it grow into what it is today.
Which brings us to, I think, the most fundamental thing
you have to understand about it.
The core duality.
The duality, yeah.
I like to think of it like a restaurant.
You have the kitchen in the back, super organized complex
where all the prep work happens.
And then you have the dining room out front.
Beautiful, simplified, where the guests actually
interact with the final product.
You need that separation.
So the kitchen, the back end, where the hard work is done,
that's called Providence.
Yes.
That's the data management and cataloging application.
That's where the curators and archivists spend their days,
meticulously documenting every single item.
And the dining room, the front end, is Pawtucket 2.
Pawtucket 2, that's the public interface.
It's the website layer that takes all that organized data
from Providence and makes it look good,
makes it interactive for researchers, and well, for you.
And separating them isn't just for convenience, is it?
It's a really core architectural choice.
Why is that so important for an archive?
It's about sanity, really, and security.
If you try to manage everything in one single application,
you risk compromising your complex internal data standards
just to make a slick website.
Ah, I see.
This duality separates the incredibly detailed
internal organization from the user-friendly public discovery.
It means a web designer can go and update Pawtucket 2
without ever, ever risking the archival integrity of the data
that's locked down in Providence.
That makes perfect sense.
So before we dive deeper into Providence,
let's just take a moment to thank the supporter of this deep dive.
Good idea.
SafeServer handles the hosting of exactly this kind
of complex, high-demand software,
and they support institutions with their digital transformation.
So if you're looking for reliable hosting
for something powerful like collective access,
you can find more info at www.safeserver.de.
OK, let's go into that engine room.
Providence, this is the app that's
built to solve what I call the archivist's nightmare.
Handling extreme data diversity.
Exactly.
Yeah.
The number one feature of Providence
is just how configurable it is.
Unlike a lot of off-the-shelf software,
it's not built for just one type of collection.
It supports museums, archives, and research contexts
all at once.
When you say configurable, what does that actually
mean for the person cataloging something?
It means they are forced to sort of shoehorn their collection
into someone else's idea of what a field should be.
Providence lets you build catalogs
that conform to recognized standards, like Dublin Core,
or you can customize totally new fields.
And you can do that without a programmer?
Without writing custom code, yes.
The system adapts to the institution,
not the other way around, which is vital,
because a history museum's needs are just
wildly different from a natural science museum's.
And the media handles.
It's really impressive.
I mean, documents, images, audio, video, that's expected.
But it also handles 3D models.
And that detail tells you how modern the system is.
That's the key technical achievement, really.
How so?
Because managing a huge 3D scan of a fragile artifact, which
is basically a cloud of data points,
the same system as a simple PDF of a letter from 1910,
that requires immense flexibility on the back end.
It treats them both as collection items.
So it's not just about the variety of data.
It's also got to handle the day-to-day work
of running an institution.
What kind of workflow features are built in?
It's all about bulk processing.
An archive might acquire 10,000 records at once.
Providence supports batch importing, vexporting.
You can bring in massive data sets, clean them up,
and then quickly output reports, like inventory lists
as PDFs or spreadsheets.
It's built to scale.
So Providence is the highly granular professional tool
for data integrity.
Right.
But the public never sees any of that.
And all that effort is useless if people
can't find the stuff.
Absolutely.
So now we shift from the structured back end
to the elegant front end, Pawtucket II.
This is where the collection comes alive.
OK, so Pawtucket II takes that data
and focuses on the user experience.
You can style it to match a museum's brand.
But the real power is in the discovery tools, right?
That's right.
It goes way beyond a simple keyword search.
Pawtucket II lets you use customizable facets
and filters.
A facet is what for a non-technical user?
It's basically a preset category.
So instead of you having to search
for blue dress Victorian, you can just
filter by object type, dress, then era, Victorian,
and then color blue.
It lets you explore the collection,
even if you don't know exactly what you're looking for.
And it can turn static data into something interactive.
I've seen examples with maps and timelines.
How does it do that?
It leverages the structured data from Providence.
So if an archivist entered a date and a location
for an object, Pawtucket II can visualize it.
A timeline takes that static date field
and turns it into a navigable journey.
A map takes a GPS coordinate and shows you
where an object is from.
It completely changes the user's relationship
with the collection.
You're not just viewing, you're exploring.
And there are also features for public engagement, things
that let the community contribute.
Precisely.
The institution can decide to turn on public commenting
or tagging or even rating items.
For something like a local history archive,
that's invaluable.
You can have the community help identify people in old photos.
It turns the public into research assistants.
In a way, yes.
And this is not theoretical software.
It's running some really high-profile digital exhibits.
Oh, absolutely.
We're talking about projects like the Crest Digital
Archive at the National Gallery of Art or the Chicago Film
Archive.
They're all using this framework.
OK, so we've established the core duality, Providence,
the rock solid engine, and Pawtucket II,
the engaging public face.
Now let's get to the new stuff.
It was a long wait since version 1.7.
A very long wait.
But Collective Access version 2.0 brings
some huge leaps forward.
The focus seems to be on stability, connectivity,
and this is the big one, AI integration.
Yeah, version 2.0 is basically the developers saying, OK,
we're ready for the next decade.
First, they just future-proof the platform.
It's now compatible with modern server tech,
like PHP 8.2 and 8.3.
Not a flashy feature, but essential for security
and performance.
Vital.
But the real user benefits are somewhere else.
One key improvement is in historical tracking.
You mean provenance, knowing where an object has been.
Exactly.
V2.0 has a new, much more flexible system
for tracking change over time.
It's not just the creation date anymore.
It's about granular history, location changes,
ownership shifts, you name it.
OK, so better accuracy.
But the features that got everyone talking
are about automation, the AI stuff.
I was genuinely surprised when I read
about the transcription feature.
It's a massive leap in efficiency.
V2.0 uses machine learning in two huge areas.
First, automated translation.
The system supports services like DeepL and Google
Translate, so staff using Providence
can work in their own language.
But that immediately lowers the barrier
to entry for global institutions.
It does.
But the truly transformative feature
is the automated transcription.
It uses models like OpenAI Whisper
to automatically transcribe audio and video materials
right inside the workflow.
Just think about that.
An archivist gets 50 years of oral history interviews,
manually transcribing that could take months, maybe years.
And this integration automates it.
It turns all that spoken word into searchable text
almost instantly.
It redefines what's possible for searching media.
So much of that content was basically locked away
from keyword searches before.
This unlocks it.
Wow.
So that's efficiency through AI.
What about connectivity?
How does v2.0 talk to other systems?
They completely modernized it.
They introduced a new GraphQL-based API.
This is a huge deal for any developer trying
to build custom tools that need to talk to the Providence data.
For listeners who aren't developers,
why is switching to GraphQL so important?
Well, a traditional API often makes
you download a big packet of data,
even if you only need one tiny piece of it.
GraphQL is about precision.
A developer can ask for only the specific fields they need.
So it's faster, less resource intensive.
Much faster.
For custom visualizations, mobile apps,
any kind of integration, it's a major upgrade.
And there's also a new support for external media, right?
Yes, another efficiency thing.
V2.0 can now reference content that
lives somewhere else on YouTube, Vimeo, the internet archive.
So the museum doesn't have to eat up all its server space
storing a huge video file if it's already
hosted somewhere else.
Exactly, they just catalog the link.
They keep the metadata integrity and providence,
but save a fortune on storage costs.
And finally, digital preservation.
Long term integrity is everything.
Right, and V2.0 has a new export system
that uses something called configurable Bag IT packages.
Bag IT packages.
Think of them like standardized tamper-proof digital shipping
containers.
They bundle up all the data and the metadata
in a way that can be validated for integrity checks years
or decades down the line.
It's the gold standard.
So it's really clear.
Version 2.0 is all about automation, next gen
connectivity, and making these massive data
sets manageable and secure.
Absolutely, collective access is now, I think,
firmly positioned as a cutting edge open source powerhouse.
You have the surgical precision of Providence for management
and the dynamic presentation of Pawtucket 2.0 for the public.
These updates, especially the AI and the new API,
they just solidify its role for the future.
So the practical takeaway for you, listening,
is that you now understand that two-part architecture.
When you see a beautiful online museum
collection with interactive maps and timelines,
you know there's a good chance software like this
has the engine behind it all, letting
them present their world without compromising their data.
But this does raise an important question, something
for you to mull over as these tools become standard.
Given the new support for automated transcription
and translation, what are the ethical or contextual challenges
that come up?
When we use machine learning to interpret or tag
historical records, can an algorithm
truly capture the necessary historical nuance or context?
Can a machine understand what's not being said in an interview?
Exactly.
That's something every institution
using these amazing new tools is going to have to grapple with.
A fascinating and really crucial point to end on.
Thank you for joining us for this deep dive
into collective access.
And remember, this deep dive was supported by Safe Server.
If you are looking for reliable hosting for complex software
or support for your digital transformation projects,
you can find more information at www.safeserver.de.
We'll be back soon to unpack another stack of sources.
Until then, keep digging.