Thomas Piggott
User Experience Lead

Digital Scholar Lab: Bringing text and data mining to the humanities

The practice of Digital Humanities is a new frontier for study in the humanities. There's a lot that computers can help us with, but it's not how study has traditionally been done in subjects likes literature, philosophy, and history. Scholars were most likely to do deep, close reading of their materials. But doing so was labor intensive and analyzing more than just a few documents could take months or even a lifetime. With the Digital Humanities, text and data mining opens up a whole new way for scholars to explore their domains.

The Challenge - Make Text & Data Mining Approachable

Our research showed that the idea of taking thousands of documents and analyzaing them in minutes didn't come naturally to many of the graduate students we were targeting with this product. They were so familiar with their tried-and-true methods that they had trouble seeing how they could apply these new methods to their research. However, once they understood the idea, their excitement grew as they began to see the possibilities. So we needed to not just provide the tools, but the path to understanding.

Developing and Prioritizing Personas

We wrote 26 personas that encompassed a wide range of users and needs and prioritized them into 1 Primary Persona, 2 Secondary Personas, and 3 Tertiary Personas. The value of this priortization is to get all the stakeholders in a room together and agree on priorities. There are so many needs and diretions a product could go, we needed to get everyone to agree on one direction before we could effectively design.

Product Workflow

We broke our product into 3 steps to help users understand the workflow:

Build: Desiging Search Results for Bulk Document Collection

Search results needed to be different than the standard experience on our platform because instead of looking for a few documents, users would need to determine if they wanted to work with thousands of documents quickly. To facilitate this, we made the metadata more prominent and enabled results to be selected in bulk.

Clean: Making the Need to Clean Understandable

Why Clean? Many documents start as an image scan of a primary source document from the source library. To make it usable, text is created using Optical Character Recognition (OCR). While OCR text algorithms are improving, they aren't perfect and this can lead to incorrect words or characters, especially if the original document scan is low quality or handwritten. The second reason to clean your documents is that certain analysis methods may perform better depending on how they're cleaned. A great example is the use of "stop words". "Stop words" are words that get ignored during analysis to improve results by removing common or meaningless terms like "the" or "a".
How to Make it Understandable? We did a lot of testing to understand how people familiar with this process clean do it and what their reasons were. We learned that the ability to play with these options was crucial. So, we designed a way to preview how your settings would affect your documents.

Analyze: Creating Useful Visualiztions

Being able to interrogate your set of documents in multiple ways is key to exploring those documents in new ways. We aimed to provide flexibility in our visiualizations so that users could explore their documents in multiple ways:

Ask me more!

That's a lot about this project, but there's so much more! I'd love to dive deeper with you on any of the topics above or any other aspect of this project. Here are some starters:

Get in Touch