Presentation type: Presentation
Abstract:Even though the creation of digital heritage collections began with the acquisition in image mode, several decades later to search in the content of some of these images still belongs to a more or less distant future. This apparent paradox originates in two facts: digital libraries (DLs) first focused on applying OCR to their printed materials, which renders major services in terms of information retrieval; Searching in large collections of images remains a technical challenge, despite the efforts of the scientific community to address the underlying challenging tasks. However, the needs are very real, if one believes user surveys or statistical studies of user behavior. But DLs images collections are generally inadequate, given the broad spectrum of areas of knowledge and time periods surveyed by users. However, DLs are rich in many other iconographic sources (e.g. newspapers and magazines). But organized in data silos that are not interoperable, most often lacking the descriptors required for image search, and exposed through text-oriented user interfaces. While the querying of iconographic content poses specific chalenges, answers to various use cases, targets different knowledge domains, and finally calls for specific human-machine interac-tions.
This work presents a proposal for a solution to two of these challenges, the creation of an encyclopedic heritage image database and its hybrid querying modalities. It presents an ETL (extract-transform-load) approach to this need, that aims to: Identify and extract iconography wherever it may be found, in image collections but also in printed materials (on the condition that the collections are IIIF compliant); Transform, harmonize and enrich the image descriptive metadata, thanks to deep learning approaches and IIIF for images access: in-house Convolutional Neural Network based classification model of images genres (map, picture, drawing, comics…) and SaS for visual analysis (IBM Watson, Google Cloud Vision); Load it all into a web app dedicated to hybrid image retrieval. IIIF is used as an end-to-end versatile tool (as inputs to the AI processing as well as resources for the web app), and this use case is fully commented.
The approach is pragmatic, since it involves leveraging existing digital resources, standard DLs protocols (particulary IIIF) and (virtually) on-the-shelf AI technologies. The resulting web app, GallicaPix, is showcasing WW1 iconographic materials from two digital libraries (Gallica and the Wellcome Library, the last one harvested thanks to Europeana). GallicaPix is referenced on Gallica Studio, the Gallica Lab, for user testing. Its database is publicly avalaible on api.bnf.fr for reuse by researchers (digital humanities and computing sciences).
Topics:
- Using IIIF material for Machine Learning and AI,
- IIIF Implementation Spectrum: large-scale or small-scale projects