Museums and other cultural heritage institutions are increasingly discovering the potential of using text and image embeddings to offer rich semantic search. Yet, embeddings have a number of complications and challenges that can complicate their deployment at institutional scale and require careful thought.
- Model Expiration. Nearly every other form of AI-powered record enrichment that museums can use (ASR, OCR, translation, description, tagging, etc) generates additional data for an item record that can continue on with that record in perpetuity, long after the software that generated it becomes obsolete. For example, an OCR transcript of a scanned book generated by an early version of Abby FineReader in the late 1990s can continue to be used today. In contrast, embedding models generate a numeric representation of an input that is meaningless without the model. To search a collection, the user query must be run through the same embedding model to generate a similar numeric representation that can then be compared against the item embeddings. The problem is that embedding models, like all software, become obsolete. Commercially-hosted models can be depreciated and removed from access in as little as a year after release, meaning an institution that paid millions of dollars to embed their entire archive might have to pay several million a year or every other year to annually reembed their entire archive. Open source models tend to rely on supporting frameworks and libraries that can quickly become obsolete and stop working, meaning even a fully open workflow will still likely have to be constantly updated and rerun, forcing institutions to absorb constant ongoing costs.
- Multilingual Mismatch. Museums often have material and item records in a vast range of languages. The plethora of "multilingual" embedding models often tempts museums into thinking that they can just feed all of this content into a single model and allow users searching in one language to identify material in another. Unfortunately, even SOTA multilingual models derived from LLMs perform exceptionally poorly at looking across languages, with two entirely unrelated documents in the same language typically being scored as more similar than the same document in two different languages. In our own work, we've found that the majority of models perform best in English and thus we machine translate all of our content into English when working with embedding models. While machine translation is far from perfect, we've found that it can be tailored to the unique nuances of a given model and that models tend to be highly robust to translation error.
- Task Specific Embeddings. Museums often think of models as just producing an "embedding", but in reality all SOTA models today generate task-specific embeddings, with different tuning for search vs clustering vs retrieval vs matching etc. It is important that museums understand and extensively test which task works best for each of their applications.
- Linguistic Code Switching. Code switching is enormously common in most museum collections, in which an item record contains text in multiple languages (such as the original name of an item or creator and their English transliterated version or the Latin and English versions of a title). Traditional machine translation like NMT cannot properly handle code switching and requires advanced (and far more computationally expensive) translation models like LLMs. This again points to the importance of translating records apriori. In our use cases, we typically create two "English Translation" fields – one for human consumption that is maximal accuracy (and may be generated over time as cost permits) and one that is designed for embedding use in which a lightweight translation designed purely for the embedding models to see is used. Importantly, this means that the human-viewable translation can be upgraded independently of the embedding-centric translation, since the embedding translations may be tuned specifically for a given embedding model and upgraded enmass when the model is changed.
- Vocabulary. It is extremely likely that many of the specialized terms in a museum's item listings are not directly represented in the embedding model they use. This can be addressed through model tuning, but tuning is not available on all commercial models and can pose maintainability challenges. Instead, in our own work we typically use a model-specific preprocessor that rewrites each item listing, replacing unsupported vocabulary terms with those the model understands and writing this rewritten version to a dedicated database field. For example, we've developed a number of workflows that make it possible to inventory all of the unsupported terms and identify and map them onto semantically-similar words the model encodes. In this way, both translation and vocabulary mapping simply become preprocessing workflows.
- Creation Cost. Both commercial and self-hosted embedding models can be quite expensive at scale. Advanced models typically require substantial GPU/TPU accelerator resources or run exceptionally slowly on CPUs, meaning computing embeddings at scale can require a substantial investment. Commercial models can have rate limiting and other quotas that can slow down a conversion workflow or require additional contractual processes to acquire larger quotas. Self-hosted models often require reasonably powerful hardware to run on and often need cloud-scale infrastructure to run at scale. Importantly, museums frequently underestimate the infrastructure complexity and cost of actually feeding all of their content through the embedding model, from error handling and retry workflows to the data storage and networking costs when working with image and video embedding models.
- Query Cost. Once a museum has actually generated embeddings for all of their items, the most common use case is to enable interactive semantic search. This means that all user queries need to be run through the embedding model, which can require substantial hardware. But, the most complex and expensive piece of all is the actual "vector search" infrastructure: search engines that can perform vector search across large embedding archives are exceptionally specialized systems and systems that can scale to billions of embeddings are still highly specialized research experiments at this time.