The first 30 years of the Web was largely defined by humans using web browsers to manually search, browse and explore, using search engines to locate content and visiting myriad individual websites (or walled gardens like social media platforms) to consume content. The past year has fundamentally altered the very nature of how the web is used: increasingly it is a landscape of automated AI "agents" operating on behalf of humans to find the information they are looking for. For cultural heritage institutions in particular, this means fundamentally rethinking how they present their collections and how they build their underlying technical infrastructure.
Many institutions still think of automated traffic in terms of search engine indexing bots and harvesting bots used by academic researchers and AI training companies to build datasets. They tend to focus on technical interventions that enable the former while rejecting the latter through various technical measures. This year those two categories have been joined by a third: AI models operating under the direction of a human user (so-called "agentic" traffic).
Agentic traffic is VERY different from training traffic and is focused on high-intensity massive surges of traffic that prioritize realtime response to their human user over the stability of the site they are accessing. Often, the human user has no awareness their quick question to their favorite chatbot about a painting they saw, triggered that chatbot to spawn 20 subagents to run searches against the museum's website in parallel and navigate dozens or even hundreds of pages from that museum site to answer the user's question.
Perhaps the most common misconception is that agentic AI is limited to advanced highly technical users specializing in AI. In fact, nearly every AI chatbot today features at least some form of agentic capabilities. Ask ChatGPT any question involving something that might be in a museum and there is a good chance that instead of instantly generating a response, you'll instead see ChatGPT say it is searching various websites in parallel for the answer or offering to set up a scheduled search every morning for the result. Agentic AI is now so embedded into the chatbot landscape that even the least technical user of a chatbot now likely commands dozens of agentic tasks a day without ever knowing what an AI "agent" is.
Many museums still think of unknown automated traffic as AI training set construction, either attempting to block it or hoping to offload it by offering dedicated APIs or dataset downloads for AI users). Those approaches aren't effective for agentic traffic, since agents represent AI models operating on-demand to answer a human query in realtime, meaning they use the same public website that their human operator would use. Worse, modern agentic traffic is increasingly harnessing actual web browsers for its web access (increasingly using the user's own web browser on their own device), meaning it is increasingly effectively impossible to differentiate human versus human-directed-machine traffic. All of those fancy Cloudflare-style JavaScript gateway tools are useless when agents use a person's actual browser running on their actual laptop and navigate and interact with the site exactly like a human (especially as computer use models are increasingly trained to mimic the nuances and oddities of human browser use, down to how humans move their mouses).
Building robust and resilient technical web infrastructure to handle agentic traffic is utterly unlike traditional web hosting design. Agentic systems utilize massive overwhelming parallelism to answer a user's query as quickly as possible and rarely impose chatbot-wide rate-limiting for any given website. This means that museum websites must be capable of absorbing immense sudden out-of-the-blue surges of traffic that can scale to millions of queries in a few minutes and then back down again.
What is a cultural heritage institution to do when their site is being barraged by millions or even billions of agentic queries a day? In our own case, one of our key websites used by journalists and scholars was collapsing under a massive surge of agentic traffic. Just switching the site's infrastructure from a large fleet of web servers to GCP's Global External Load Balancer, GCS bucket and Cloud CDN means it is now powered by a single tiny VM running idle at 1% traffic and seamlessly scaling to massive surges of traffic without us ever noticing a thing.
While such infrastructure imposes significant costs, museums must recognize that the future of the web is increasingly defined by AI agents rapidly searching on behalf of humans, rather than humans leisurely strolling through the digital wonderland.