Perhaps the greatest lesson we have been learning first-hand across all of our traffic over the last few months is the exponential and relentless rise of autonomous agentically-controlled AI traffic. Absent the occasional bad actor, previous generations of crawling bots were overall reasonably well-behaved, keeping their crawling rate to reasonable levels, being respectful of server resources and adjusting their crawling rates fleet-wide in realtime as a server's response time slowed. The new generation of agentic AI crawlers prioritize absolute speed-to-response for their end user over the safety and stability of remote websites, spawning in a few milliseconds dozens or even hundreds of requests in parallel to a server to read through everything it needs to give the user a response as fast as absolutely possible. Worse, as every company races to integrate agentic search into their chatbots and services, suddenly every chatbot request worldwide about a certain specialized set of topics can lead to what amounts to a never-ending DDOS attack. Cultural heritage institutions, by virtue of providing public access to unique and highly specialized knowledge archives, are often on the frontline of that traffic deluge as agents learn to use them as authoritative sources for their collections. What is a modern museum to do to withstand all this new traffic? It turns out that just patching together cloud storage and cloud load balancers, cultural heritage institutions can provide rock-solid access to their collections without blinking an eye.
In our own case, one of our key sites used by academic researchers and journalists had become largely unavailable in recent months as the server infrastructure hosting it literally collapsed under the weight of incoming traffic. Live human users were stuck in line behind myriad agentic AI bots researching topics for other humans. By transitioning the site to GCP's Global External Load Balancer we were able to enable GCP's CDN global edge caching with a single checkbox. Despite traffic to the site soaring vertically, we were able to scale the underlying hardware down to a single GCE VM that now spends almost its entire time at 1% CPU as the managed GLB and CDN network services handle all of the traffic magically.
Importantly, we use the Global External Load Balancer's ability to serve content directly from a GCS bucket to handle all of the bandwidth-intensive serving. This means that our entire infrastructure for that site has been reduced to a single GCE VM, a single GCS bucket and a Global External Load Balancer with Cloud CDN enabled sitting in front of that VM and GCS bucket. The load balancer and CDN now handle nearly the entire traffic load entirely on their own, with the end result that our VM is now sitting nearly idle and visitors see nearly instantaneous response times as content is served from Google's global geographically distributed edge caching network (the same network that serves Google itself).
In fact, this new infrastructure is working so well that we don't even notice even the most massive onslaughts of agentic swarms as the GLB and CDN simply absorb all of that traffic seamlessly, deflecting all of it away from our actual server.
How might a cultural heritage institution leverage this model for their own websites and archives? Imagine an institution that has 100PB of high resolution imagery, video, audio, text and other material they want to share with the public. They can simply load it all into a GCS bucket and set up a Global External Load Balancer pointing to the bucket and redirect their website content subdomain to the load balancer's IP address in DNS. A quick clicking of the "Enable Cloud CDN" checkbox and the setup is complete. Within seconds they can now serve even the most massive global-scale traffic surges without having to think about anything and without their system administrators having to lift a finger.
We are seeing first-hand how incredibly well this new model is working for us by the day!