As automated bot traffic continues to grow exponentially almost daily with the rise of agentic AI, one of the most commonly recommended countermeasures is to use JavaScript to render or gatekeep page contents. Historically, most homebrewed crawling bots were little more than dumb Python scripts that expected a complete HTML page to be returned and thus adding a bit of JavaScript to the page to render the contents or redirect the user to the real page served as an easy and reasonably effective mitigation. Advanced agentic AI bots represent a brave new world in which modern bots tend to run highly advanced full-fledged browsers that can render JavaScript and take many actions that look extremely similar to a human user.
As just one glimpse into the scale of the problem, in just one 9 day period, one of our sites received more than 3M requests strongly indicative of automated agentic traffic, but where the bot's processing of advanced instrumentation JavaScript was indistinguishable from a legitimate human-controlled browser. Amazingly, a single GCE VM was able to handle all of this traffic due to the GCP Load Balancer absorbing all of this traffic and serving it from the GCP CDN global edge cache, but as we move towards relaunching our various APIs and search interfaces, we are having to fundamentally rethink what handling traffic in the agentic era means.