One of the most powerful capabilities enabled by our migration to GCP's Global External Load Balancer is the incredibly rich and holistic view we now have of our incoming traffic. Instead of trying to assess global-scale traffic patterns from many large distributed clusters of web servers, we can now see our entire global traffic state via the centralized view of the GLB before it is sharded out to our servers, along with a wealth of additional insights such as the geography and global routing of our traffic. We've begun exploring what new kinds of insights we can derive from all this new routing data, especially along the dimension of security and traffic management.
Browsing through our logs this morning, at first glance nothing seemed amiss: just a typical day of traffic, with no single IP address generating any measurable amount of traffic. It wasn't until we superimposed some of the GLB's additional routing information that something stood out: what appeared to be a single user distributed across 14 IP addresses systematically crawling one of our sites over 13 hours using what appears to be an agentically-controlled browser as a bulk crawler. If confirmed, this would mark the first time we've seen a legitimate browser on a home internet connection operating as a bulk crawler under the remote control of an agentic reasoning agent.
In all, we see 14 IP addresses associated with a residential ISP provider in a city in Africa that have never appeared in our logs and then appear together during a 13 hour period and then vanish again from our logs. The 14 IP addresses are scattered across the IP range of the provider, with 2 active for a period of time making steady requests, then both IPs cease requesting from our site at the same time and are replaced with two new IPs that then cease their requests at the same time and are replaced with the next two, with 7 rotations like this in all. Over the 13 hour period we see just over 6,000 requests spaced evenly but with randomized delays, averaging in total 7 requests a minute: not even noise over the given duration.
Looking at the requests in time order, we see nothing systematic about them, just a user clicking through and looking at a few items here and there. But, if we step back and take all of the requests from those 14 IP addresses over that 13 hour period and sort them by URL, we can see instantly that they are systematic, but not linearly so: the requests are not blindly requesting items in date order. Instead of simply iterating over all dates in the given range, the requests are intelligent – the requester read through the information on the page and adjusted their queries based on that information and then randomized those requests to mask their activity. The fact that requests were only made to dates with available items suggests the requester was able to "read" the information on the pages rather than the typical exhaustive date scan that a standard crawler would perform.
While the user agent claims to be a standard Chrome browser, that self-provided information is obviously entirely untrustworthy and typically forged by many crawlers. Instead, request fingerprinting tells us that two distinct browsers were used and that both fingerprints match current Chrome browser editions. Both fingerprints match only to current known legitimate Chrome editions running on Windows – specifically the editions claimed in the user-agent field. They do not match any known versions of headless Chrome or variants designed specifically for remote control. Both editions of Chrome support built-in agentic remote control via model integration.
Lending additional support, each request fetched all page resources, including those managed by JavaScript code async loading resources on the page, meaning this was not a typical automated bulk crawler, and profiling and signatures of those requests again match typical Chrome under Windows and do not match those of automatable versions of Chrome other than through agentic integration. The crawler appears to have responded to what it saw on each page, adjusting future requests accordingly, suggesting something far more advanced than a typical batch script and a few regexes.
Overall, based on a deep analysis of the full traffic and subsequent research and conversations, we are highly confident that this traffic originated via an agentically-controlled browser being repurposed as a bulk crawler in a first that we have observed on our own sites, though we've never before been able to observe our traffic at this resolution and detail. If confirmed, this poses a unique challenge for traffic management in that if legitimate actual browsers operating from residential IP addresses, rotating their IPs regularly, and keeping their traffic to an unnoticeable trickle, are now operating as crawlers, it is unclear how one might filter that traffic from legitimate human-driven traffic in order to prioritize human use: questions we will be actively exploring over the coming months.