Over the past 24 hours since we announced that we are beginning to move our externally-facing infrastructure behind GCP's Global External Load Balancer, we've gotten a tremendous outpouring of interest in why we're making this move, what factors drove our adoption, what benefits we see already and the improvements we hope to see long-term and general observations. Most importantly, we've been asked a lot about why we chose to use GCP's GLB instead of just using DNS or building our own load balancer frontend if all we need to do is distribute requests more evenly across our servers. Today we're going to highlight just a few of the major considerations in our migration and the benefits we're already leveraging.
Security
Believe it or not, the first and foremost reason for our migration has nothing to do with scalability and everything to do with security. Driving the urgency of our transition has been the rise of AI and agentic attack workflows. To address this, we are making full use of GCP's Global External Load Balancer, HTTPS Offloading, Google Front End (GFE) Client Termination, Google-Managed SSL Certificates and Cloud Armor, to name just a few of the GCP services we are bringing to bear.
- Responding To Novel (AI + Agentic) Attack Vectors. First and foremost to our migration is the fact that agentic and AI-infused workflows are fundamentally transforming the cyber landscape. While the world is just on the cusp of this brave new world, we are already observing the early results on our own servers. Historically, we typically have seen a steady stream of the usual kinds of vulnerability probing and then waves and vertical surges of probes and attempted attacks as new vulnerabilities and exploits become known. In recent months we've begun to see a fundamentally new kind of attack: a steadily increasing stream of novel probing in which every request is different and ranges from what appear to be brute force attempts to exercise the entire codebase of the standard web serving infrastructure for any weaknesses to dynamic attacks that seem to be powered by reasoning models that are actively reacting to the results of each probe. By placing our web infrastructure behind GLB, we offload key portions of the defensive workload to GCP, especially at the challenging protocol level. Client connections terminate at the GFE and the GLB then opens a new connection to our servers and forwards a sanitized version of the request, meaning defending against many forms of HTTPS and protocol-level exploits become the responsibility of the GFE. Using Cloud Armor we can further filter a wide range of probes and attempted attacks and instantly respond with mitigations at a global fleet level as new attacks are observed and identified. This shift is allowing us to focus more of our defensive efforts on our direct stack and leaving more of the lower-level attack surface to the GFE to defend, much as we did when we switched to GCP's IAP for infrastructure.
- GFE Client Termination + Cloud Armor. In addition to responding to novel attack vectors, the combination of GFE client termination and GCP Cloud Armor allows us to filter out the obnoxious steady state traffic of attackers probing for the usual known vulnerabilities, especially buffer overflow attempts – preventing those from ever reaching our servers and bogging them down with useless traffic that can consume valuable RAM and CPU time.
- HTTPS Offloading. The vast majority of website admins never think about all of the CPU time taken up in handling the "S" in HTTPS. The cryptological work of connection setup and encrypting and decrypting traffic begins accounting for a not insignificant amount of server resources as you reach levels of scale. The GLB allows us to offload all of that to GCP's Google Front End (GFE) edge servers. All client connections terminate at GFE edge servers, which then communicate with our query servers using HTTP over GCP's internal networking and then repackage and reencrypt the results for transmission to the user. This allows us to push the entire HTTPS workload to GCP's edge servers.
- Cert Management. While it would seem that keeping a web server's SSL certificate up-to-date is a solved problem in today's web, the reality is that strange things happen at scale in the real world. Over the years we've used both commercial and free cert vendors, including some of the web's largest and most established providers. The majority of times, automated renewal and updating workflows work as intended: certificates are renewed, the necessary cert files issued by the provider, the files downloaded and verified and the certs installed to the server and all is well. However, as any server administrator knows, in real life, nothing works perfectly all the time, and the sheer variety of failures we've seen, from vendors failing to renew certs on schedule, to vendors issuing expired certs, to vendors issuing corrupted certs, to having to spend days helping a major certificate provider debug a rare edge case in their systems that prevented our certs from renewing, to the most bizarre server errors to myriad other problems that you won't find in any administrator textbook. With the transition to GCP's GLB, we are making use of the GLB's seamless integration with GCP's certificate management infrastructure. Instead of managing our own certificates, we are now using Google-managed SSL certificates, which the GCP seamlessly renews and installs on the GLBs for us without us ever having to think about them anymore.
- Patching & Upgrading. The administrative work of constantly patching and upgrading all of the myriad libraries, toolkits, plugins, integrations and other connection points for all of the various cryptography libraries required to support HTTPS is far from insignificant. This is compounded by the fact that upgrades to these libraries don't always go smoothly and can require complex manual interventions and upgrade workflows when a breaking zero-day security patch fails to compile against a particular library that a critical server daemon uses. Responding to zero-day mitigations and patches immediately, regardless of the time of day, adds an additional toll. Adopting GLB's HTTPS offloading allows us to push all of this to Google's own GFE servers.
- Supporting The Latest HTTPS Options. In parallel to maintaining the latest patches and security mitigations is the need to constantly upgrade to support the latest HTTPS standards and options. By offloading the HTTPS process to the GFE's, we are able to benefit from Google's relentless advances in supporting (and often creating or improving) the latest HTTPS standards and approaches, including even post-quantum TLS!
Scalability & Traffic Management
Just behind security and related in many ways to it, is our need to manage and shape our ever-increasing incoming traffic, deal with a rapidly growing volume of poorly and maliciously behaved clients and, in this brave new AI era, gracefully handle sudden massive onslaughts of agentic swarms.
- Traffic Management. As users of our APIs and web sites know, over the past few years we have been increasingly struggling to respond to the exponential growth in our traffic, especially the rise of sudden and often massive traffic surges. We operate our own load balancing and traffic management layers, including advanced dynamic load shedding infrastructure that dynamically shapes and rejects traffic by combining current backend server load (ElasticSearch, databases, caches, etc) with an on-the-fly estimation of the projected cost of each incoming query by itself and combined with the current queue. While this infrastructure long allowed us to manage our traffic reasonably well, the sheer traffic volumes of the modern (especially agentic) web has made it increasingly difficult to manage all of this traffic entirely ourselves. In particular, it takes time to scale hardware – a sudden vertical surge of millions or even tens of millions of additional queries per second that arrives out of nowhere in a single second cannot be readily absorbed by autoscaling systems that take time to spin up additional hardware. Today we handle load balancing ourselves, meaning that if a sudden surge of 25M queries arrives in a single second, our servers cannot actually absorb that traffic in order to decide what to accept and what to reject and so all of our traffic, including all of the legitimate traffic, either times out or is rejected. By offloading traffic management to the GLB, all incoming client traffic arrives at a Google-scale GFE that can handle effectively unlimited traffic. That GFE, in conjunction with Cloud Armor, autoscaling Managed Instance Groups, our prioritization rules and other workflows, will be able to properly sort and prioritize that incoming traffic. In this way a vertical surge of 100M requests in a single second can be trivially absorbed by the GFE and trimmed, throttled and rejected before our servers ever see a single request, ensuring that legitimate user requests continue unimpeded.
- Agentic Swarms. In the past, massive sudden vertical traffic surges tended to be either DDOS attacks or an academic researcher firing up a campus computing cluster to run a large analysis using our data and either misconfiguring their software or not realizing the consequences of the options they selected. We've long gotten a fairly steady daily stream of especially academic users with large academic clusters that would fire up thousands of machines each opening tens of thousands of parallel connections. These typically were relatively easy to mitigate simply by throttling that cluster's IP range and reaching out to the academics in question to suggest a better approach to their downloading. However, over the past few years we've been seeing a rapid increase in agentic swarm traffic in which a user will commission an agentic AI system to perform an analysis on our data and that agentic system will open thousands, tens of thousands and sometimes even hundreds of thousands of connections to our servers in parallel. Unlike academic users on a single cluster, these agentic systems tend to be fully distributed, with each request coming from a different IP address, making it nearly impossible to isolate and shed the traffic. Worse, the growing use of self-hosted agentic clusters and so-called "computer control" models that can command legitimate browsers across huge numbers of machines in parallel have made it increasingly difficult to isolate this traffic from human traffic. While we have some experimental self-learning systems that are able to achieve reasonable accuracy at separating agentic and human traffic for several of our endpoints, they are of little use if the servers cannot actually absorb the incoming traffic in order to reject or accept it. By offloading all of the client connections to the GFEs, we can dynamically manage and prioritize traffic using Cloud Armor, as well as setting hard limits that protect our servers from the most extreme traffic surges.
- Cloud CDN. We are especially excited about the ability to enable Cloud CDN global edge caching on a per-service basis with a single click. For each service, we can now allow Google's globally distributed edge CDN network to cache frequently-requested content geographically proximate to each user and vastly reduce the traffic on our servers. We can even enable automatic CDN compression of formats like JSON and CSS, offloading all of the compression CPU to the edge from our servers.
- Load Balancing. As their name implies, Global Load Balancers support highly sophisticated advanced traffic management and load balancing. Coupled with GCP's autoscaling Managed Instance Groups and the native autoscaling abilities of services like Spanner, BigTable, etc, this means that as we transition more and more of our infrastructure behind the GLBs, we'll be able to seamlessly scale to absorb traffic vastly more efficiently than we could ever do on our own.
- Connection Pooling. Establishing, maintaining and tearing down an HTTP/S connection to a remote client takes resources. While a single server can theoretically handle many millions of clients at once, the reality is that all of the CPU and RAM, kernel context switching, and other overhead becomes a not insignificant limiting factor at scale. Instead, by using GFE edge termination, Google's GFE servers handle all of the connection management and so instead of having to manage millions of simultaneous connections, our servers now only have to manage a small number of keepalive connections with the GLB servers.
- Poorly Behaved Clients. We're not quite sure why, but we've seen exponential growth in recent years of poorly behaved clients, from broken protocol implementations to all actions occurring just below timeout levels to download speeds of bytes per second to using hundreds and even thousands of connections, to attempted use of byte ranges of tens of bytes per request to a huge range of other problems. We've seen near-vertical growth of these problems coinciding with the rise of agentic AI in particular. The GFEs automatically reject certain kinds of malformed requests and using Cloud Armor we can filter out many of these poorly behaved requests.
- Slow Clients. Slow clients are a particular type of poorly behaved client that require special consideration and handling. Given GDELT's focus, we have a large userbase in developing countries with extremely limited connectivity, NGO and other fieldworkers using limited share satellite connectivity, users in repressive regimes conducting state telecommunications throttling, filtering and interruption, mobile users on the go tower-hopping and just ordinary slow or overwhelmed internet connections like shared links. These are simply ordinary users whose connections are extremely slow due to no fault of their own, but those slow connections eat up significant resources for our servers. By using GFE Client Termination, the user's slow request is buffered by the GFE until completion, then sent to our servers, which can respond instantly with the entire request and move on to the next user, while the GFE then slowly streams the results back to the client.
- Bad & Malicious Clients. Poorly behaved and slow clients are problematic, but far more dangerous are malicious clients that are actively attempting to attack our endpoints. As mentioned earlier, the rise of agentic AI has seen a near-vertical increase in these clients and the ever-increasing creativity and volume of their attempted attacks. GLB's GFE Client Termination and Cloud Armor means that an entire swath of these attacks are now rejected directly at the GLB level before ever reaching our servers.
Advanced Routing & Visibility
Finally, the Global Load Balancer's advanced routing capabilities are opening a wealth of new opportunities for us, as are the new visibility metrics we have into our global traffic, from IPv4 vs IPv6, to routing and origin geography.
- Routing. Historically, routing traffic amongst our many endpoints, datasets and websites required myriad solutions glued together, joining DNS, HTTP redirects, server-side proxying and a range of other approaches. With the GLB, all of our routing needs are now natively supported by the GLB itself. This centralizes all our routing in a single place from its previously "decentralized and distributed in many conflicting places" approach and, most importantly, means we can now route traffic dynamically, both in respond to changing load volume and composition and for A/B and canary testing of new rollouts. When rolling out an upgraded endpoint, instead of handling traffic scaleup ourselves, we can simply adjust a single number in our GLB routing table to gradually transition traffic to the new endpoint, with the ability to instantly shift it back to the original if problems arise. The ability to dynamically route traffic even at the URL component level opens exciting new doors for both traffic management and the rollout of new capabilities.
- Visibility. The GLB brings with it powerful new visibility tools for understanding the global flow of traffic through all our external surfaces. While the GLB already routes client requests through the geographically nearest POP and to Google's global private fiber network for maximum performance and the CDN caches content in Google's edge datacenters around the world, dynamic queries like searches and API requests must still transit across the globe limited at the speed of light, incurring significant latency for certain geographies. The GLB is already helping us understand the user geography of each of our services with an eye towards future capacity planning of where we may place additional service replicas down the road to cut latency for regions with high usage densities of specific services.