GDELT 3.0: HTTP 451 Errors Completely Disappear

In response to the rise in HTTP 451 errors we've observed since the implementation of GDPR, we've completely reimagined how we think about geography in our global crawler infrastructure, rebuilding our GDELT 3.0 fleet queuing and dispatch systems to be geographically aware. Prototype implementations suggested these changes would completely eliminate these errors and with the final cutover to the new production infrastructure yesterday, in the 24 hours since we've seen just four HTTP 451 errors as the system learns from a few very low volume outlets that hadn't published anything in the last few days or don't publish on weekends. To date 575 domains have been given geographic routing assignments by the infrastructure, meaning they are diverted away from a region that generates 451 errors for them. This entire process is automatic in which the system automatically retries URLs across the world, mapping domain error rates by geography and learning optimal placement strategies for each individual domain. We're tremendously excited about this new architecture and its possibilities as we look to the future and increasingly interventionist governments across the world.