It has been two and a half weeks since the VGKG completed its move onto GDELT's new GEN4 architecture and the results have been nothing short of extraordinary. Despite breaking our previous inlined architecture into a pipeline of separate process images, aggregate throughput has increased dramatically and hardware utilization efficiency has also increased, largely due to the shift from application-level task scheduling and interleaving to kernel-level resource management. Our new hypervisor architecture has also dramatically increased our ability to fully utilize the underlying hardware platform of each VM. Finally, the isolation of all networking components within separate C-only process images allows us to avoid the significant overhead of library binding interfaces and reduce networking-related CPU by 99%, freeing nearly the entirety of the underlying VM hardware execution thread for processing, rather than networking tasks.

The end result is a massively more scalable and robust crawling architecture that has outperformed even our most optimistic projections for the GEN4 architecture at production scale.