As we ramp up the launch of our new digital twin infrastructure, GCP's BigTable is an obvious contender as a globally-accessible high-performance key-value storage fabric. How does it perform in use cases approximating our own, including our need for immediate global consistency, extreme scalability and performance, secure read/write access from any authorized VM anywhere in the world and reasonable performance even for the most geographically remote clients? Here we test a very basic scenario of a variable number of read/write workers running on a VM in London and connecting to two different single-node SSD BigTable instances: one co-located in the same zone in London and one in Ohio. To maintain simplicity and match some of our use cases, we used only serial read/writes (no batching).

The end result is that for single key existence checks and key+value reads (including range scans), the GCS JSON API vastly outperformed BigTable where the VM is geographically distant from the storage system. We did not test write performance. At the same time, BigTable has a number of feature advantages over GCS, allows for greater server-side filtration of results and, critically, allows large-scale batching that would likely vastly improve performance and which we will test in a future experiment. At the very least, the results here offer a reminder of the critical importance of benchmarking, the potential for creative repurposing (object-based storage as a high-performance key-value lookup) and the fact that cloud scale systems can often surprise in their adaptability and performance at tasks they were never intended for.

You can see the final benchmarks below:

London BigTable <-> London VM: 10K Records
25 Workers: 0m2.743s Write / 0m4.616s Read
50 Workers: 0m2.305s Write / 0m3.425s Read
100 Workers: 0m3.041s Write / 0m3.686s Read

London BigTable <-> London VM: 1M Records
25 Workers: 6m29.338s Write (39.9% BigTable Node CPU) / 2m57.373s Read (17.3% BigTable Node CPU)
50 Workers: 3m47.796s Write (61%) / 1m52.973s Read (7.4%)
100 Workers: 2m9.891s Write (62%) / 1m11.098s Read (6.7%)
200 Workers: 1m51.071s Write (63%) / 0m56.868s Read (6.2%)

Ohio BigTable <-> London VM: 10K Records
25 Workers: 1m26.593s Write / 0m44.248s Read
50 Workers: 0m44.728s Write / 0m23.296s Read
100 Workers: 0m24.463s Write / 0m13.319s Read

Ohio BigTable <-> London VM: 1M Records
50 Workers: 72m11.335s Write (7.2% BigTable Node CPU) / 35m55.026s Read (7.2% BigTable Node CPU)
100 Workers: 36m49.283s Write (7.2%) / 18m26.209s Read (13.5%)
200 Workers: 18m33.936s Write (11.1%) / 9m15.180s Read (19.4%)

We've previously demonstrated GCS' impressive performance as a key-value store. How does reading from GCS compare with reading from BigTable? In both cases we'll use a London-based VM and a bucket in the US Multiregion or BigTable instance in Ohio. Due to differences in their approaches to writes, we'll just test reads, using the example of an application that is largely read-heavy and must have exact consistency with globally-distributed VMs. For GCS, we'll test both using the gcloud CLI ("gcloud storage ls" for key reads and "gcloud storage cat" for key+value reads) for simplicity and the GCS JSON API for maximal performance. For GCS we store the columnar data in a JSON block written as an object into GCS.

As the results below demonstrate, the GCS JSON API vastly outperforms BigTable for key existence queries (a key focus of a digital twin system and which more complex columnar structures can be represented as in our use case), outperforming even range reads and full value reads. In contrast, gcloud was considerably slower than BigTable. It is important to note that we did not test writes here, only key existence and key+value reads.

Read 1K Single Records: 64 Workers (1 Per Core)
GCS GCLOUD: US Multiregion GCS <-> London VM: 0m31.976s (88% VM CPU)
GCS JSON API: US Multiregion GCS <-> London VM: 0m4.985s (22% VM CPU)
BigTable: Ohio BigTable <-> London VM: 0m22.071s (60% BigTable Node CPU)

Read 10K Single Records: 64 Workers (1 Per Core)
GCS GCLOUD: US Multiregion GCS <-> London VM: 5m39.214s (89% VM CPU)
GCS JSON API: US Multiregion GCS <-> London VM: 0m44.729s (21% VM CPU)
BigTable: Ohio BigTable <-> London VM: 3m22.141s (59% BigTable Node CPU)

Read Range Of Records & Return Keys Only (~around 1500 matching keys/objects x 1000 iterations): 64 Workers (1 Per Core)
GCS GCLOUD: US Multiregion GCS <-> London VM: 0m49.169s (80% VM CPU)
GCS JSON API: US Multiregion GCS <-> London VM: 0m4.985s (22% VM CPU)
BigTable: Ohio BigTable <-> London VM: 0m7.336s (15% BigTable Node CPU)

Read Range Of Records & Return All Fields (~around 1500 matching keys/objects x 1000 iterations): 64 Workers (1 Per Core)
GCS GCLOUD: US Multiregion GCS <-> London VM: 1m15.931s (80% VM CPU)
GCS JSON API: US Multiregion GCS <-> London VM: 0m5.395s (20% VM CPU)
BigTable: Ohio BigTable <-> London VM: 0m7.256s (15% BigTable Node CPU)

Read Range Of Records & Return All Fields (~around 1500 matching keys/objects x 10K iterations): 64 Workers (1 Per Core)
GCS JSON API: US Multiregion GCS <-> London VM: 0m45.300s (20% VM CPU)
BigTable: Ohio BigTable <-> London VM: 4m3.443s (60% BigTable Node CPU)