While nearly all GCP APIs offer library interfaces for many major programming languages like Python, there are many use cases where absolute raw performance or overhead matters, where they are being embedded in a legacy code base or where implementation simplicity and longevity is of greatest concern. In this case, calling the API directly via a CURL POST is often the preferred solution. For example, invoking the GCS JSON API directly via CURL is so efficient it can actually be used as a simplistic queuing system with latency and scalability on par with dedicated queuing servers when deployed globally. However, typically most code samples are designed to work both on GCE and on developers' local machines and thus obtain the necessary bearer token for authentication using "gcloud auth print-access-token". Switching to the GCE instance metadata server, we can obtain our bearer token in a fraction of the time and achieve a 6.8x speedup by changing just a single line of our CURL calls.

In most code samples you'll see a bearer token being obtained via:

time gcloud auth print-access-token
real 0m0.912s
user 0m0.788s
sys 0m0.102s

This has the benefit of running both within GCE VM instances and on remote development environments. It is also extremely slow, as seen above, taking nearly a full second to run. You'll typically see it being wrapped as a variable and passed directly inline in a CURL call like:

>time curl -X GET \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://storage.googleapis.com/storage/v1/b/data.gdeltproject.org/o?prefix=folder/ABC&fields=items(name)&matchGlob=folder/ABC*zip"
real 0m1.073s
user 0m0.831s
sys 0m0.121s

This allows the command to run anywhere, both within GCE and on a user's development machine. However, it is extremely slow, taking more than a second to perform this simple ls.

When running within GCE itself on a VM, there is a vastly better and faster way to get the needed bearer token:

>time curl -s "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" -H "Metadata-Flavor: Google" | jq -r '.access_token'
real    0m0.038s
user    0m0.038s
sys     0m0.013s

This returns in just 0.038s, making it suitable for high-volume latency-sensitive operations. Let's repeat our GCS ls operation from above, but this time use the metadata server instead of gcloud:

>time curl -X GET \
  -H "Authorization: Bearer $(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" -H "Metadata-Flavor: Google" | jq -r '.access_token')" \
  "https://storage.googleapis.com/storage/v1/b/data.gdeltproject.org/o?prefix=folder/ABC&fields=items(name)&matchGlob=folder/ABC*zip"
real 0m0.158s
user 0m0.062s
sys 0m0.014s

Here we can see it runs 6.8 times faster than before and actually is far more scalable for high-volume applications.