Metadata check intermittently fails resulting in badly formatted logs

[dansiviter]

Expected Behavior

GoogleCloudPropertyDefiner correctly identifies it's running in GCP every time resulting in google_cloud_logging being set consistently.

Actual Behaviour

GoogleCloudPropertyDefiner intermittently fails to detect it's running in GCP when running in GKE with Istio/ASM. I believe this is due to the Istio sidecar having a small delay being bound into the mesh (see kubernetes/kubernetes#65502) which means all requests fail until it is ready.

Steps To Reproduce

1. Deploy service using GKE with Istio/ASM with Dynamic appender selection enabled,
2. Restart Pod Several times.
3. Observe that not all of the Pods will have JSON formatting.

Environment Information

• GKE
• Istio or Anthos Service Mesh

Example Application

No response

Version

micronaut-gcp v3.5.0

[viniciusccarvalho]

@dansiviter If I understood this correctly, the problem is that we detect gcp environment by trying to call the metadata endpoint of gcp, and the istio side car is returning an error (4xx/5xx) and then the request fails?

We could probably fix this by adding some sort of retry mechanism, having the default to fail immediately (to avoid causing latency startup on the service), and make it configurable. I'm not sure how long would take the proxy to be ready.