Add GCE Signer

[dhermes]

See: https://cloud.google.com/iam/reference/rest/v1/projects.serviceAccounts/signBlob

Relevant history:
googleapis/google-cloud-python#922

[sparhomenko]

@elibixby In the last update here you mentioned that the implementation is on hold as the GCE/GKE metadata server team plans to add blob signing support. Do you know if they had any progress, or if there is a public ticket for this effort that we can subscribe to?

[theacodes]

Slight update: the ability for the GCE metadata server to sign bytes is planned, but no word on timeline right now.

[elibixby]

@sparhomenko Sorry about the delay. There was some confusion on the eng team about the use case, but that has now been resolved.

[theacodes]

#108 adds IAM-backed signing which is a temporary workaround until GCE can sign directly.

[ThatsAMorais]

It appears that IAM-Signer is exactly what I need to sign from a GKE pod; I have token credentials, but don't have access to the private key, and I would like to use the environment's default credentials to accomplish the following.... but it feels like a hack to reach into the SDK like this.

    gcp_client = client.Client()
    credentials = gcp_client._credentials
    credentials.refresh(
        # What request object, and how do I init it with the Session??
    )
    signer = iam.Signer(requests.Request, credentials, credentials.service_account_email)
    string_to_sign = ('{0}\n{1}\n{2}\n{3}\n{4}'.format(
        method, md5, content_type,
        expiration, canonicalized_resource))
    signature = signer.sign(string_to_sign)

I originally tried to use something like this, but the _credentials are a mere token when deployed to GKE. Otherwise, this worked when I specified the GOOGLE_APPLICATION_CREDENTIALS in my local environment as a path to a service account key.

query_params = get_signed_query_params(
        client.Client()._credentials, expiration,
        string_to_sign=string_to_sign)

Am I missing some way to just call google.cloud.storage.sign_this(string_to_sign) (a purely hypothetical method) that just works automatically with the credentials supplied in the environment?

[theacodes]

but it feels like a hack to reach into the SDK like this

It is a bit and it's somewhat intentional. We want you to opt into this because it's non-trivial how it works.

Am I missing some way to just call google.cloud.storage.sign_this(string_to_sign) (a purely hypothetical method) that just works automatically with the credentials supplied in the environment?

This might actually be useful (google.auth.sign) perhaps file a new issue and we'll prioritize it?

[ygbr]

any updates on this? is it possible to sign download URLs using another method or do we still need to use the workaround mentioned above?

[theacodes]

Still need to use the workaround.

…

On Mon, May 21, 2018 at 11:18 AM Ygor Lemos ***@***.***> wrote: any updates on this? is it possible to sign download URLs using another method or do we still need to use the workaround mentioned above? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#50 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc6sdgZEDJusbyj3kSnLucOlk0STMks5t0wT4gaJpZM4KfWsd> .

[JacobHayes]

I know this package isn't directly related to Google Cloud Functions, but this documentation uses a standard Compute Engine service account (explicitly enabling roles/iam.serviceAccountTokenCreator) from application default credentials to call Blob.generate_signed_url. Testing it out from a cloud function does not work (which is inline with this ticket still being open) and issues:

you need a private key to sign credentials.the credentials you are currently using <class 'google.auth.compute_engine.credentials.Credentials'> just contains a token. see https://google-cloud-python.readthedocs.io/en/latest/core/auth.html?highlight=authentication#setting-up-a-service-account for more details.

It seems the Cloud Function documentation thought google.auth.compute_engine.credentials.Credentials implemented google.auth.credentials.Signing. Could that be reasonably done or should this just be a Cloud Functions documentation fix?

[theacodes]

@JacobHayes probably, can you file a separate bug?

[jezhumble]

Here's a minimally hacky way to sign blobs with the python client in compute engine and cloud functions: https://gist.github.com/jezhumble/91051485db4462add82045ef9ac2a0ec

[MoSehsah]

Any Update on this ?

[david-wb]

Our team is dealing with this issue now. We're running on Google Cloud-Run where generating signed URLs works just fine with the Nodejs client, but not with the Python client. We would greatly appreciate if this problem was fixed without needing hacky workarounds

[jkevingutierrez]

According to constraints/iam.disableServiceAccountKeyCreation in https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints the export of service account keys is discouraged.

It would be really helpful if this issue could be fixed soon

[Capstan]

Is this issue about adding a compute metadata URL to support blob signing directly from workloads and then adapting that into the auth library?

Rather, is there any reason not to just adapt the code to add a specific call to the IAM Credentials API using the provided Auth tokens that already exist in the metadata? Is the expected problem that customers' Workload Identity service accounts might not generally have iam.serviceAccount.signBlob permission via roles/iam.serviceAccountTokenCreator or one of the *Agent roles?

[Capstan]

Leaving a breadcrumb for googleapis/google-cloud-python#922, since it was duped to #92 which was then duped to this.

[adriangb]

Another +1 for this getting resolved as soon as possible, for the many valid reasons stated above. I'll just add that I've done this before in AWS and it was seamless, I was surprised to find out all of the hoops I'm going to have to jump through in GCP 😞 .

[adriangb]

So after trying the workaround, there's one big gotcha: there are quota limits on the APIs used by the workaround (6,000 req/min or 60k req/min, I can't tell, ref). So you're limited in how many operations you can make against GCS. Which is not only a deal breaker, but even worse a deal breaker you probably won't find out about until it's too late (which is what happened to us).

[Ark-kun]

Google is now discouraging exporting the service account keys to files.

I've spent hours and days trying to sign a GCS URL without triggering the security.

This issue is 6 years old. Maybe it's time to fix it?

[salrashid123]

the only way to sign a url in gce, cloud run or cloud functions is to utilize an IAM api's signBlob() . A GCE's metadata server simply has no way to sign arbitrary bytes by itsel

Its a bit awkward to use the iam api but you need to allow a type of "self-impersonation".

i have a small writeup here about this

[jahnagoldman]

@adriangb what have you done to workaround the quota limits??

just wanted to add another +1 here to get this fixed. it seems like a pretty major issue? i can't really figure out a good workaround since the aforementioned one has quota limits, and the other solution of uploading the service account key seems pretty bad security wise, and gcp specifically calls out never to upload your service account key to cloudrun/gce/or whatever you're using.

any further suggestions / movement on this?? for someone who's using gcp, python and signed urls i don't see a good solution. other than switching to aws or something??

[jahnagoldman]

@salrashid123 doesn't that solution still use the iam api and therefore is subject to the quota of 6000 or 60000 reqs per minute? doesn't seem like a scalable option?

[astrberg]

update on this?