Add PKCS12 Keystore support for OPA server client cert, CA cert and private key files

[mariuspodean]

What is the underlying problem you're trying to solve?

In some scenarios it is not feasible to have stored on disk unprotected, sensitive (m)TLS files, such as the server certificate and its private key. Using a password protected keystore improves security in this situation.

Currently, in order to use (m)TLS, we need to pass as params the paths for the following files: server cert & private key and the CA cert. While there is no problem for having the CA cert on disk, there is a different matter when it comes to the certificate itself and especially for the private key. This issue is mainly present on setups that don't use (service) containers and OPA is installed alongside other pieces of software.

Keystores provide a secure way to store this sensitive information, making sure that only authorized entities can access and use the stored keys and certificates.

Describe the ideal solution

Ideally, instead of passing three params to identify the mTLS required 3 files (cert, private key and CA cert), we could pass the path where the keystore is stored and the password to access the data within it.

So instead of using:

...
--tls-cert-file cert.pem \
--tls-private-key-file key.key  \
--tls-ca-cert-file ca.pem \
...

one could use something like the following:

...
--tls-keystore-file keystore.p12 \
--tls-keystore-pass secret \
...

As not everyone needs this level of control, probably both methods for configuring TLS should be in place, leaving to the user to choose.

Ideally we would use PKCS12 keystores, but, if for some reasons that's not possible, JKS would be OK too.
PKCS12 is a standardized and language-neutral way of storing encrypted private keys and certificates, and one can use something like the following to bundle together in a secured way the resources:

openssl pkcs12 -export \
    -in cert.pem \
    -inkey cert_key.key \
    -chain \
    -CAfile CA.pem \
    -out keystore_name.p12 \
    -passout keystore_pass \
    -name resources_name_in_keystore

Describe a "Good Enough" solution

The goal here is to protect sensitive resources (server cert & key) and the best way to do it is through the use of keystores. If that is not something that's on the scope for OPA (although one might argue that such a component should use "all means" to secure itself), allowing password protected keys would also be a step up.

Password protected keys would require that we pass an additional param, the key's password. This approach allows us to secure one resource (the certs private key), but leaves the server cert exposed.

Additional Context

All this might not make sense for some setups were we ca use other infra means to protect these resources, but please bear in mind that that's not always possible.

[ashutosh-narkar]

Thanks for filing this detailed issue.

The goal here is to protect sensitive resources (server cert & key) and the best way to do it is through the use of keystores

The scenario here is an attacker has gotten access to the system where OPA is deployed and can now access the files on that system. Is this correct?

On the keystores itself, can you point to examples of other open-source projects where this has been utilized. Also this would require adding a 3rd-party dep iiuc.

[mariuspodean]

The scenario here is an attacker has gotten access to the system where OPA is deployed and can now access the files on that system. Is this correct?

Yes, that would be one of the cases, albeit the most extreme one. Another one would be a machine (Windows ) that hosts other pieces of software, and most importantly, multiple users can have access to it, and you don't fully trust them with this kind of data. Unfortunately, this scenario is more common than one would expect ...

On the keystores itself, can you point to examples of other open-source projects where this has been utilized.

Sure, a lot of projects rely on keystores, one of them being Keycloak (a Cloud Native Computing Foundation incubation project for identity and access management).

Keycloak supports both methods of specifying certs & keys, one could use the files directly:

...
--https-certificate-file=/path/to/certfile.pem \
--https-certificate-key-file=/path/to/keyfile.pem
...

or, by using a keystore:

...
--https-key-store-file=keystore.p12
--https-key-store-password=password
...

Moreover, they allow also to specify the CA chain (or other trusted certs) using truststores (essentially keystores that only store certs, that being the main difference between the two):

...
--https-trust-store-file=truststore.jks
--https-trust-store-password=password
...

Funny thing is that they planed on deprecating the truststore support as mentioned above, but in the latest release (which was out a few days ago), they decided that:
Based on the community feedback, we decided to undeprecate https-trust-store-* options to allow better granularity in trusted certificates. source

This comes to prove that these features are more wildly adopted "in the wild" than anticipated.

Also this would require adding a 3rd-party dep iiuc.

As far as I know, this package supports reading PKCS12 keystores: x/crypto. This is not standard library but it extends the crypto.

https://github.com/SSLMate/go-pkcs12/blob/fa70679f0f1622a2705336a97225ee8d6c555f96/pkcs12.go#L413

Checking the Snyk page for it, it seems pretty safe https://snyk.io/advisor/golang/software.sslmate.com/src/go-pkcs12, but yes, it is an external package. For JKS (Java keystores), I thing that's only achievable through a 3rd-party dep, but I'm no Go expert, don't quote me on this :)
Between the two formats, I believe PKCS12 is the desired one as it is standardized and language-neutral.

[ashutosh-narkar]

Another one would be a machine (Windows ) that hosts other pieces of software, and most importantly, multiple users can have access to it, and you don't fully trust them with this kind of data. Unfortunately, this scenario is more common than one would expect ...

In this case, shouldn't it be the admin of the machine that should control access. I could imagine if there are other software components in addition to OPA, they may use files with sensitive info as well. What I'm trying to get at is this something that needs to be supported in OPA or can be achieved with some integration or can be addressed in how OPA is deployed.

[mariuspodean]

If you are aware of any integration that can help us load the files securely (in the specified setup, not containers), please let me know.

I believe that if a piece of software requires some sensitive data in order to work, it should also support the means to load them securely, like through the use of keystores (in this case), or maybe pull them from (a) vault. Otherwise it seems like we're violating the principles of zero trust security. There are multiple ways of loading secrets securely, like keystores, vaults, encrypted files using SOPS etc., and thus not having to rely on what the admin of the machine specifies as access policies for the security of a software.

I strongly believe that this should be supported by OPA because a software that determines to what resources one can have access to (basically authorization) should be as secure as possible.

[anderseknert]

Hi there! And thanks for filing this request.

Supporting keystores in OPA might be a good idea simply for the sake of supporting what users already have and are used to working with. I don't think this does much in terms of security though.

If you have untrusted users with access to the same machine that's running OPA, what would stop them from just running ps aux (or whatever the Windows-equivalent command might be) and get the password from there? There must be dozens of ways for someone with access to a host to retrieve the parameters passed to programs, both currently and in the past. Same thing applies if the password is provided via a file, env var or whatever.

The cert-manager project provides an informative summary on why they don't consider this a security feature: https://cert-manager.io/docs/faq/#why-are-passwords-on-jks-or-pkcs12-files-not-helpful

I'm inclined to agree.

Otherwise it seems like we're violating the principles of zero trust security.

I'd say that ship sailed long ago if untrusted users are allowed anywhere security matters ;)

[mariuspodean]

Hi @anderseknert, thanks for your feedback.

I'd say that ship sailed long ago if untrusted users are allowed anywhere security matters ;)

I couldn't agree more ... in the scenario that I'm describing we're aiming more on mitigating risk with all the tools that we have at hand, rather implementing best practices ... unfortunately, but I guess that's how it goes all the time when dealing with legacy systems.

The cert-manager project provides an informative summary on why they don't consider this a security feature: https://cert-manager.io/docs/faq/#why-are-passwords-on-jks-or-pkcs12-files-not-helpful

I'm inclined to agree.

While I do generally agree with the main point, I don't fully agree with some of the arguments backing up this perspective:

• passwords don't need to be loaded directly from plain text, one can use SOPS, Vault etc.,
• the choice of encryption algorithms and passwords don't need to default to the weakest form (not saying that it doesn't occasionally do), or
• that the keystores need to reside on the same machine with the files that it also contains.

Indeed, you are changing one problem, managing the private key, with another one, managing a new password, but I would argue that between the two, the passwords can be secured easier than a key.

For example, if I would use SOPS and keep the cert & key in an encrypted file, when I start OPA I'm forced to write those to disk, whereas with a keystore password, I could load that password in a process specific env var.

If you have untrusted users with access to the same machine that's running OPA, what would stop them from just running ps aux (or whatever the Windows-equivalent command might be) and get the password from there?

True! Again, this is simply about risk mitigation and raising the bar through all the means available.

[charlieegan3]

Hi @mariuspodean & @alexrohozneanu, thanks for raising the issue and continuing the discussion here. The issue has come to discuss a few different topics:

• Security of environment variables vs files on windows systems
• Where secrets are loaded from at runtime (disk, env var, Vault, SOPS etc.)
• Why OPA needs to have a secure identity in the first place
• What other CNCF projects do in this area

I will try to share my thoughts on each of these matters below.

Regarding the security of environment variables vs files on windows systems. It appears that the situation is much the same as it is on Linux, where access to files and environment variables can be controlled with permissions. If a bad actor were to gain access to a machine running another program, then these permissions would be the only thing stopping them in both cases.

Users of OPA are encouraged to use short-lived identities for OPA instances. This is so that if the identity of an OPA instance where to be exposed, then the time a bad actor had to use it would be reduced.

While your proposal would support the reloading of the keystore itself, the password could not be rotated without the need to restart the instance. My fear is that the challenge that this poses to operators might encourage them to run OPA in a less secure way by rotating the password less often.

Also note that if a bad actor is able to change the loaded policy, then opa.runtime() will also contain this environment variable.

Regarding the matter of being able to load secrets from elsewhere, such as vault or SOPS. I would say that this is really the responsibility of the orchestration tooling that is running OPA, rather than OPA itself. Primitives like in memory volumes, environment variables etc are used to abstract away the underlying infrastructure to programs like OPA where the process of loading secrets is not the primary concern.

Regarding the need for OPA to have a secure identity in the first place, specifically:

I strongly believe that this should be supported by OPA because a software that determines to what resources one can have access to (basically authorization) should be as secure as possible.

OPA is a security tool, it has a means of securely identifying itself using mTLS. However, arguably OPA is no different from other applications in this regard. All applications should have a means of proving their identity to callers, and the mechanisms by which they get the identities to do so with is the responsibility of the underlying orchestration tooling. Identity is managed in very different ways in different organisations, and so this is where the primitives like in memory volumes, environment variables come in again, to abstract these differences away from generic tools like OPA.

Regarding the point about other CNCF tooling supporting keystores. While keycloak do support this feature, here are a list of graduated projects that do not, to my knowledge:

• Kubernetes API server
• Prometheus
• Envoy
• Vitess
• Jaeger
• ...

My understanding as to why it's supported in keycloak is rather related to keycloak being a java application where keystores are more commonly used, rather than it being a security tool with a higher bar for security.

In summary, my view is that keystores are no more secure than the existing options available to in OPA. Depending on how often the password was rotated, I might argue that it makes an OPA deployment less secure. However, I can see that for your use case, using a keystore might be more convenient.

---

I would like to ask a few clarifying questions:

that the keystores need to reside on the same machine with the files that it also contains.

Can you elaborate here - is this suggesting that the keystore is being loaded from a remote location?

Does the challenge of restarting OPA in order to use a new keystore password not pose operational challenges for you?

Have you considered running OPA behind a webserver that supports keystores?

[stale]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.

[charlieegan3]

Hey, @mariuspodean & @alexrohozneanu. Have you had a chance to think about the above and review if this is still something you're interested in?

[alexrohozneanu]

Hi @charlieegan3, thanks for your input!

We are still interested in supporting keystores in OPA. We are currently looking for workarounds for this, but ideally for us would be OPA to support the load of keystores.

Hereunder my answers to your questions:

Can you elaborate here - is this suggesting that the keystore is being loaded from a remote location?

Not in our case for the moment, but ideally this is what we want in the future. Although this is not our priority at the moment, implementing at least local keystore support in OPA would open the doors for future improvements like fetching passwords or keystores from a remote source (Vault).

Does the challenge of restarting OPA in order to use a new keystore password not pose operational challenges for you?

Not that much, I mean is not the ideal solution, but is something that we can compromise for the gaining of keystore support.

Have you considered running OPA behind a webserver that supports keystores?

Yes we considered this, but at the moment we are not able to make any architectural changes in our infrastructure and running OPA behind a web server would complicate too much our environment.

[charlieegan3]

Ok, thanks for getting back to me here. I'm still really hesitant about the wider value for this feature but we can keep the issue open for some time to see if others with similar needs come by here.

Regarding this:

Not in our case for the moment, but ideally this is what we want in the future.

How would OPA get access to the keystore data over a mutually trusted channel? Presumably OPA, or the glue downloading the keystore would need to attest that it was downloading the data from a trusted source? Similarly, how would the server ensure it was sending the keystore file to the correct OPA workload?

Are there any patterns for the remote loading of keystore files?