Implement System.Security.Cryptography.Native.Android PAL

[steveisok]

Since we're not able to ship Openssl w/ Android, we need to implement as much as we can with the Android API's.

Implementation

• Ensure all tests pass Ensure Tests Pass Using System.Security.Cryptography.Native.Android PAL #45740
• CI build + test
  • Test runs have been running out of memory in runtime-staging builds - haven't been able to repro locally. Emulator used in CI for x64:
    • Build ID: QPP6.190730.005.B1
    • Image: system-images;android-29;default;x86_64
    • Command line arguments: -avd <emulatorName> -memory 3072 -wipe-data -delay-adb -skip-adb-auth -no-boot-anim -no-window -no-audio -gpu swiftshader_indirect -logcat-output '/tmp/<emulatorName>-logcat.log'

System.Security.Cryptogaphy

• Fix reading of PEM certificate bytes where the certificate label is not the first bytes.
  • Seems like Android expects the label to be the first thing (doesn't handle explanatory text or whitespace before the label)

System.Net.Security

• Basic SSL stream implementation - [AndroidCrypto] Basic SSL stream implementation #50519
• SSL configuration options
  • SSL protocols - [AndroidCrypto] Handle setting non-default SslProtocols #50987
  • Application protocols - [AndroidCrypto] Handle setting non-default application protocols #51187
    • HttpClient sets ApplicationProtocols only if intending to use HTTP/2+
  • Cipher suites
    • CipherSuitesPolicy throws PNSE on Windows, so this is nice to have, but not critical
  • Encryption policy - [AndroidCrypto] Handle setting non-default SslProtocols #50987
  • Request client cert (server mode with RemoteCertRequired) - [AndroidCrypto] Handle setting non-default application protocols #51187
• Implement SslStreamPal.QueryContextChannelBinding - [AndroidCrypto] Fix handling of no peer certificates #51316
• Implement CertificateValidationPal.GetRequestCertificateAuthorities
  • https://developer.android.com/reference/javax/net/ssl/SSLSession#getLocalCertificates()
• Automated tests
  • Many existing innerloop e2e scenario tests rely on non-system-default certificate validation, which there isn't a good way for us to support (see 'Not supported' section below)
  • Test app could be created such that it bundles in a network security configuration file and test root certificates: https://developer.android.com/training/articles/security-config

Not supported

The following are not supported on Android. They should be documented, marked with UnsupportedOSPlatform, and/or throw PlatformNotSupportedException as appropriate.

System.Security.Cryptogaphy

• RC2 algorithm
  • CreateDecryptor and CreateEncryptor currently throw PNSE, Create does not
• IncrementalHash.GetCurrentHash using HMAC
• Ignoring verification during chain building that can't be bypassed on Android: AllowUnknownCertificateAuthority, IgnoreInvalidName, IgnoreInvalidPolicy, IgnoreTimeNotValid
  • X509Chain.Build will return false and have PartialChain status with no certificates
• AIA fetching during chain building
• Revocation checking options - these intentionally do not throw and are treated as supported options instead
  • X509RevocationMode: Offline treated as Online
  • X509RevocationFlag: EntireChain treated as ExcludeRoot
• Custom certificate stores (should be possible to do as a future improvement if desired)
• Revocation checking via OCSP on versions of Android without PKIXRevocationChecker (API level <24)

System.Net.Security

• Non-system-default certificate validation
  • Any RemoteCertificateValidationCallback will only get an opportunity to validate certificates that have already been accepted by the system's built-in trust manager. - resolved in .NET 8
  • This means that the use cases of the callback for self-signed certificates or custom trust will not work. - resolved in .NET 8
• Authenticating with EncryptionPolicy.NoEncryption (on SslClientAuthenticationOptions.EncryptionPolicy or SslServerAuthenticationOptions.EncryptionPolicy)
• Certain protocols (on HttpClientHandler.SslProtocols, SslClientAuthenticationOptions.EnabledSslProtocols, or SslServerAuthenticationOptions.EnabledSslProtocols):
  • SslProtocols.Ssl2 - not supported
  • SslProtocols.Ssl3 - not supported
  • SslProtocols.Tls13 - only on some versions of Android (>= API level 29)
• Underscores in host name (SslClientAuthenticationOptions.TargetHost)

Future improvements

The following should be considered after the initial implementation. If it is determined they are desired, they can be split into their own issues.

• Better error messages
  • All exceptions are just dumped to the Android error log and not propagated to the calling API, so errors just come through as a generic CryptographicException.
  • Some concept of storing/getting/clearing last exception at the shim API layer could be introduced in order to get actual error messages

System.Security.Cryptogaphy

• Use Java Pbkdf2 implementations when on available API levels.
• Use Java PBE implementations when on available API levels.
• Use Java RSA signature APIs directly when processing data (not hashes). Requires minimum API level 23 (where all padding options are supported in Java) to be worth the cost
• Custom certificate stores
  • Use some well-known location for loading/storing a KeyStore
• Improve chain error status on versions of Android without CertPathValidatorException.getReason (API level <24)
  • Try checking all the wrapped exceptions for specific CertificateException types to determine a better status

System.Net.Security

• Investigate using JNI NIO support for byte buffers in SSL stream implementation - potential perf improvement
• Investigate handling non-system-default certificate validation
  • Would require implementing a TrustManager. Android doesn't allow defining a class via JNI, so doing this would involve actually creating/shipping a Java class.
  • Possible issues around Android store validation rejecting applications with trust managers deemed insecure
  • If allowed certificates are known ahead of time, it would be possible to load/create KeyStore and use TrustManagerFactory to create and initialize a trust manager with it.

Clean-up

• Remove OpenSSL as a dependency in the build and all branching based on AndroidCrypto vs. OpenSSL on Android
• Rename native library:

  runtime/src/libraries/Native/Unix/System.Security.Cryptography.Native.Android/CMakeLists.txt

  Lines 45 to 47 in f64246c

  	# TODO: Use "System.Security.Cryptography.Native.Android" name (will require a lot of csproj changes here and there)
  	set_target_properties(System.Security.Cryptography.Native.Android PROPERTIES OUTPUT_NAME "System.Security.Cryptography.Native.OpenSsl")
  	set_target_properties(System.Security.Cryptography.Native.Android-Static PROPERTIES OUTPUT_NAME "System.Security.Cryptography.Native.OpenSsl")

• Cleaner split of Android vs. OpenSSL implementations for ciphers / hashes / HMAC.
  • With the original plan to just implement P/Invokes, the managed side of these were shared. Making a clean split and reworking the shim APIs to align more with Android would simplify things and reduce a bunch of interop calls.

Other

• Implement pinvokes needed to perform a simple https request - Perform simple https request with System.Security.Cryptography.Native.Android PAL #45739
• Tracking status of pinvokes implemented.

[JamesNK]

• Non-system-default certificate validation

  • Any RemoteCertificateValidationCallback will only get an opportunity to validate certificates that have already been accepted by the system's built-in trust manager.
  • This means that the use cases of the callback for self-signed certificates or custom trust will not work.

This makes testing against a local ASP.NET Core web app that is using a self-signed development certificate difficult.

Couldn't this be supported by creating a custom SSLSocketFactory? If RemoteCertificateValidationCallback is configured then there is a factory that trusts all certificates. Cert details are then passed to the callback so it can make a decision about whether the cert is allowed or not.

SO answer about it - https://stackoverflow.com/a/6378872
Blog post - http://howardism.org/Technical/Java/SelfSignedCerts.html

[steveisok]

@JamesNK we made the decision early on to make this a purely JNI integration. Android doesn't support creating Java classes out of JNI, which I believe makes a custom trust manager not possible.

This is something we could look at trying to support in net7.

[JamesNK]

There are a number of options suggested in the SO thread that don't involve creating new Java classes. However, I don't know Java and Android HTTP APIs so I don't know what approaches are valid.

[Eilon]

@JamesNK said:

This makes testing against a local ASP.NET Core web app that is using a self-signed development certificate difficult.

I started a discussion topic on how to connect from Android emulators to a local ASP.NET Web API running on Windows: dotnet/maui#8131

Please check that out and let us know if you have any feedback on any of the solutions presented.

[rgroenewoudt]

Using SSLStream or Websockets on Android < 9 is also currently broken, which is a breaking change for people coming from the classic Xamarin

Using HTTPClient SSL also doesn't work in Android 5-6, which is also a breaking change.

[awakecoding]

Any update on this? We are hard-blocked on this to migrate RDM Android from Xamarin to .NET 6 since it breaks SQL Server connections, and it's unfortunately a feature that many of our users consider critical. What can we do to help expedite the process? While I don't particularly like the idea, would it be possible to work around the issue temporarily by building the .NET runtime from source with the changes from this branch? Is there enough implemented to fix the SQL server connection issue?

[steveisok]

Any update on this?

For dotnet/SqlClient#1656, I suspect #78849 would fix things for you. Unfortunately, that fix won't be available until the February update of .NET 7 and .NET 6.

You could build from source yourself, but I think it'll be rather painful updating all the places you need to. There isn't any real guidance on how to do that.

[jkoritzinsky]

Marking the remainder of this issue as Future like #45737

[dotMorten]

I'm hitting an issue with being able to use Android's Certificate picker (KeyChain.ChoosePrivateKeyAlias), and reusing the certificate with SocketsHttpHandler. Since installed certificates can't export the private key, I can't really convert it to a .NET certificate.
With the old Android client handlers, you would override ConfigureCustomSSLSocketFactory and returning the SSLSocketFactory to use the chosen certificate.
Would this issue also address this need?

[borrrden]

Has anything regarding this been done in .NET 8? Specifically I need to know if I can use a custom TLS certificate verification callback yet. In .NET 6 / 7 the only way to accomplish this is by having a system trusted certificate setup ahead of time (usually via network_security.config or whatever it is called), but in a P2P model this fails because often certs are generated on the fly purely to get the encryption portion. This means they are rejected without any remedy to explicitly trust them programmatically (the callback is simply not called and the certs are marked as untrusted)

Wait reading back I see

Non-system-default certificate validation

Any RemoteCertificateValidationCallback will only get an opportunity to validate certificates that have already been accepted by the system's built-in trust manager.
This means that the use cases of the callback for self-signed certificates or custom trust will not work.

So this is never going to be supported?

[CedreLo]

Any news on this ?

[simonrozsival]

@borrrden @CedreLo hey! we shipped support for RemoteCertificateValidationCallback on Android in .NET 8 (#77386). You should also be able to use HttpClientHandler. ServerCertificateCustomValidationCallback and SocketsHttpHandler.SslOptions.RemoteCertificateValidationCallback.

I'll update this issue to reflect this, sorry for the confusion. Let me know if you run into any issues with the custom certificate validation.

[CedreLo]

Great news!

as soon as visual studio will support ..NET8, i will switch to .NET8

[borrrden]

To clarify, will this also work with SslStream using the validation callback passed into the constructor? That is the use case we are currently using. The linked issue indicates yes.

[simonrozsival]

@borrrden yes, SslStream will now use the validation callback you pass into the constructor.

[akoeplinger]

as soon as visual studio will support ..NET8, i will switch to .NET8

@CedreLo Visual Studio 2022 version 17.8 has full support for .NET8: https://devblogs.microsoft.com/visualstudio/visual-studio-17-8-now-available/

[dotMorten]

@simonrozsival Will we now also be able to use the certificates installed on the device? Like the ones coming from KeyChain.ChoosePrivateKeyAlias ? We can create the X509Certificate with the Java Pointer to the Java certificate just fine, but the private key isn't directly available for .NET to use, and need to use the underlying Java APIs to authenticate the stream. (So far I've only been able to use certificates that are loaded from a file on disk and has full access to the private key, but that's not generally how customers deploy their PKI certificates, since it creates a risk of exporting the private key).

[simonrozsival]

@dotMorten I think there currently isn't a good way to access the certificates installed on device without using the Java APIs directly. Have you already opened an issue with the use case? If not, could you please open one? We could then discuss if that's something to look into in .NET 9.

[dotMorten]

@simonrozsival For some reason I missed your comment, and for a while thought the issue was the same as this one - I've added a separate issue: #99874