Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Hashicorp Vault to store Tessera keys #565

Merged
merged 61 commits into from Dec 17, 2018
Merged

Add support for Hashicorp Vault to store Tessera keys #565

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
7ffaee1
Add initial service and config to interact with Hashicorp Vault
chris-j-h Nov 28, 2018
770eb14
Add Hashicorp config to KeyConfiguration object
chris-j-h Nov 28, 2018
44089b2
Add factory to create hashicorp service using the token auth method
chris-j-h Nov 28, 2018
8e558f1
Read Hashicorp key data from config and marshall to new key pair type
chris-j-h Nov 28, 2018
f82ba26
Add tests for Hashicorp Service and ServiceFactory
chris-j-h Nov 28, 2018
7375b4d
Add tests for Hashicorp vault config additions
chris-j-h Nov 28, 2018
5bf1b9a
Update validation to cover incomplete Hashicorp keydata
chris-j-h Nov 28, 2018
d54a432
Add validation to require Hashicorp config if providing Hashicorp keys
chris-j-h Nov 28, 2018
35b09a1
Add logic to KeyPairConverter for Hashicorp key pair types
chris-j-h Nov 28, 2018
e37f6d9
Add vault config validation tests and update other tests
chris-j-h Nov 28, 2018
7bae62b
Add ability to save generated keys in a Hashicorp Vault
chris-j-h Nov 29, 2018
d677855
Provide more descriptive message if error encountered writing secret
chris-j-h Nov 29, 2018
832f062
Add test for changes made to key generator factory
chris-j-h Nov 29, 2018
382139a
Change Vault SDK used and add support for TLS comms w/ Vault
chris-j-h Nov 30, 2018
699ed56
Support TLS for keygen by adding certificate CLI option
chris-j-h Nov 30, 2018
d20d526
Remove spring-vault-core implementation
chris-j-h Nov 30, 2018
5f459b5
Remove spring-vault-core-implementation
chris-j-h Nov 30, 2018
169f9e0
Introduce custom objects for parameters of the KeyVaultService methods
chris-j-h Nov 30, 2018
346513e
Add tests for Hashicorp data, exception and service classes
chris-j-h Dec 3, 2018
55c76bc
Start writing tests for Hashicorp vault-java-driver service
chris-j-h Dec 3, 2018
a67b4d9
Move Vault client creation into new class to facilitate testing
chris-j-h Dec 4, 2018
4e7a4ac
Add tests for new Hashicorp config factory classes
chris-j-h Dec 4, 2018
3ee62ea
Begin migration to KeyVaultService methods using SecretData types
chris-j-h Dec 4, 2018
c681778
Add GetSecretData & VaultClient factories for use in KeyPairConverter
chris-j-h Dec 4, 2018
b6654af
Migrate keygeneration code to use SetSecretData types
chris-j-h Dec 4, 2018
a2733da
Move SecretData objects to config package so can be used across the app
chris-j-h Dec 4, 2018
535851c
Clean up no longer needed classes and methods
chris-j-h Dec 4, 2018
7e035d8
Improve TLS support to mutual authentication
chris-j-h Dec 4, 2018
444b8a9
Add support for non-default approle auth method paths
chris-j-h Dec 4, 2018
2750dac
Update tests for key generation and key pair converter
chris-j-h Dec 4, 2018
57c1c3f
Update hashicorp tests
chris-j-h Dec 4, 2018
a524462
Add test for VaultCallback class
chris-j-h Dec 5, 2018
5b07e93
Add tests for Get/SetSecretData classes
chris-j-h Dec 5, 2018
80d1415
Add hashicorp config items to cli for key generation
chris-j-h Dec 5, 2018
8aa6f6c
Add tests to KeyVaultClientFactory and HashicorpKeyVaultConfig
chris-j-h Dec 5, 2018
ba86887
Add additional KeyGenerationParser tests
chris-j-h Dec 5, 2018
c57ffee
Allow both mutual and 1 way TLS authentication to Vault server
chris-j-h Dec 5, 2018
cd79a42
Add more descriptive message to Vault authentication exception
chris-j-h Dec 5, 2018
9738595
Update cli help
chris-j-h Dec 5, 2018
899ffb8
Add test for AppRole authentication exception
chris-j-h Dec 5, 2018
1237004
General clean up
chris-j-h Dec 5, 2018
3224fa0
Merge branch 'master' into feature/hashicorp-vault-support
chris-j-h Dec 5, 2018
0f3145f
Merge branch 'master' into feature/hashicorp-vault-support
prd-fox Dec 6, 2018
d1c34dd
Merge branch 'master' into feature/hashicorp-vault-support
chris-j-h Dec 6, 2018
b366713
Fix whitespace checkstyle issue
chris-j-h Dec 6, 2018
97690f6
minor refactor - reduce unnecessary visibility
namtruong Dec 7, 2018
0e8da5e
Split Hashicorp secretPath into 2 params and use keystores in config
chris-j-h Dec 12, 2018
9d66a0d
[WIP] Change client library and update ServiceFactory
chris-j-h Dec 12, 2018
bcc2ca3
[WIP] Use separate SecretData properties for secretengine and secretname
chris-j-h Dec 13, 2018
8e0287d
[WIP] Update KeyVaultService to use new client and SecretData types
chris-j-h Dec 13, 2018
c081674
[WIP] Create KeyValueOperationsDelegate to aid testing
chris-j-h Dec 13, 2018
3310e7e
[WIP] Remove KeyVaultClientFactory class as no longer in use
chris-j-h Dec 13, 2018
b588955
Add keygen CLI options for keystores and secret engine name
chris-j-h Dec 13, 2018
2459912
Remove unnecessary classes and fix tests broken after previous changes
chris-j-h Dec 14, 2018
bb7a507
Allow previous versions of a Hashicorp secret to be retrieved from Vault
chris-j-h Dec 14, 2018
64c4523
Add vault dependencies to dep mgmt and minor changes/tests
chris-j-h Dec 14, 2018
d52402c
Minor clean up
chris-j-h Dec 14, 2018
b6acba6
Remove unused files & allow lowercase vault type in keygen CLI
chris-j-h Dec 14, 2018
1444b99
Remove use of PrivateKey toString in file keygen
chris-j-h Dec 17, 2018
e1bad42
Use more specific exception types
chris-j-h Dec 17, 2018
7145583
Merge branch 'master' into feature/hashicorp-vault-support
melowe Dec 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions config-cli/src/main/java/com/quorum/tessera/config/cli/DefaultCliAdapter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,42 @@ private Options buildBaseOptions() {
.build()
);

options.addOption(
Option.builder("keygenvaultapprole")
.desc("AppRole path for Hashicorp Vault authentication (defaults to 'approle')")
.hasArg()
.optionalArg(false)
.argName("STRING")
.build()
);

options.addOption(
Option.builder("keygenvaultkeystore")
.desc("Path to JKS keystore for TLS Hashicorp Vault communication")
.hasArg()
.optionalArg(false)
.argName("PATH")
.build()
);

options.addOption(
Option.builder("keygenvaulttruststore")
.desc("Path to JKS truststore for TLS Hashicorp Vault communication")
.hasArg()
.optionalArg(false)
.argName("PATH")
.build()
);

options.addOption(
Option.builder("keygenvaultsecretengine")
.desc("Name of already enabled Hashicorp v2 kv secret engine")
.hasArg()
.optionalArg(false)
.argName("STRING")
.build()
);

options.addOption(
Option.builder("pidfile")
.desc("Path to pid file")
Expand Down
64 changes: 51 additions & 13 deletions config-cli/src/main/java/com/quorum/tessera/config/cli/parsers/KeyGenerationParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
package com.quorum.tessera.config.cli.parsers;

import com.quorum.tessera.config.ArgonOptions;
import com.quorum.tessera.config.AzureKeyVaultConfig;
import com.quorum.tessera.config.KeyVaultConfig;
import com.quorum.tessera.config.KeyVaultType;
import com.quorum.tessera.config.*;
import com.quorum.tessera.config.cli.CliException;
import com.quorum.tessera.config.keypairs.ConfigKeyPair;
import com.quorum.tessera.config.util.JaxbUtil;
import com.quorum.tessera.key.generation.KeyGenerator;
import com.quorum.tessera.key.generation.KeyGeneratorFactory;
import com.quorum.tessera.key.generation.KeyVaultOptions;
import org.apache.commons.cli.CommandLine;

import javax.validation.ConstraintViolation;
Expand All @@ -18,6 +16,7 @@
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
Expand All @@ -41,14 +40,15 @@ public class KeyGenerationParser implements Parser<List<ConfigKeyPair>> {
public List<ConfigKeyPair> parse(final CommandLine commandLine) throws IOException {

final ArgonOptions argonOptions = this.argonOptions(commandLine).orElse(null);
final KeyVaultOptions keyVaultOptions = this.keyVaultOptions(commandLine).orElse(null);
final KeyVaultConfig keyVaultConfig = this.keyVaultConfig(commandLine).orElse(null);

final KeyGenerator generator = factory.create(keyVaultConfig);

if (commandLine.hasOption("keygen")) {
return this.filenames(commandLine)
.stream()
.map(name -> generator.generate(name, argonOptions))
.map(name -> generator.generate(name, argonOptions, keyVaultOptions))
.collect(Collectors.toList());
}

Expand All @@ -68,6 +68,12 @@ private Optional<ArgonOptions> argonOptions(final CommandLine commandLine) throw
return Optional.empty();
}

private Optional<KeyVaultOptions> keyVaultOptions(final CommandLine commandLine) {
Optional<String> secretEngineName = Optional.ofNullable(commandLine.getOptionValue("keygenvaultsecretengine"));

return secretEngineName.map(KeyVaultOptions::new);
}

private List<String> filenames(final CommandLine commandLine) {

if (commandLine.hasOption("filename")) {
Expand All @@ -88,23 +94,55 @@ private Optional<KeyVaultConfig> keyVaultConfig(CommandLine commandLine) {
return Optional.empty();
}

String t = commandLine.getOptionValue("keygenvaulttype");
final String t = commandLine.getOptionValue("keygenvaulttype");

KeyVaultType keyVaultType;
try {
KeyVaultType.valueOf(t);
keyVaultType = KeyVaultType.valueOf(
t.trim()
.toUpperCase()
);
} catch(IllegalArgumentException | NullPointerException e) {
throw new CliException("Key vault type either not provided or not recognised. Ensure provided value is UPPERCASE and has no leading or trailing whitespace characters");
throw new CliException("Key vault type either not provided or not recognised");
}

String keyVaultUrl = commandLine.getOptionValue("keygenvaulturl");

//Only Azure supported atm so no need to check keyvaulttype
KeyVaultConfig keyVaultConfig = new AzureKeyVaultConfig(keyVaultUrl);
KeyVaultConfig keyVaultConfig;

if(keyVaultType.equals(KeyVaultType.AZURE)) {
keyVaultConfig = new AzureKeyVaultConfig(keyVaultUrl);

Set<ConstraintViolation<AzureKeyVaultConfig>> violations = validator.validate((AzureKeyVaultConfig)keyVaultConfig);
Set<ConstraintViolation<AzureKeyVaultConfig>> violations = validator.validate((AzureKeyVaultConfig)keyVaultConfig);

if(!violations.isEmpty()) {
throw new ConstraintViolationException(violations);
}
} else {
if(!commandLine.hasOption("filename")) {
throw new CliException("At least one -filename must be provided when saving generated keys in a Hashicorp Vault");
}

if(!violations.isEmpty()) {
throw new ConstraintViolationException(violations);
String approlePath = commandLine.getOptionValue("keygenvaultapprole");

Optional<Path> tlsKeyStorePath = Optional.ofNullable(commandLine.getOptionValue("keygenvaultkeystore"))
.map(Paths::get);

Optional<Path> tlsTrustStorePath = Optional.ofNullable(commandLine.getOptionValue("keygenvaulttruststore"))
.map(Paths::get);

keyVaultConfig = new HashicorpKeyVaultConfig(
keyVaultUrl,
approlePath,
tlsKeyStorePath.orElse(null),
tlsTrustStorePath.orElse(null)
);

Set<ConstraintViolation<HashicorpKeyVaultConfig>> violations = validator.validate((HashicorpKeyVaultConfig)keyVaultConfig);

if(!violations.isEmpty()) {
throw new ConstraintViolationException(violations);
}
}

return Optional.of(keyVaultConfig);
Expand Down
10 changes: 5 additions & 5 deletions config-cli/src/test/java/com/quorum/tessera/config/cli/DefaultCliAdapterTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ public void keygenWithConfig() throws Exception {
Files.write(publicKeyPath, Arrays.asList("SOMEDATA"));

FilesystemKeyPair keypair = new FilesystemKeyPair(publicKeyPath, privateKeyPath);
when(keyGenerator.generate(anyString(), eq(null))).thenReturn(keypair);
when(keyGenerator.generate(anyString(), eq(null), eq(null))).thenReturn(keypair);

Path unixSocketPath = Files.createTempFile(UUID.randomUUID().toString(), ".ipc");

Expand All @@ -143,7 +143,7 @@ public void keygenWithConfig() throws Exception {
assertThat(result.getConfig()).isNotNull();
assertThat(result.isSuppressStartup()).isFalse();

verify(keyGenerator).generate(anyString(), eq(null));
verify(keyGenerator).generate(anyString(), eq(null), eq(null));
verifyNoMoreInteractions(keyGenerator);

}
Expand Down Expand Up @@ -191,7 +191,7 @@ public void output() throws Exception {
Files.write(publicKeyPath, Arrays.asList("SOMEDATA"));

FilesystemKeyPair keypair = new FilesystemKeyPair(publicKeyPath, privateKeyPath);
when(keyGenerator.generate(anyString(), eq(null))).thenReturn(keypair);
when(keyGenerator.generate(anyString(), eq(null), eq(null))).thenReturn(keypair);

Path generatedKey = Paths.get("/tmp/" + UUID.randomUUID().toString());

Expand Down Expand Up @@ -376,7 +376,7 @@ public void allowStartupForKeygenAndConfigfileOptions() throws Exception {
Files.write(publicKeyPath, Arrays.asList("SOMEDATA"));

FilesystemKeyPair keypair = new FilesystemKeyPair(publicKeyPath, privateKeyPath);
when(keyGenerator.generate(anyString(), eq(null))).thenReturn(keypair);
when(keyGenerator.generate(anyString(), eq(null), eq(null))).thenReturn(keypair);

final Path configFile = createAndPopulatePaths(getClass().getResource("/sample-config.json"));

Expand All @@ -389,7 +389,7 @@ public void allowStartupForKeygenAndConfigfileOptions() throws Exception {
public void suppressStartupForKeygenAndVaultUrlAndConfigfileOptions() throws Exception {
final KeyGenerator keyGenerator = MockKeyGeneratorFactory.getMockKeyGenerator();
final FilesystemKeyPair keypair = new FilesystemKeyPair(Paths.get(""), Paths.get(""));
when(keyGenerator.generate(anyString(), eq(null))).thenReturn(keypair);
when(keyGenerator.generate(anyString(), eq(null), eq(null))).thenReturn(keypair);

final Path configFile = createAndPopulatePaths(getClass().getResource("/sample-config.json"));
final String vaultUrl = "https://test.vault.azure.net";
Expand Down
157 changes: 82 additions & 75 deletions config-cli/src/test/java/com/quorum/tessera/config/cli/OverrideUtilTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
package com.quorum.tessera.config.cli;

import com.quorum.tessera.config.*;
import com.quorum.tessera.config.Config;
import com.quorum.tessera.config.KeyConfiguration;
import com.quorum.tessera.config.Peer;
import com.quorum.tessera.config.SslAuthenticationMode;
import com.quorum.tessera.config.util.JaxbUtil;
import org.junit.Ignore;
import org.junit.Test;
Expand Down Expand Up @@ -29,80 +32,84 @@ public class OverrideUtilTest {
public void buildOptions() {

final List<String> expected = Arrays.asList(
"jdbc.username",
"jdbc.password",
"jdbc.url",
"jdbc.autoCreateTables",
"peer.url",
"keys.passwordFile",
"keys.passwords",
"keys.keyData.config.data.aopts.algorithm",
"keys.keyData.config.data.aopts.iterations",
"keys.keyData.config.data.aopts.memory",
"keys.keyData.config.data.aopts.parallelism",
"keys.keyData.privateKeyPath",
"keys.azureKeyVaultConfig.url",
"alwaysSendTo",
"unixSocketFile",
"useWhiteList",
"disablePeerDiscovery",
"serverConfigs.sslConfig.serverTrustStore",
"serverConfigs.influxConfig.dbName",
"serverConfigs.sslConfig.knownClientsFile",
"serverConfigs.influxConfig.hostName",
"serverConfigs.sslConfig.serverTrustCertificates",
"serverConfigs.sslConfig.clientTrustCertificates",
"serverConfigs.sslConfig.clientTrustStorePassword",
"serverConfigs.sslConfig.generateKeyStoreIfNotExisted",
"serverConfigs.influxConfig.pushIntervalInSecs",
"serverConfigs.bindingAddress",
"serverConfigs.sslConfig.serverKeyStore",
"serverConfigs.sslConfig.serverTrustStorePassword",
"serverConfigs.sslConfig.serverKeyStorePassword",
"serverConfigs.sslConfig.clientTrustMode",
"serverConfigs.sslConfig.clientKeyStorePassword",
"serverConfigs.communicationType",
"serverConfigs.sslConfig.clientTlsCertificatePath",
"serverConfigs.sslConfig.serverTlsKeyPath",
"serverConfigs.sslConfig.clientKeyStore",
"serverConfigs.sslConfig.serverTrustMode",
"serverConfigs.influxConfig.port",
"serverConfigs.sslConfig.clientTlsKeyPath",
"serverConfigs.app",
"serverConfigs.sslConfig.clientTrustStore",
"serverConfigs.enabled",
"serverConfigs.sslConfig.serverTlsCertificatePath",
"serverConfigs.sslConfig.tls",
"serverConfigs.sslConfig.knownServersFile",
"server.hostName",
"server.sslConfig.knownServersFile",
"server.sslConfig.clientTrustStorePassword",
"server.sslConfig.clientKeyStorePassword",
"server.sslConfig.clientTlsKeyPath",
"server.sslConfig.clientTrustCertificates",
"server.sslConfig.knownClientsFile",
"server.communicationType",
"server.sslConfig.serverTrustStorePassword",
"server.sslConfig.serverTrustCertificates",
"server.sslConfig.clientTrustStore",
"server.sslConfig.tls",
"server.sslConfig.serverTlsCertificatePath",
"server.grpcPort",
"server.sslConfig.serverKeyStore",
"server.influxConfig.port",
"server.port",
"server.sslConfig.generateKeyStoreIfNotExisted",
"server.sslConfig.clientTlsCertificatePath",
"server.sslConfig.serverTlsKeyPath",
"server.influxConfig.hostName",
"server.sslConfig.serverTrustStore",
"server.bindingAddress",
"server.sslConfig.serverTrustMode",
"server.sslConfig.clientKeyStore",
"server.influxConfig.dbName",
"server.sslConfig.clientTrustMode",
"server.influxConfig.pushIntervalInSecs",
"server.sslConfig.serverKeyStorePassword"
"jdbc.username",
"jdbc.password",
"jdbc.url",
"jdbc.autoCreateTables",
"peer.url",
"keys.passwordFile",
"keys.passwords",
"keys.keyData.config.data.aopts.algorithm",
"keys.keyData.config.data.aopts.iterations",
"keys.keyData.config.data.aopts.memory",
"keys.keyData.config.data.aopts.parallelism",
"keys.keyData.privateKeyPath",
"keys.azureKeyVaultConfig.url",
"keys.hashicorpKeyVaultConfig.approlePath",
"keys.hashicorpKeyVaultConfig.tlsKeyStorePath",
"keys.hashicorpKeyVaultConfig.tlsTrustStorePath",
"keys.hashicorpKeyVaultConfig.url",
"alwaysSendTo",
"unixSocketFile",
"useWhiteList",
"disablePeerDiscovery",
"serverConfigs.sslConfig.serverTrustStore",
"serverConfigs.influxConfig.dbName",
"serverConfigs.sslConfig.knownClientsFile",
"serverConfigs.influxConfig.hostName",
"serverConfigs.sslConfig.serverTrustCertificates",
"serverConfigs.sslConfig.clientTrustCertificates",
"serverConfigs.sslConfig.clientTrustStorePassword",
"serverConfigs.sslConfig.generateKeyStoreIfNotExisted",
"serverConfigs.influxConfig.pushIntervalInSecs",
"serverConfigs.bindingAddress",
"serverConfigs.sslConfig.serverKeyStore",
"serverConfigs.sslConfig.serverTrustStorePassword",
"serverConfigs.sslConfig.serverKeyStorePassword",
"serverConfigs.sslConfig.clientTrustMode",
"serverConfigs.sslConfig.clientKeyStorePassword",
"serverConfigs.communicationType",
"serverConfigs.sslConfig.clientTlsCertificatePath",
"serverConfigs.sslConfig.serverTlsKeyPath",
"serverConfigs.sslConfig.clientKeyStore",
"serverConfigs.sslConfig.serverTrustMode",
"serverConfigs.influxConfig.port",
"serverConfigs.sslConfig.clientTlsKeyPath",
"serverConfigs.app",
"serverConfigs.sslConfig.clientTrustStore",
"serverConfigs.enabled",
"serverConfigs.sslConfig.serverTlsCertificatePath",
"serverConfigs.sslConfig.tls",
"serverConfigs.sslConfig.knownServersFile",
"server.hostName",
"server.sslConfig.knownServersFile",
"server.sslConfig.clientTrustStorePassword",
"server.sslConfig.clientKeyStorePassword",
"server.sslConfig.clientTlsKeyPath",
"server.sslConfig.clientTrustCertificates",
"server.sslConfig.knownClientsFile",
"server.communicationType",
"server.sslConfig.serverTrustStorePassword",
"server.sslConfig.serverTrustCertificates",
"server.sslConfig.clientTrustStore",
"server.sslConfig.tls",
"server.sslConfig.serverTlsCertificatePath",
"server.grpcPort",
"server.sslConfig.serverKeyStore",
"server.influxConfig.port",
"server.port",
"server.sslConfig.generateKeyStoreIfNotExisted",
"server.sslConfig.clientTlsCertificatePath",
"server.sslConfig.serverTlsKeyPath",
"server.influxConfig.hostName",
"server.sslConfig.serverTrustStore",
"server.bindingAddress",
"server.sslConfig.serverTrustMode",
"server.sslConfig.clientKeyStore",
"server.influxConfig.dbName",
"server.sslConfig.clientTrustMode",
"server.influxConfig.pushIntervalInSecs",
"server.sslConfig.serverKeyStorePassword"
);

final Map<String, Class> results = OverrideUtil.buildConfigOptions();
Expand Down
Loading