Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AwsSecretsManager Plugin with Application Server (Tomcat) Resource Configuration - Credentials not read from secret #305

Closed
ChaosMarc opened this issue Jan 12, 2023 · 4 comments
Closed

AwsSecretsManager Plugin with Application Server (Tomcat) Resource Configuration - Credentials not read from secret #305

ChaosMarc opened this issue Jan 12, 2023 · 4 comments
Assignees
@karenc-bq
Labels
duplicate This issue or pull request already exists

Comments

@ChaosMarc
Copy link

I couldn't find any information if it is possible to use this wrapper by tomcat xml configuration only but tried anyway. After a lot of trial and error I'm stuck now and would kindly ask for help.

I'm trying to connect my java application (Liferay) running on a tomcat 9.0.67 in an EKS 1.24.7 cluster with an Aurora PostgreSQL 11.13. My database configuration is done in tomcat/conf/Catalina/localhost/ROOT.xml and looks like this:

<Context crossContext="true">
    ...
    <Resource name="jdbc/LiferayPool"
              auth="Container"
              type="software.amazon.jdbc.ds.AwsWrapperDataSource"
              factory="software.amazon.jdbc.ds.AwsWrapperDataSourceFactory"
              driverClassName="software.amazon.jdbc.Driver"
              jdbcUrl="jdbc:aws-wrapper:postgresql://${jdbc.host}/${jdbc.database}?sslmode=require"
              user="-"
              password="-"
              wrapperPlugins="awsSecretsManager"
              secretsManagerSecretId="liferay-db-secrets"
              secretsManagerRegion="eu-central-1"
              targetDataSourceClassName="org.postgresql.ds.PGSimpleDataSource"
              jdbcProtocol="jdbc:postgresql:"
              serverPropertyName="serverName"
              portPropertyName="port"
              urlPropertyName="jdbcUrl"
              databasePropertyName="databaseName"
              connectionTimeout="30000"
              idleTimeout="600000"
              maxLifetime="1800000"
              connectionTestQuery="SELECT 1"
              minimumIdle="10"
              maxTotal="30"
              maximumPoolSize="40"
              registerMbeans="true"
    />
</Context>

and I have the following dependencies as jar files in my tomcat/lib/ext directory:

com.fasterxml.jackson.core:jackson-annotations:2.14.1
com.fasterxml.jackson.core:jackson-core:2.14.1
com.fasterxml.jackson.core:jackson-databind:2.14.1
org.reactivestreams:reactive-streams:1.0.4
org.slf4j:slf4j-api:2.0.6
software.amazon.awssdk:auth:2.19.13
software.amazon.awssdk:aws-core:2.19.13
software.amazon.awssdk:aws-json-protocol:2.19.13
software.amazon.awssdk:aws-query-protocol:2.19.13
software.amazon.awssdk:endpoints-spi:2.19.13
software.amazon.awssdk:http-client-spi:2.19.13
software.amazon.awssdk:json-utils:2.19.13
software.amazon.awssdk:metrics-spi:2.19.13
software.amazon.awssdk:profiles:2.19.13
software.amazon.awssdk:protocol-core:2.19.13
software.amazon.awssdk:rds:2.19.13
software.amazon.awssdk:regions:2.19.13
software.amazon.awssdk:sdk-core:2.19.13
software.amazon.awssdk:secretsmanager:2.19.13
software.amazon.awssdk:sts:2.19.13
software.amazon.awssdk:third-party-jackson-core:2.19.13
software.amazon.awssdk:url-connection-client:2.19.13
software.amazon.awssdk:utils:2.19.13
software.amazon.jdbc:aws-advanced-jdbc-wrapper:1.0.0

My problem is, that I cannot omit (or provide an empty string for) the two attributes "user" and "password", as I receive the following exception if I do:

12-Jan-2023 06:51:30.837 WARNING [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference
        java.lang.NullPointerException
                at software.amazon.jdbc.ds.AwsWrapperDataSourceFactory.getObjectInstance(AwsWrapperDataSourceFactory.java:58)
                at org.apache.naming.factory.FactoryBase.getObjectInstance(FactoryBase.java:96)
                at java.naming/javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:341)
                at org.apache.naming.NamingContext.lookup(NamingContext.java:864)
                at org.apache.naming.NamingContext.lookup(NamingContext.java:158)
                at org.apache.naming.NamingContext.lookup(NamingContext.java:850)
                at org.apache.naming.NamingContext.lookup(NamingContext.java:158)
                at org.apache.naming.NamingContext.lookup(NamingContext.java:850)
                at org.apache.naming.NamingContext.lookup(NamingContext.java:158)
                at org.apache.naming.NamingContext.lookup(NamingContext.java:850)
                at org.apache.naming.NamingContext.lookup(NamingContext.java:172)
                at org.apache.naming.SelectorContext.lookup(SelectorContext.java:161)
                at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
                at com.liferay.portal.kernel.jndi.JNDIUtil._lookup(JNDIUtil.java:134)
                at com.liferay.portal.kernel.jndi.JNDIUtil.lookup(JNDIUtil.java:33)
                at com.liferay.portal.dao.jdbc.DataSourceFactoryImpl.initDataSource(DataSourceFactoryImpl.java:172)
                at com.liferay.portal.kernel.dao.jdbc.DataSourceFactoryUtil.initDataSource(DataSourceFactoryUtil.java:39)
                at com.liferay.portal.dao.init.DBInitUtil._initDataSource(DBInitUtil.java:223)
                at com.liferay.portal.dao.init.DBInitUtil.init(DBInitUtil.java:70)
                at com.liferay.portal.spring.context.PortalContextLoaderListener.contextInitialized(PortalContextLoaderListener.java:240)
                at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4769)
                at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5231)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
                at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
                at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
                at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690)
                at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889)
                at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
                at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
                at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:583)
                at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:473)
                at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618)
                at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319)
                at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
                at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
                at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
                at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:946)
                at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396)
                at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
                at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
                at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:919)
                at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.StandardService.startInternal(StandardService.java:432)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)

But If I provide some dummy string like "-" this information is used instead of the actual secrets value even thouh I can see that the the awsSecretsManager Plugin is used in the stacktrace:

12-Jan-2023 07:04:07.055 SEVERE [main] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.liferay.portal.spring.context.PortalContextLoaderListener]
      java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "-"
              at com.liferay.portal.spring.context.PortalContextLoaderListener.contextInitialized(PortalContextLoaderListener.java:250)
              at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4769)
              at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5231)
              at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
              at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
              at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
              at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
              at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690)
              at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889)
              at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
              at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
              at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
              at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
              at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:583)
              at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:473)
              at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618)
              at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319)
              at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
              at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
              at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
              at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:946)
              at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
              at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
              at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396)
              at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386)
              at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
              at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
              at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
              at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:919)
              at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265)
              at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
              at org.apache.catalina.core.StandardService.startInternal(StandardService.java:432)
              at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
              at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
              at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
              at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
              at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.base/java.lang.reflect.Method.invoke(Method.java:566)
              at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
              at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
      Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "-"
              at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:646)
              at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:180)
              at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235)
              at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
              at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:247)
              at org.postgresql.Driver.makeConnection(Driver.java:434)
              at org.postgresql.Driver.connect(Driver.java:291)
              at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
              at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
              at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:103)
              at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:87)
              at software.amazon.jdbc.DataSourceConnectionProvider.connect(DataSourceConnectionProvider.java:104)
              at software.amazon.jdbc.plugin.DefaultConnectionPlugin.connect(DefaultConnectionPlugin.java:126)
              at software.amazon.jdbc.ConnectionPluginManager.lambda$connect$4(ConnectionPluginManager.java:314)
              at software.amazon.jdbc.ConnectionPluginManager.lambda$makePluginChainFunc$0(ConnectionPluginManager.java:246)
              at software.amazon.jdbc.ConnectionPluginManager.lambda$null$1(ConnectionPluginManager.java:250)
              at software.amazon.jdbc.plugin.AwsSecretsManagerConnectionPlugin.connect(AwsSecretsManagerConnectionPlugin.java:151)
              at software.amazon.jdbc.ConnectionPluginManager.lambda$connect$4(ConnectionPluginManager.java:314)
              at software.amazon.jdbc.ConnectionPluginManager.lambda$makePluginChainFunc$2(ConnectionPluginManager.java:250)
              at software.amazon.jdbc.ConnectionPluginManager.executeWithSubscribedPlugins(ConnectionPluginManager.java:228)
              at software.amazon.jdbc.ConnectionPluginManager.connect(ConnectionPluginManager.java:311)
              at software.amazon.jdbc.wrapper.ConnectionWrapper.init(ConnectionWrapper.java:131)
              at software.amazon.jdbc.wrapper.ConnectionWrapper.<init>(ConnectionWrapper.java:85)
              at software.amazon.jdbc.ds.AwsWrapperDataSource.getConnection(AwsWrapperDataSource.java:123)
              at software.amazon.jdbc.ds.AwsWrapperDataSource.getConnection(AwsWrapperDataSource.java:77)
              at com.liferay.portal.dao.jdbc.util.DBInfoUtil.lambda$getDBInfo$0(DBInfoUtil.java:47)
              at com.liferay.petra.concurrent.ConcurrentMapperHashMap.lambda$computeIfAbsent$1(ConcurrentMapperHashMap.java:114)
              at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
              at com.liferay.petra.concurrent.ConcurrentMapperHashMap.computeIfAbsent(ConcurrentMapperHashMap.java:111)
              at com.liferay.portal.dao.jdbc.util.DBInfoUtil.getDBInfo(DBInfoUtil.java:44)
              at com.liferay.portal.spring.hibernate.DialectDetector.getDialect(DialectDetector.java:51)
              at com.liferay.portal.dao.init.DBInitUtil._initDataSource(DBInitUtil.java:226)
              at com.liferay.portal.dao.init.DBInitUtil.init(DBInitUtil.java:70)
              at com.liferay.portal.spring.context.PortalContextLoaderListener.contextInitialized(PortalContextLoaderListener.java:240)
              ... 41 more

The secret is provided through a k8s service account that references an IAM role with the permission to read the secret in question. My environment variables look like this:

env | grep AWS
AWS_ROLE_ARN=arn:aws:iam::XXXXXXXXXX:role/YYYYYYYYYYYYYYYYYYYYYYYY
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_STS_REGIONAL_ENDPOINTS=regional
AWS_DEFAULT_REGION=eu-central-1
AWS_REGION=eu-central-1

I have verified that the connection and permissions are correctly setup by modifying one of the sample projects you provided and running it on the pod in question with tomcat/lib/ext/ as classpath:

package com.example.aws.sdk.test;

import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueRequest;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueResponse;
import software.amazon.awssdk.services.secretsmanager.model.SecretsManagerException;

public class AwsTest {
    public static void main(String[] args) {
        try {
            SecretsManagerClient secretsClient = SecretsManagerClient.builder().region(Region.EU_CENTRAL_1)
                .credentialsProvider(DefaultCredentialsProvider.create()).build();

            GetSecretValueRequest valueRequest = GetSecretValueRequest.builder().secretId("liferay-db-secrets").build();

            GetSecretValueResponse valueResponse = secretsClient.getSecretValue(valueRequest);
            String secret = valueResponse.secretString();
            System.out.println(secret);

        } catch (SecretsManagerException e) {
            System.err.println(e.awsErrorDetails().errorMessage());
            System.exit(1);
        }
    }
}

Am I missing something or is this not intended to work like this?
@karenc-bq
Copy link
Contributor

Hi @ChaosMarc, thank you for reporting this issue. We will look into the problem and will share more info as we investigate. Thank you for your patience!

@karenc-bq
Copy link
Contributor

Hi @ChaosMarc, this issue should also have been fixed by PR #292. Would you be able to test the changes with our snapshot build and let us know if the issue persists? Thank you.

@ChaosMarc
Copy link
Author

Hey @karenc-bq, thank you for the pointer. I didn't recogize #292 as the same problem as it only mentions the iam plugin. but nevertheless the current snapshot fixes my problem 😃

are there any plans for a release date of a 1.0.1 final? Policies prevent me from using a snapshot in production, so this would have to wait until then.

Thanks again

@hsuamz hsuamz added duplicate This issue or pull request already exists pending release Resolution implemented, pending official release labels Jan 17, 2023
@karenc-bq
Copy link
Contributor

Hi @ChaosMarc, we've released a new version of the driver that resolves this issue. Please download the latest 1.0.1 version of the driver. We'll be closing this ticket. Thank you for your patience.

@hsuamz hsuamz removed the pending release Resolution implemented, pending official release label Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@karenc-bq karenc-bq

Labels
duplicate This issue or pull request already exists
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ChaosMarc @karenc-bq @hsuamz