Localstack Testcontainers module behaviour on Jenkins

[akulik512]

Hello guys
I use MySQL and Localstack Testcontainers modules in tests and tests work well locally. But in Jenkins, I see that Localstack container can't create buckets at the same time everything is ok with MySQL DB

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>mysql</artifactId>
     <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.github.terma</groupId>
    <artifactId>javaniotcpproxy</artifactId>
    <version>${javaniotcpproxy.version}</version>
    <scope>test</scope>
</dependency>

From the log belove you can see that container was started but no buckets were created:

[2023-11-17 09:34:54.557] INFO  t.l.3.2 - Container localstack/localstack:2.3.2 is starting: 856f52f50009fb130286a2240c6734c4982813cbb2e50b6be0699e649fb86044
[2023-11-17 09:34:55.938] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 
[2023-11-17 09:34:55.938] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: LocalStack version: 2.3.2
[2023-11-17 09:34:55.941] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: LocalStack Docker container id: 856f52f50009
[2023-11-17 09:34:55.941] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: LocalStack build date: 2023-10-03
[2023-11-17 09:34:55.941] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: LocalStack build git hash: 1bccf790
[2023-11-17 09:34:55.941] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 
[2023-11-17 09:34:57.433] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 2023-11-17T09:34:57.432  INFO --- [-functhread4] hypercorn.error            : Running on https://0.0.0.0:4566/ (CTRL + C to quit)
[2023-11-17 09:34:57.433] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 2023-11-17T09:34:57.432  INFO --- [-functhread4] hypercorn.error            : Running on https://0.0.0.0:4566/ (CTRL + C to quit)
[2023-11-17 09:34:57.626] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: Ready.
[2023-11-17 09:34:57.626] INFO  t.l.3.2 - Container localstack/localstack:2.3.2 started in PT3.693212202S

And this is how the same logs look like locally:

[2023-11-17 14:09:12.064] INFO  t.l.3.2 - Container localstack/localstack:2.3.2 is starting: 9e501bc6b83fca9a79b40df0549c4f96f4499ae9ae23a649a38589bbba9ca516
[2023-11-17 14:09:12.719] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 
[2023-11-17 14:09:12.720] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: LocalStack version: 2.3.2
[2023-11-17 14:09:12.724] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: LocalStack Docker container id: 9e501bc6b83f
[2023-11-17 14:09:12.724] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: LocalStack build date: 2023-10-03
[2023-11-17 14:09:12.725] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: LocalStack build git hash: 1bccf790
[2023-11-17 14:09:12.725] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 
[2023-11-17 14:09:14.886] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 2023-11-17T12:09:14.885  INFO --- [-functhread6] hypercorn.error            : Running on https://0.0.0.0:4566/ (CTRL + C to quit)
[2023-11-17 14:09:14.886] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 2023-11-17T12:09:14.885  INFO --- [-functhread6] hypercorn.error            : Running on https://0.0.0.0:4566/ (CTRL + C to quit)
[2023-11-17 14:09:14.958] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 2023-11-17T12:09:14.957  INFO --- [  MainThread] localstack.utils.bootstrap : Execution of "start_runtime_components" took 902.54ms
[2023-11-17 14:09:14.960] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: Ready.
[2023-11-17 14:09:14.968] INFO  t.l.3.2 - Container localstack/localstack:2.3.2 started in PT2.980449716S
[2023-11-17 14:09:18.484] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 2023-11-17T12:09:18.483  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.CreateBucket => 200
[2023-11-17 14:09:25.599] INFO  c.o.t.i.AbstractIntegrationTest - STDOUT: 2023-11-17T12:09:25.598  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.PutObject => 200

As a result, I have:

com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: c9d288fb-d1de-4e1a-a3eb-dfe3e0a4d7f7; S3 Extended Request ID: null; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)

Also, I see a reason for that but don't quite a root issue. I printed the bucket creation operation result in logs (S3_CONTAINER.execInContainer("awslocal", "s3", "mb", inputBucketAddress);) and got this:

Container.ExecResult(exitCode=1, stdout=, stderr=make_bucket failed: s3://channel-list-testcontainers Could not connect to the endpoint URL: "http://172.17.0.1:4566/channel-list-testcontainers"
)

I don't know what does it led to this issue on Jenkins. The container set up doesn't have any special things except one. I use TCP proxy to get static container port:

    @Container
    public static final MySQLContainer<?> MY_SQL_CONTAINER =
            new MySQLContainer<>(DockerImageName.parse("mysql:8.1.0"))
                    .withDatabaseName("telegram")
                    .withUsername("root")
                    .withPassword("test")
                    .withLogConsumer(new Slf4jLogConsumer(LOGGER))
                    .withInitScript("changelog/database_structure.sql")
                    .withConfigurationOverride("changelog/mysql-configuration-override");

    @Container
    public static final LocalStackContainer S3_CONTAINER =
            new LocalStackContainer(DockerImageName.parse("localstack/localstack:2.3.2"))
                    .withServices(S3)
                    .withLogConsumer(new Slf4jLogConsumer(LOGGER))
                    .withCopyFileToContainer(MountableFile
                            .forClasspathResource(VALID_DATA_FILE), DOCKER_FILE_PATH_TO_VALID_FILE)
                    .withCopyFileToContainer(MountableFile
                            .forClasspathResource(INVALID_DATA_FILE), DOCKER_FILE_PATH_TO_INVALID_FILE);

    protected AmazonS3 amazonS3TestClient = buildTestAmazonS3Client();

    @BeforeAll
    static void beforeAll() throws IOException, InterruptedException {
        ScriptUtils.runInitScript(new JdbcDatabaseDelegate(MY_SQL_CONTAINER, ""), "changelog/database_data.sql");
        log.info("### MySQL URL: {}", MY_SQL_CONTAINER.getJdbcUrl());

        String inputBucketAddress = "s3://" + CHANNELS_INPUT_BUCKET;
        String outputBucketAddress = "s3://" + CHANNELS_OUTPUT_BUCKET;
        Container.ExecResult createBucketExecResult1 = S3_CONTAINER.execInContainer("awslocal", "s3", "mb", inputBucketAddress);
        Container.ExecResult createBucketExecResult2 = S3_CONTAINER.execInContainer("awslocal", "s3", "mb", outputBucketAddress);
        log.info("### '{}' bucket createBucketExecResult1: {}", CHANNELS_INPUT_BUCKET, createBucketExecResult1);
        log.info("### '{}' bucket createBucketExecResult2: {}", CHANNELS_OUTPUT_BUCKET, createBucketExecResult2);

        String validFileBucketPath = inputBucketAddress + "/" + VALID_DATA_FILE;
        Container.ExecResult copyFileExecResult1 =
                S3_CONTAINER.execInContainer("awslocal", "s3", "cp", DOCKER_FILE_PATH_TO_VALID_FILE, validFileBucketPath);
        String invalidFileBucketPath = inputBucketAddress + "/" + INVALID_DATA_FILE;
        Container.ExecResult copyFileExecResult2 =
                S3_CONTAINER.execInContainer("awslocal", "s3", "cp", DOCKER_FILE_PATH_TO_INVALID_FILE, invalidFileBucketPath);
        log.info("### copyFileExecResult1: {}", copyFileExecResult1);
        log.info("### copyFileExecResult2: {}", copyFileExecResult2);

        log.info("### AWS S3 URL: {}", S3_CONTAINER.getEndpointOverride(S3).toString());
        log.info("### Bucket list: {}", S3_CONTAINER.execInContainer("awslocal", "s3api", "list-buckets"));

        initializeMysqlTcpProxy();
        initializeAwsS3TcpProxy();
    }

    private static void initializeMysqlTcpProxy() {
        var config = new StaticTcpProxyConfig(
                3705,
                MY_SQL_CONTAINER.getHost(),
                MY_SQL_CONTAINER.getMappedPort(3306)
        );
        config.setWorkerCount(1);
        var tcpProxy = new TcpProxy(config);
        tcpProxy.start();
    }

    private static void initializeAwsS3TcpProxy() {
        var config = new StaticTcpProxyConfig(
                8535,
                S3_CONTAINER.getHost(),
                S3_CONTAINER.getMappedPort(4566)
        );
        config.setWorkerCount(1);
        var tcpProxy = new TcpProxy(config);
        tcpProxy.start();
    }

I've tried commenting out the proxy creation methods but it doesn't help.
Can you please give me any hints? I'm stuck with this problem and don't understand where to dig.

[eddumelendez]

Hi, fixed ports are totally unrelated to the issue and is not recommended by Testcontainers. You should configure S3 client properly.

AWS SDK V1

AmazonS3 s3 = AmazonS3ClientBuilder
                .standard()
                .withEndpointConfiguration(
                    new AwsClientBuilder.EndpointConfiguration(
                        localstack.getEndpoint().toString(),
                        localstack.getRegion()
                    )
                )
                .withCredentials(
                    new AWSStaticCredentialsProvider(
                        new BasicAWSCredentials(localstack.getAccessKey(), localstack.getSecretKey())
                    )
                )
                .build();

AWS SDK V2

S3Client s3 = S3Client
                .builder()
                .endpointOverride(localstack.getEndpoint())
                .credentialsProvider(
                    StaticCredentialsProvider.create(
                        AwsBasicCredentials.create(localstack.getAccessKey(), localstack.getSecretKey())
                    )
                )
                .region(Region.of(localstack.getRegion()))
                .build();

Instead of sharing snippets would be great if you can share a minimal project that reproduces the issue attaching the archive in the discussion or a github repository.

Due to the lack of information, I am guessing that VALID_DATA_FILE is a shell script and DOCKER_FILE_PATH_TO_VALID_FILE points to one of the hooks? The reason I say this is because of this line Execution of "start_runtime_components" took 902.54ms. If that's the case updating the wait strategy should fix it. See this gist

Also, Can you share the Testcontainers version are you using?

[akulik512]

Hello @eddumelendez
Yeah, I know about fixed ports but in my situation, I don't have any options.
Yep, I'm already configuring my AmazonS3 client as you pointed out above.
Sorry, let me share more information.

1. Here are my variables (it's just file names I have in test/resources):

public static final String VALID_DATA_FILE = "channels.csv";
public static final String INVALID_DATA_FILE = "channels_bad.csv";

private static final String DOCKER_FILE_PATH_TO_VALID_FILE = "/tmp/" + VALID_DATA_FILE;
private static final String DOCKER_FILE_PATH_TO_INVALID_FILE = "/tmp/" + INVALID_DATA_FILE;

2. Testcontainers version: 1.19.1
3. I also working on a minimal project that reproduces the issue.

Probably there is one important thing. This is how I attach Docker to my tests. This is the Docker socket mounting setup:

agent {
    docker {
        image 'maven:3.8.5-eclipse-temurin-17-alpine'
        args '-v ${JENKINS_HOME}/.m2:/root/.m2 -v /etc/maven:/root/maven-conf:ro -v /var/run/docker.sock:/var/run/docker.sock'
    }
}

BTW, @eddumelendez I have a question about your gist. I see there is an aws-script.sh script to create a bucket but in the shared snippet, you never execute it. Maybe it should be like this?

@Container
    private final LocalStackContainer localstack =
            new LocalStackContainer(DockerImageName.parse("localstack/localstack").withTag("0.14.2"))
                    .withCopyFileToContainer(MountableFile.forClasspathResource("/localstack/aws-script.sh"), "/docker-entrypoint-initaws.d")
                    .withServices(Service.S3)
                    .waitingFor(Wait.forLogMessage(".*s3 bucket created\\.\n", 1));

Could you please take a look at it? I guess it can be something that I need to try instead of my way:

Container.ExecResult createBucketExecResult1 = S3_CONTAINER.execInContainer("awslocal", "s3", "mb", inputBucketAddress);

[eddumelendez]

Please use the thread when replying to an specific message. I think the path has changed in recent localstack versions. Will wait for the project that reproduce the issue.