Could not connect to Ryuk during running java tests via a docker container.

[ggili]

Hi All

We are running our tests within a maven-java container during our gitlab pipeline. Recently we had to migrate to the newest docker version (since gitlab runner got upgraded. ) and we are getting the following connection issue between the two containers (java-maven and ryuk).

java.lang.IllegalStateException: Could not connect to Ryuk at 172.17.0.1:56144 at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:227) at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:219) at org.testcontainers.DockerClientFactory$1.getDockerClient(DockerClientFactory.java:101) at com.github.dockerjava.api.DockerClientDelegate.authConfig(DockerClientDelegate.java:107) at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:316) at TestMariaDB.mariaDBContainer(TestMariaDB.java:14) at TestMariaDB.name(TestMariaDB.java:20) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

With some trials I was able to reproduce it locally (after I ugraded to docker 20.x ; I must say all worked find with docker 19.x but not an option anymore on our gitlabserver to stay on 19.x).

To give you an idea some snippets we do and can be reproduced.

docker run -v ~/work:/build 3.8.3-jdk-11 mvn test -f /build/sample/pom.xml
I am using on my mac (but fails also on linux gitlab runner)

 docker -v
Docker version 20.10.8, build 3967b7d

I was investigation what goes wrong and found:

When from 'maven-container' wants to contact ryuk via the gateway 172.17.0.1:56144 the connection cannot be established. But I could curl the ryuk container myself using the container's specific ip and 8080 port.
curl 172.18.0.2:8080 ACK ACK ACK ACK

Checking the code of test-containers I've seen its using the

public final class ResourceReaper {
.....
public static String start(DockerClient client) {
....

   String host = containerState.getHost();
        Integer ryukPort = containerState.getFirstMappedPort();
.....

Now I started wondering and playing around what goes wrong between the two versions of docker.

If I start ryuk manually with -p 1234:8080 I can easily curl ryuk from mvn-java container

Docker inpsect:

 "HostConfig": {
      "Binds": [
        "/var/run/docker.sock:/var/run/docker.sock"
      ],
      "ContainerIDFile": "",
      "LogConfig": {
        "Type": "json-file",
        "Config": {}
      },
      "NetworkMode": "default",
      "PortBindings": {
        "8080/tcp": [
          {
            "HostIp": "",
            **"HostPort": "1234"**
          }
        ]
      },

docker run -v /var/run/docker.sock:/var/run/docker.sock -p 1234:8080 testcontainers/ryuk:0.3.3
curl 172.17.0.1:56144 ACK ACK ACK ACK

Now the problem comes if I start ryuk with a random port which will be the case what is done via test-containers.

docker run -v /var/run/docker.sock:/var/run/docker.sock -p 8080 testcontainers/ryuk:0.3.3
`
Docker inspect

"HostConfig": {
           "Binds": [
               "/var/run/docker.sock:/var/run/docker.sock"
           ],
           "ContainerIDFile": "",
           "LogConfig": {
               "Type": "json-file",
               "Config": {}
           },
           "NetworkMode": "default",
           "PortBindings": {
               **"8080/tcp": [
                   {
                       "HostIp": "0.0.0.0",
                       "HostPort": "0"
                   }**
               ]
           },
...............
NetworkSettings": {
           "Bridge": "",
           "SandboxID": "546e2822939191384c59a4c4686f677f0137fdead848adaa1c22548c2dfee782",
           "HairpinMode": false,
           "LinkLocalIPv6Address": "",
           "LinkLocalIPv6PrefixLen": 0,
           "Ports": {
               "8080/tcp": [
                   {
                       "HostIp": "0.0.0.0",
                       "HostPort": "57226"
                   }
               ]
           },
           "SandboxKey": "/var/run/docker/netns/546e28229391",
           "SecondaryIPAddresses": null,
           "SecondaryIPv6Addresses": null,
           "EndpointID": "d553bcc6bf272dac330f90315caacbb5b77673f64bf1ca34cd80dd01e9dd8480",
           "Gateway": "172.17.0.1",
           "GlobalIPv6Address": "",
           "GlobalIPv6PrefixLen": 0,
           "IPAddress": "172.17.0.2",

Notice that the HostIp its 0.0.0.0 and HostPort: 0 on HostConfig. Having this kind of setup with 'random' port start I could not
curl anymore the ryuk (with newer docker) from my maven container curl 172.17.0.1:57226 only its container ip curl 172.17.0.2:8080.

I suspect that there is something has to do with the way the newer docker is handling the random ports.

"PortBindings": {
                **"8080/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "0"
                    }**
                ]

I forced a port addition during ryuk container creation and that did the trick ryuk was reachable. As well when the newer docker engine created the ryuk container. Actually it filled the HostPort to "HostPort": "1234";

String ryukContainerId = client.createContainerCmd(ryukImage)
                .withHostConfig(
                    new HostConfig()
                        .withAutoRemove(true)
                        .withPortBindings(new PortBinding(**Ports.Binding.bindPort(1234),** ryukExposedPort))
                )
                .withExposedPorts(ryukExposedPort)
                .withName("testcontainers-ryuk-" + DockerClientFactory.SESSION_ID)
                .withLabels(Collections.singletonMap(DockerClientFactory.TESTCONTAINERS_LABEL, "true"))
                .withBinds(binds)
                .withPrivileged(TestcontainersConfiguration.getInstance().isRyukPrivileged())
                .exec()
                .getId(); 

Sure we could work it around by disabling ryuk and doe some work around for our mariadb container like assining a port or getting the container IP. But perhaps there is some settings / fixes which can be done fixing this.
Any thoughts ?

Thanks
Alpar

[rosell]

Hi!

I have the same problem with another version of docker:

Client:
 Cloud integration: v1.0.20
 Version:           20.10.10
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        b485636
 Built:             Mon Oct 25 07:43:15 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.10
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       e2f740d
  Built:            Mon Oct 25 07:41:30 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.11
  GitCommit:        5b46e404f6b9f661a205e28d59c982d3634148f8
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[ggili]

As a workaround we disabled ryuk for the moment during our pipeline and override the Mariadb container URL .

 final MariaDBContainer mariaDBContainer = new MariaDBContainer(MARIA_DB_DOCKER_IMAGE){
                @Override
                public String getJdbcUrl() {
                    if (DockerClientConfigUtils.IN_A_CONTAINER) {
                        final String ipAddress = getContainerInfo().getNetworkSettings().getIpAddress();
                        final String port = getExposedPorts().get(0).toString();
                        String additionalUrlParams = constructUrlParameters("?", "&");
                        return "jdbc:mariadb://" + ipAddress + ":" + port + "/" + getDatabaseName() + additionalUrlParams;
                    } else {
                       return super.getJdbcUrl();
                    }
                }
            }

Hope this helps somebody. I did not had time to dig in more .. perhaps a bit later.

I think docker changed the way to expose random ports (or its a regression).
i.e.
docker run -v /var/run/docker.sock:/var/run/docker.sock -p 8080 testcontainers/ryuk:0.3.3 This will create a random port but its not reachable.

If explicitly we specify then it will be accessible.

docker run -v /var/run/docker.sock:/var/run/docker.sock -p 1234:8080 testcontainers/ryuk:0.3.3