Invalid Username or Password

[pedro93]

Hello,

I've started hitting an Invalid Username or Password error for the past hours when trying to run tests locally. No changes on the repository have been, does test-containers have some dependency on a server/service that has been changed in the past 24 hours?

Here is the full stack trace of a test that has been failing me locally.

09:08:19.576 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy from ~/.testcontainers.properties, will try it first
09:08:19.656 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Will use 'okhttp' transport
09:08:20.005 [ducttape-0] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Pinging docker daemon...
09:08:20.034 [ducttape-0] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: org.testcontainers.dockerclient.transport.okhttp.OkHttpDockerCmdExecFactory$1@60e956ac
09:08:20.143 [main] INFO org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
09:08:20.159 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='pedro.cardoso'
    registryPassword='<password>'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=pedro.cardoso,registryPassword=5KRA3BG3,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/pedro.cardoso/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

09:08:20.161 [main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is localhost
09:08:20.164 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.exec.InfoCmdExec@60641ec8
09:08:20.349 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.exec.VersionCmdExec@297ea53a
09:08:20.361 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 18.06.0-ce
  API Version: 1.38
  Operating System: Ubuntu 18.04.3 LTS
  Total Memory: 31786 MB
09:08:20.366 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=quay.io/testcontainers/ryuk:0.2.3,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0,execution=com.github.dockerjava.core.exec.ListImagesCmdExec@61526469]
09:08:20.385 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.3
09:08:20.386 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /home/pedro.cardoso/.docker/config.json (exists) and commandPathPrefix: 
09:08:20.395 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - registryName [quay.io] for dockerImageName [quay.io/testcontainers/ryuk:0.2.3]
09:08:20.396 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - no matching Auth Configs - falling back to defaultAuthConfig [AuthConfig{username=pedro.cardoso, password=hidden non-blank value, auth=blank, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]
09:08:20.397 [main] DEBUG org.testcontainers.dockerclient.auth.AuthDelegatingDockerClientConfig - Effective auth config [AuthConfig{username=pedro.cardoso, password=hidden non-blank value, auth=blank, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]
09:08:22.430 [tc-okhttp-stream-467632528] ERROR com.github.dockerjava.core.async.ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://quay.io/v2/testcontainers/ryuk/manifests/0.2.3: unauthorized: Invalid Username or Password"}

	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:276)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.lambda$executeAndStream$1(OkHttpInvocationBuilder.java:292)
	at java.lang.Thread.run(Thread.java:745)

com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://quay.io/v2/testcontainers/ryuk/manifests/0.2.3: unauthorized: Invalid Username or Password"}


	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:276)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.lambda$executeAndStream$1(OkHttpInvocationBuilder.java:292)
	at java.lang.Thread.run(Thread.java:745)

Thank you.

[rnorth]

Hi @pedro93,

I'm afraid nothing has changed as far as I can tell. Testcontainers pulls an image from the quay.io registry (https://quay.io/repository/testcontainers/ryuk), but there have been no changes there. I can pull that image, unauthenticated, just fine, and our CI is doing the same.

From your logs, it looks like Testcontainers has found some credentials in your environment, but these should only be applied to pulls from index.docker.io:

09:08:20.396 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - no matching Auth Configs - falling back to defaultAuthConfig [AuthConfig{username=pedro.cardoso, password=hidden non-blank value, auth=blank, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]

I'm afraid I'm not quite sure what is going on. Please could you try temporarily moving your /home/pedro.cardoso/.docker/config.json file somewhere else, and see if that makes a difference?

[pedro93]

Hello @rnorth

Thank you for getting back to me so quickly, I've moved the config file as you have suggested but test-containers is somehow available to still find the config at least for logging purposes.

09:47:32.446 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='pedro.cardoso'
    registryPassword='<password>'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=pedro.cardoso,registryPassword=5KRA3BG3,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/pedro.cardoso/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

09:50:12.059 [main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is localhost
09:50:12.061 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.exec.InfoCmdExec@60641ec8
09:50:12.248 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.exec.VersionCmdExec@297ea53a
09:50:12.260 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 18.06.0-ce
  API Version: 1.38
  Operating System: Ubuntu 18.04.3 LTS
  Total Memory: 31786 MB
09:50:12.265 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=quay.io/testcontainers/ryuk:0.2.3,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0,execution=com.github.dockerjava.core.exec.ListImagesCmdExec@61526469]
09:50:12.283 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.3
09:50:12.283 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /home/pedro.cardoso/.docker/config.json (does not exist) and commandPathPrefix: 
09:50:12.283 [main] WARN org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.3, configFile: /home/pedro.cardoso/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/pedro.cardoso/.docker/config.json (No such file or directory)
09:50:12.284 [main] DEBUG org.testcontainers.dockerclient.auth.AuthDelegatingDockerClientConfig - Effective auth config [AuthConfig{username=pedro.cardoso, password=hidden non-blank value, auth=blank, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]
09:50:13.541 [tc-okhttp-stream-949673757] ERROR com.github.dockerjava.core.async.ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://quay.io/v2/testcontainers/ryuk/manifests/0.2.3: unauthorized: Invalid Username or Password"}

	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:276)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.lambda$executeAndStream$1(OkHttpInvocationBuilder.java:292)
	at java.lang.Thread.run(Thread.java:745)

com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://quay.io/v2/testcontainers/ryuk/manifests/0.2.3: unauthorized: Invalid Username or Password"}


	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:276)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.lambda$executeAndStream$1(OkHttpInvocationBuilder.java:292)
	at java.lang.Thread.run(Thread.java:745)

However, the process now seems unable to lookup the config?

09:50:12.283 [main] WARN org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.3, configFile: /home/pedro.cardoso/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/pedro.cardoso/.docker/config.json (No such file or directory)

[pedro93]

More info:
I am able to pull the quay image 0.2.3 manually via terminal but not using test-containers.
If I do pull the quay image then the test executes as expected.

Conclusion, this seems to be some auth problem surrounding the quay registry, disabling test-containers with quay may be an option.
That said, how can I disable ryuk? I've tried with System.setProperty("TESTCONTAINERS_RYUK_DISABLED", "true"); to no avail.

[rnorth]

TESTCONTAINERS_RYUK_DISABLED is an environment variable, not a system property. You could disable it, but I'd caution against it - you may end up with stopped containers left over (particularly if the test JVM doesn't manage to exit cleanly).

If AuthConfig is still resolving configuration despite you moving the config file it implies that you must either have system properties or configuration that docker-java is picking up. Are you sure that none of the various configuration settings for docker-java are in effect?

[pedro93]

TESTCONTAINERS_RYUK_DISABLED is an environment variable, not a system property. You could disable it, but I'd caution against it - you may end up with stopped containers left over (particularly if the test JVM doesn't manage to exit cleanly).

A very valid point, since I want run these tests both locally and in gitlab-ci I am not very concerned at the moment since locally is somewhat sandboxed and gitlab-ci is a docker-in-docker so ruyk has to be disabled anyway.

If AuthConfig is still resolving configuration despite you moving the config file it implies that you must either have system properties or configuration that docker-java is picking up. Are you sure that none of the various configuration settings for docker-java are in effect?

Thank you for the link, indeed I do have username and password credentials in $HOME/.docker-java.properties but that is for another project.
What I do not understand is why having these configs would mean that I am unable to access the image through test-containers given that I can do so through the terminal.

[rnorth]

Thank you for the link, indeed I do have username and password credentials in $HOME/.docker-java.properties but that is for another project.
What I do not understand is why having these configs would mean that I am unable to access the image through test-containers given that I can do so through the terminal.

Testcontainers uses docker-java, but the command line docker utility does not. With Docker it's the client that handles registry auth - i.e. Testcontainers+docker-java is finding authentication details, but the docker CLI ignores $HOME/.docker-java.properties and connects to the registry without auth.

The $HOME/.docker-java.properties credential type is very old and doesn't support more than one set of credentials (i.e. it doesn't allow different credentials for each registry). This is, I think, why it is trying to use your other credentials for pulling from quay.io.

I suspect all these problems might go away if your migrated your docker auth details into Docker's ~/.docker/config.json file, which does support credentials-per-registry. You could therefore have credentials set (in an auth block) for your private registry, but leave every other registry in unauthenticated mode by not having an entry for them.

[pedro93]

Confirmed that $HOME/.docker-java.properties was the culprit in all of this, thank you so much for your help @rnorth !

[nrmaridu]

@pedro93 Did you delete the .docker-java.properties file to fix this issue?

[pedro93]

I did @NageswaraRaoMaridu

[dermoritz]

i have the same issue on a fresh computer (there is no .docker-java.properties). manually pulling the images works but is cumbersome. i get this for every image needed.

[kiview]

@dermoritz You are likely running into #5121, try out the suggested workarounds:
#5121 (comment)