Latest tag fails with "Command [/usr/local/bin/dub path /etc/kafka/ writable] FAILED !"

[astenman]

Tried to update my containers today, but gets this error

[salsa-dev]

same thing: Command [/usr/local/bin/dub path /etc/kafka/ writable] FAILED !
only those that pushed one month ago work as expected

[DRoppelt]

Happens here too, 7.0.1 (which latest points to) has this error while using 7.0.0 does not

[DRoppelt]

Here is a diff between 7.0.0 and 7.0.1, some changes were done to folder permissions, but I am not knowledgeable to say why/which change caused this

v7.0.0...v7.0.1

[achref-riahi]

Happens here too, 7.0.1 (which latest points to) has this error while using 7.0.0 does not

Perfect :D This is really working out for me.

[mihajloracic]

Probably something with log4j or something like that. I downgraded 6.x.x version and worked fine for me. In couple of days latest tag will work again.

[ThanksForAllTheFish]

@mihajloracic I see the same with 6.2.2 as well

[andrewegel]

Hi Folks - Apologies for the disruption, we're putting in a fix for all latest released docker images in #129 - You'll see a new ${VERSION}-2-${OS_SUFFIX} and ${VERSION} tags appear on the docker hub by End of Week.

The original intention of #116 is to make these images more secure by not having world writeable directories. This issue highlights a testing gap that I've raised with our internal Quality Engineering teams.

[vianini]

For 7.0.0 working. i'm using aks and image: confluentinc/cp-kafka:latest if have a problem they automatic change.

[vijaycse]

Is there any workaround until then? All our builds are failing. we pinned the versions as far as 6.XX

[vijaycse]

@vianini Could you please explain your interim fix?

[andrewegel]

Is there any workaround until then? All our builds are failing. we pinned the versions as far as 6.XX

• Pin to the last version that worked for you. That was likely 6.2.1.
• Derive your own image with chown appuser:appuser -R /etc/kafka ...

[sv3ndk]

I'm using that image as part of docker-compose. There, just setting user: "0" fixed the issue for me. If you use docker run, then the --user=0 argument should achieve the same effect.

What's happening is that /etc/kafka is owned by root, and at startup Kafka checks that is has write access there.

On my computer I'm running my container in rootless mode with Podman, so that root UID inside it actually is owned by my own user, not the root of the host.

[lvicentesanchez]

This also affects the kafka connect image.

[mikaello]

This also affects the kafka connect image.

Also schema-registry-images, this was reported 4 days ago: confluentinc/schema-registry-images#48 (comment) , and explains the problem arising with OpenShift.

[andrewegel]

I'm using that image as part of docker-compose. There, just setting user: "0" fixed the issue for me. If you use docker run, then the --user=0 argument should achieve the same effect.

Well user 0 or root has access to everything, so of course that works, but now you're running a container as root, which is an anti-pattern.

Also schema-registry-images, this was reported 4 days ago: confluentinc/schema-registry-images#48 (comment) , and explains the problem arising with OpenShift.

I've asked that this be tracked as a separate issue, as this scenario is when things are ran in an OpenShift cluster, where-as things here are just broken out-of-the-box, which we're releasing a fix for, but likely won't resolve things when ran in an OpenShift cluster.

[sv3ndk]

Well user 0 or root has access to everything, so of course that works, but now you're running a container as root, which is an anti-pattern.

I agree. It's only root inside the container though, I'm running my containers in root-less mode, so a container is really just a child process started by my own user, UID 0 inside it maps to my own UID on the host, whereas any other UID inside the container maps to yet another separate unprivileged user. I believe it's one of safest way to execute containers. Here is some background for info: https://www.redhat.com/en/blog/understanding-root-inside-and-outside-container

Though yeah, avoid root even inside is certainly better.

Anyhow, thanks also for your other comments, great to hear another image is on the way with adapted permissions on that folder.

[tonyfosterdev]

USER root
RUN chown appuser:appuser -R /etc/kafka
USER appuser

RUN confluent-hub install --no-prompt ...

Any reason not to just temporarily switch to root, apply chown, and switch back to appuser? This built for me.

[andrewegel]

Any reason not to just temporarily switch to root, apply chown, and switch back to appuser? This built for me.

Thats in essence, what this PR is doing #129

We're rebuilding images with these changes, and sending them through some sanity testing before we release new tags. By end of week you should see updated tags.

[tonyfosterdev]

@andrewegel I only run the cp-kafka-connect-base image from the confluent stack. I updated it to 7.0.1, but information endpoint still shows version 7.0.0-ccs:

{
  "version": "7.0.0-ccs",
  "commit": "c6d7e3013b411760",
  "kafka_cluster_id": "..."
}

Is that expected?

[andrewegel]

I only run the cp-kafka-connect-base image from the confluent stack. I updated it to 7.0.1, but information endpoint still shows version 7.0.0-ccs:

That snip-it doesn't provide a lot of context so I can't really answer, but I think thats something connecting to a kafka broker running the Confluent Community Server version 7.0.0.

[andrewegel]

Hello Folks - I have pushed new tags to the docker-hub with #129

The following tags have been updated:

• latest, latest-ubi8
• 7.0.1, 7.0.1-1-ubi8
• 6.2.2, 6.2.2-2-ubi8
• 6.1.4, 6.1.4-2-ubi8
• 6.0.5, 6.0.5-2-ubi8

I miss-entered a revision value for 7.0.1, so it "clobbered" the 7.0.1-1-ubi8 tag (I should have pushed a revision 2 7.0.1-2-ubi - I'll be flogging myself for this), but hey, it was broken anyways.

Please re-pull these tags (docker ... --pull always should do the trick) or update your Dockerfiles to pull the new revision tags.

[andrewegel]

This is what I've been using to reproduce the issue, and its functioning now:

% docker run --pull always --rm -it -e "CONNECT_BOOTSTRAP_SERVERS=foo:9092" -e "CONNECT_GROUP_ID=foo" -e "CONNECT_CONFIG_STORAGE_TOPIC=foo" -e "CONNECT_OFFSET_STORAGE_TOPIC=foo" -e "CONNECT_STATUS_STORAGE_TOPIC=foo" -e "CONNECT_KEY_CONVERTER=foo" -e "CONNECT_VALUE_CONVERTER=foo" -e "CONNECT_REST_ADVERTISED_HOST_NAME=foo" confluentinc/cp-kafka-connect-base:latest

latest: Pulling from confluentinc/cp-kafka-connect-base
2a99c93da168: Already exists
4418ace46c3d: Already exists
8abef5b25a9b: Already exists
ca529bf52797: Already exists
8d1c11c3d930: Already exists
b98516e7304f: Already exists
f387683c8792: Already exists
7cb8d248bef1: Already exists
6e572ab1ebf8: Already exists
0c076ef132d6: Already exists
88ab77aaac56: Already exists
15b1e0182e16: Pull complete
1329d2ed9ad9: Pull complete
Digest: sha256:91f08f01e1f919d0525d9f75fec03610ddbe7d925c959469c0694544d0f9db9f
Status: Downloaded newer image for confluentinc/cp-kafka-connect-base:latest
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ...
===> Check if Kafka is healthy ...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (io.confluent.admin.utils.cli.KafkaReadyCommand).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

% docker run --pull always -e KAFKA_ZOOKEEPER_CONNECT=localhost:2181 -e KAFKA_ADVERTISED_LISTENERS=BROKER://localhost:9092  confluentinc/cp-kafka:latest
latest: Pulling from confluentinc/cp-kafka
2a99c93da168: Pull complete
4418ace46c3d: Pull complete
8abef5b25a9b: Pull complete
ca529bf52797: Pull complete
8d1c11c3d930: Pull complete
b98516e7304f: Pull complete
f387683c8792: Pull complete
7cb8d248bef1: Pull complete
6e572ab1ebf8: Pull complete
0c076ef132d6: Pull complete
88ab77aaac56: Pull complete
Digest: sha256:8946c3da0dddc3dab7a37ca8a970d1f5cd645237e0337cc6578fbcdb2c588016
Status: Downloaded newer image for confluentinc/cp-kafka:latest
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ...
===> Check if /var/lib/kafka/data is writable ...
===> Check if Zookeeper is healthy ...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (io.confluent.admin.utils.cli.ZookeeperReadyCommand).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
^C^C^C^C

[astenman]

Thank you @andrewegel ! It is working for me now.

[tnagel1]

Do we know when this will be also released to dockerhub?! If i pull the image from docker hub i get still the same error
Command [/usr/local/bin/dub path /etc/kafka-connect/ writable] FAILED !

[andrewegel]

@tnagel1 @david-dufour-kw : I've released fixed docker images that resolve the original issue. If you find that the latest tags on docker hub don't resolve the problem you are encountering (and please check you are pulling the new image tags), please provide more details in a separate issue:

• The image with the issue
• The docker run ... command you are using
• Alternatively a docker-compose.yml file that displays the problem.

Once I can reproduce locally, I can work on fixing things.

[david-dufour-kw]

@tnagel1 @david-dufour-kw : I've released fixed docker images that resolve the original issue. If you find that the latest tags on docker hub don't resolve the problem you are encountering (and please check you are pulling the new image tags), please provide more details in a separate issue:

* The image with the issue

* The `docker run ...` command you are using

* Alternatively a docker-compose.yml file that displays the problem.

Once I can reproduce locally, I can work on fixing things.

Sorry, I was mistaken. I delete the cp-kafka images from my local Docker and was able to resolve the issue.

[andrewegel]

Sorry, I was mistaken. I delete the cp-kafka images from my local Docker and was able to resolve the issue.

Thats why I said here:

Please re-pull these tags (docker ... --pull always should do the trick) or update your Dockerfiles to pull the new revision tags.

The --pull always checks for an updated image against your registry.

[Zefirrat]

latest-ubi8 still has this error. It's easy to fix with chown but when you cant access production machine or for others developers that testing your things it's annoying more even hard.

kafka_1      | ===> Running preflight checks ...
kafka_1      | ===> Check if /var/lib/kafka/data is writable ...
kafka_1      | Command [/usr/local/bin/dub path /var/lib/kafka/data writable] FAILED !

  kafka:
    image: confluentinc/cp-kafka:latest-ubi8
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: SASL_PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:SASL_PLAINTEXT
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      ZOOKEEPER_SASL_ENABLED: "false"
      KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf"
      KAFKA_INTER_BROKER_LISTENER_NAME: SASL_PLAINTEXT
      KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /kafka_server_jaas.conf:/etc/kafka/kafka_server_jaas.conf
      - /data:/var/lib/kafka/data

volumes:
  kafka-data:
    driver: local

[ghost]

Seting user: "0:0" in docker-compose fix my problem:

 zookeeper:
    image: confluentinc/cp-zookeeper:7.1.0
    user: "0:0"
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./data/zookeeper/data:/var/lib/zookeeper/data
      - ./data/zookeeper/log:/var/lib/zookeeper/log
    networks:
      - api-network

  kafka:
    image: confluentinc/cp-kafka:7.1.0
    user: "0:0"
    container_name: kafka
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

[ekeric13]

I am getting this error for version 7.1.1 when I use ubuntu, but not when I use mac.
Is the fix for ubuntu to simply use the -1-ubi8 tag?

edit: yeah that isn't working

edit2: I think it has something to do with docker userns-remap + modifying /etc/subuid for the os i am working with

edit3: able to fix the issue using this article: https://www.jujens.eu/posts/en/2017/Jul/02/docker-userns-remap/

[jinnabaalu]

Update your docker-compose with user: root this will resolve the issues.

kafka-connect:
    image: confluentinc/cp-kafka-connect-base:6.2.0
    container_name: kafka-connect
    user: root
   ......

[c3-yuhsuan]

@andrewegel What is being written to etc/kafka that requires filesystem to enable writing access?

[c3-yuhsuan]

Issue Occur

"uid=1000660005(1000660000) gid=1000660005 groups=1000660005, Command [/usr/local/bin/dub path /etc/kafka/ writable] FAILED !"

I used -3-ubi version. I still received this error on OpenShift cluster. Is this issue being fixed?

[DilLip-Chowdary-Codes]

Getting the similar error in confluentinc/cp-kafka:7.2.1-1-ubi8.amd64

===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ... 
===> Check if /var/lib/kafka/data is writable ...
Command [/usr/local/bin/dub path /var/lib/kafka/data writable] FAILED !

[sugan2111]

Thank you @andrewegel ! It is working for me now.

Hi @astenman, can you please share the version of cp-kafka, that worked for you .

[sugan2111]

Getting the similar error in confluentinc/cp-kafka:7.2.1-1-ubi8.amd64

===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ... 
===> Check if /var/lib/kafka/data is writable ...
Command [/usr/local/bin/dub path /var/lib/kafka/data writable] FAILED !

Hi @DilLip-Chowdary-Codes , I am using the same version and getting the same issue. did you get to fix it?

[c3-yuhsuan]

We fixed that issue by building our own image

[sugan2111]

Hi @yuhsuan0323 could you provide more details on how to do that? Apparently few have mentioned that 7.0.0 version worked but did not work when I tried that.

[ACodingfreak]

Hitting same issue with the latest build

$ sudo docker container ls -a
CONTAINER ID   IMAGE                             COMMAND                  CREATED         STATUS         PORTS                                                                                                              NAMES
a80d7ba05ddd   confluentinc/cp-kafka             "/etc/confluent/dock…"   2 minutes ago   Up 2 minutes   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp, 0.0.0.0:29092-29093->29092-29093/tcp, :::29092-29093->29092-29093/tcp   broker
f7df5e36b8ad   confluentinc/cp-zookeeper         "/etc/confluent/dock…"   2 minutes ago   Up 2 minutes   2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp                                                      zookeeper

~/Kafka$ sudo docker run -it --rm --network kafka_kafka confluentinc/cp-kafka
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
KAFKA_ZOOKEEPER_CONNECT is required.
Command [/usr/local/bin/dub ensure KAFKA_ZOOKEEPER_CONNECT] FAILED !

[jared-yan]

confluentinc/cp-kafka:7.4.0

===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
Running in Zookeeper mode...
===> Running preflight checks ...
===> Check if /var/lib/kafka/data is writable ...
Command [/usr/local/bin/dub path /var/lib/kafka/data writable] FAILED !

sudo docker start kafka && sudo docker exec -u root -it kafka chown -R appuser:appuser /var/lib/kafka

[bidoudan]

I was struggling with the same problem, I tried both oldest and newer versions without any result,
The solution for me was to make volumes directory writable for all users 'chmod 777 ./volumes/zookeeper/data' locally...