Can't connect to Kafka on Apple Silicon

[Crowelling]

Hello.

When i am trying to run kafka on apple with silicon m1 chip, i have no errors on logs, but can't connect to kafka container. Tried to use simple docker-compose.yml:

services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    links:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "test-topic:5:2"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

No errors on logs, just telling that waiting for kafka to be ready and that's it. Can't connect to localhost:9092

Maybe somebody knows where is the problem?

[shubham1g5]

I am facing the same issue on M1 mac

[jdamiani27]

wurstmeister/kafka currently does not provide arm builds and it is likely that the qemu emulation layer is failing (many users encountering segfaults in various amd64 images docker/for-mac#5123)

The openjdk base image does provide arm builds so I've simply built my own image locally from this repo like so (additionally using the official zookeeper image with arm builds):

version: '3'

services:
  zookeeper:
    image: zookeeper
    ports:
      - "2181:2181"

  kafka:
    build: ./kafka-docker
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

[chetanyachopra22]

wurstmeister/kafka currently does not provide arm builds and it is likely that the qemu emulation layer is failing (many users encountering segfaults in various amd64 images docker/for-mac#5123)

The openjdk base image does provide arm builds so I've simply built my own image locally from this repo like so (additionally using the official zookeeper image with arm builds):

version: '3'

services:
  zookeeper:
    image: zookeeper
    ports:
      - "2181:2181"

  kafka:
    build: ./kafka-docker
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

@jdamiani27
Have you kept this broker and your zookeeper on same network ... or have done some extra configurations to run this broker with this cluster.. in short can I see your kafka-docker.yml or will be great if you can show configurations of both zookeeper and kafka 👍🏻

[Johny-Ch]

I was able to get it working by building my own image locally from this repo and using it to pull up kafka on my M1 MAC

[7rulnik]

For now you can use https://hub.docker.com/r/niks123123/kafka

[boratanrikulu]

Clone wurstmeister/kafka-docker project to your local and build the image.
docker build -t kafka .

Then use this image, works fine.

[mangas]

Changing the image from openjdk to azul/zulu-openjdk-alpine:8u292-8.54.0.21.

@wurstmeister would you be interested in bumping the base image to a newer one? cc @huang-x-h (not sure who to ping here)

PR raised, it's working for me, a few other people have also 👍 'd it

[mangas]

@wurstmeister is this something you'd be interested in merging?

[prescottprue]

For those looking to get something working across both M1 and Intel for Confluent Kafka, we have had success with the following:

---
version: '2'
services:
  zookeeper:
    image: antrea/confluentinc-zookeeper:6.2.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - '2181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: antrea/confluentinc-kafka:6.2.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
      - '9101:9101'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      # NOTE: Not supported by current container
      # KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      # TODO: Uncomment once enable schema registry
      # KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

[zhaojc]

For those looking to get something working across both M1 and Intel for Confluent Kafka, we have had success with the following:

---
version: '2'
services:
  zookeeper:
    image: antrea/confluentinc-zookeeper:6.2.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - '2181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: antrea/confluentinc-kafka:6.2.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
      - '9101:9101'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      # NOTE: Not supported by current container
      # KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      # TODO: Uncomment once enable schema registry
      # KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

Error response from daemon: Get "https://docker.elastic.co/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)