ARM based docker image #12944

vkuznecovas · 2021-11-23T13:07:41Z

Is your enhancement request related to a problem? Please describe.
I'd like to see an ARM based docker image for pulsar. The problem is that under new macs the pulsar containers have to be run under emulation which makes them horribly slow.

Describe the solution you'd like
ARM based docker image should be available in the apachepulsar/pulsar repo.

Describe alternatives you've considered
Building a native image has not proved trivial, so ideally someone with more knowledge of the project would be able to create one.

michaeljmarshall · 2021-11-24T05:56:53Z

+1. This would be a good feature for end users. My only concern as a contributor is that it will increase build times. That shouldn't stop this feature, though.

Ideally, we'll be able to follow the instructions here: https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/. I haven't done a mutli-arch build using maven before (we build our docker images using maven), so that is one nuance.

kezhenxu94 · 2021-12-01T13:35:50Z

I'm glad to work on this if there is nobody already working on it? I've recently created a build job to build amd64/arm64 Docker images for Apache SkyWalking and can apply the experience here, perhaps.

codelipenghui · 2021-12-02T13:30:21Z

Thanks @kezhenxu94, I have assigned the issue to you. Looking forward to your PRs.

krissetto · 2021-12-07T14:03:49Z

I am also interested in support for ARM docker images, especially for testing locally on Apple Silicon based machines

kezhenxu94 · 2021-12-07T14:12:59Z

I am also interested in support for ARM docker images, especially for testing locally on Apple Silicon based machines

Thank you @krissetto !! Will let you know when I have a docker image for testing on ARM!

lorthirk · 2021-12-09T11:52:59Z

+1!

jibbolo · 2021-12-09T14:07:52Z

+1

kezhenxu94 · 2021-12-14T07:44:54Z

Hi all in this thread, I just pushed a multi-arch image to my own repo, anyone interested is welcome to pull and test it in your environment. https://hub.docker.com/repository/docker/kezhenxu94/pulsar/tags?page=1&ordering=last_updated

To test, just replace the docker image with kezhenxu94/pulsar, example:

docker run -it \                                                                                                                  
  -p 6650:6650 \
  -p 8080:8080 \
  --mount source=pulsardata,target=/pulsar/data \
  --mount source=pulsarconf,target=/pulsar/conf \
  kezhenxu94/pulsar \
  bin/pulsar standalone

Note this is not official and not well tested now, I'll test, polish and then open a PR so everyone can review on the build process.

Any feedback are also welcome and appreciated 🙇

vkuznecovas · 2021-12-15T09:09:47Z

Hi all in this thread, I just pushed a multi-arch image to my own repo, anyone interested is welcome to pull and test it in your environment. https://hub.docker.com/repository/docker/kezhenxu94/pulsar/tags?page=1&ordering=last_updated

To test, just replace the docker image with kezhenxu94/pulsar, example:
docker run -it \                                                                                                                  
  -p 6650:6650 \
  -p 8080:8080 \
  --mount source=pulsardata,target=/pulsar/data \
  --mount source=pulsarconf,target=/pulsar/conf \
  kezhenxu94/pulsar \
  bin/pulsar standalone
Note this is not official and not well tested now, I'll test, polish and then open a PR so everyone can review on the build process.

Any feedback are also welcome and appreciated 🙇

I've tested it out, seems to work great. Thanks!

krissetto · 2021-12-15T10:59:32Z

I have also started testing it on an M1 device, and it seems to work 😄 I haven't gone very deep into it yet though, so i'll be back with feedback if I encounter any specific issues with it. Thanks for the build! Ping when you have a PR too since i'm curious

kezhenxu94 · 2021-12-15T11:07:39Z

I have also started testing it on an M1 device, and it seems to work 😄 I haven't gone very deep into it yet though, so i'll be back with feedback if I encounter any specific issues with it. Thanks for the build! Ping when you have a PR too since i'm curious

Sure, will keep you all in this thread posted. Thanks all for testing. 🙇

tdrozdowski · 2021-12-29T17:49:22Z

I can also validate that this appears to work on M1 Pro Max based machine. Would love to see the PR so we can get an official image; this works for local development in the meantime (and eliminates yet another tie to an Intel based machine!). Thanks so much!

kezhenxu94 · 2021-12-29T17:57:25Z

Heads up: The last (hard) step is to build the pulsar client with multiple architectures. Everything else is ready. I have some ideas to build pulsar client in multi architectures but need some time to verify, which may be also ready in 2 weeks (due to public holidays and my personal stuffs)

mgagliardo91 · 2022-01-27T14:29:36Z

@kezhenxu94 thanks for working on this, our company relies on pulsar and a recent move to Mac M1 has led us to run into a ton of problems with local development. Would it be possible to share the Dockerfile so that we could temporarily build arm supported images for older versions of pulsar? If not, do you have a timeline so we can be ready to start switching out our images?

kezhenxu94 · 2022-01-27T15:33:15Z

@kezhenxu94 thanks for working on this, our company relies on pulsar and a recent move to Mac M1 has led us to run into a ton of problems with local development. Would it be possible to share the Dockerfile so that we could temporarily build arm supported images for older versions of pulsar? If not, do you have a timeline so we can be ready to start switching out our images?

I can draft a PR of what I've done tomorrow (in my time zone) so that you can try it out. Stay tune please.

kezhenxu94 · 2022-01-28T09:21:55Z

Hi, I've just drafted a PR #14005

krissetto · 2022-02-21T15:20:09Z

Are there any updates on this issue? I see the draft PR hasn't changed since last month's revision

github-actions · 2022-03-24T02:03:31Z

The issue had no activity for 30 days, mark with Stale label.

heesung-sn · 2022-04-20T03:54:55Z

For those of you who are stuck at this issue, I was able to create ARM docker images on my M1 Mac by the following simple change. The test dockers from the following builds passed the shade integration test on my laptop.

# docker builds
mvn clean install -DskipTests
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests

# shade test run
./build/run_integration_group.sh SHADE

hsohn@HeesungSohns-MacBook-Pro pulsar % git diff
diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
index 27fef14fb7..201ca40115 100644
--- a/docker/pulsar/Dockerfile
+++ b/docker/pulsar/Dockerfile
@@ -76,8 +76,12 @@ RUN pip3 install pyyaml==5.4.1
 # 4. /pulsar - hadoop writes to this directory
 RUN mkdir /pulsar && chmod g+w /pulsar

-ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64
-RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security
+RUN java_home=$(ls -d1 /usr/lib/jvm/java-17-openjdk-*) ; \
+    ln -s "$java_home" /usr/lib/jvm/java
+
+ENV JAVA_HOME /usr/lib/jvm/java
+RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java/conf/security/java.security
+
 ADD target/python-client/ /pulsar/pulsar-client

 ENV PULSAR_ROOT_LOGGER=INFO,CONSOLE
diff --git a/docker/pulsar/pom.xml b/docker/pulsar/pom.xml
index 048b3da7cb..d4aa4d2eb1 100644
--- a/docker/pulsar/pom.xml
+++ b/docker/pulsar/pom.xml
@@ -75,7 +75,7 @@
                   <executable>${project.basedir}/../../pulsar-client-cpp/docker/build-wheels.sh</executable>
                   <arguments>
                     <!-- build python 3.8 -->
-                    <argument>3.8 cp38-cp38 manylinux2014 x86_64</argument>
+                    <argument>3.8 cp38-cp38 manylinux2014 aarch64</argument>
                   </arguments>
                 </configuration>
               </execution>
diff --git a/docker/pulsar/scripts/install-pulsar-client.sh b/docker/pulsar/scripts/install-pulsar-client.sh
index 56702b10db..bf81ccad97 100755
--- a/docker/pulsar/scripts/install-pulsar-client.sh
+++ b/docker/pulsar/scripts/install-pulsar-client.sh
@@ -23,3 +23,9 @@ set -x
 PYTHON_MAJOR_MINOR=$(python3 -V | sed -E 's/.* ([[:digit:]]+)\.([[:digit:]]+).*/\1\2/')
 WHEEL_FILE=$(ls /pulsar/pulsar-client | grep "cp${PYTHON_MAJOR_MINOR}")
 pip3 install /pulsar/pulsar-client/${WHEEL_FILE}[all]
+
+# The following `update build-essential python3-dev` are added due to grpcio `cc` not found error
+# TODO: remove these lines if the error got resolved.
+# WARNING: currently grpcio build takes long.
+apt-get update
+apt-get -y install build-essential python3-dev
diff --git a/tests/docker-images/java-test-image/Dockerfile b/tests/docker-images/java-test-image/Dockerfile
index b7f12b7596..99f626f1f6 100644
--- a/tests/docker-images/java-test-image/Dockerfile
+++ b/tests/docker-images/java-test-image/Dockerfile
@@ -40,8 +40,10 @@ RUN sed -i "s|http://archive\.ubuntu\.com/ubuntu/|${UBUNTU_MIRROR:-mirror://mirr
      && apt-get -y dist-upgrade \
      && apt-get -y install openjdk-17-jdk-headless

-ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64
-RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security
+RUN java_home=$(ls -d1 /usr/lib/jvm/java-17-openjdk-*) ; \
+    ln -s "$java_home" /usr/lib/jvm/java
+ENV JAVA_HOME /usr/lib/jvm/java
+RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java/conf/security/java.security

 # /pulsar/bin/watch-znode.py requires python3-kazoo
 # /pulsar/bin/pulsar-managed-ledger-admin requires python3-protobuf
diff --git a/tests/docker-images/latest-version-image/Dockerfile b/tests/docker-images/latest-version-image/Dockerfile
index f5c6726831..d6adb5cfa8 100644
--- a/tests/docker-images/latest-version-image/Dockerfile
+++ b/tests/docker-images/latest-version-image/Dockerfile
@@ -29,8 +29,8 @@ RUN apt-get install -y procps curl git build-essential

 ENV GOLANG_VERSION 1.15.8

-RUN curl -sSL https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz \
-               | tar -C /usr/local -xz
+RUN export arch=$(uname -m | sed -r 's/aarch64/arm64/g' |  awk '!/arm64/{$0="amd64"}1'); \
+curl -sSL https://golang.org/dl/go$GOLANG_VERSION.linux-$arch.tar.gz \

 # RUN wget https://dl.google.com/go/go1.13.3.linux-amd64.tar.gz && tar -xvf go1.13.3.linux-amd64.tar.gz && mv go /usr/local
 # RUN export GOROOT=/usr/local/go && export GOPATH=$HOME/go && export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
(END)

rhenwood-arm · 2022-04-26T13:41:40Z

Hi All, I'm interested to learn if I can support this effort.

michaeljmarshall · 2022-07-09T06:28:01Z

Now that #15142 is merged, there are no technical blockers for this work to continue on in the Pulsar project. I'll follow up on this one to try to push this through for the 2.11 release.

michaeljmarshall · 2022-07-09T07:20:30Z

New mailing list thread: https://lists.apache.org/thread/52bncdx2j4j982m9xob85sjmsr23tgzt

tisonkun · 2022-08-18T08:20:32Z

This can be related to #16652 .

Also, when I build on Apple M1 (ARM64), I find that pulsar-client-cpp/docker/manylinux1/Dockerfile binds to x86_64 and thus the build will fail with:

ERROR: pulsar_client-2.11.0-cp38-cp38-manylinux1_x86_64.whl is not a supported wheel on this platform.

tisonkun · 2022-08-18T10:06:32Z

Trying to workaround with:

$ pulsar-client-cpp/docker/build-wheels.sh 3.8 cp38-cp38 manylinux2014 aarch64
$ mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests -DskipBuildPythonClient=true -DskipSourceReleaseAssembly=true

We should integrate this snippet into build scripts, though.

FAILED with Python client depends on grpcio = 1.27.2, which doesn't support aarch64. A newer version can help, see also grpc/grpc#25418 and grpc/grpc#21283.

cc @BewareMyPower @shibd @RobertIndie

After a closer look, it seems the failure is not directly on arch:

File "/tmp/pip-install-od2wltzl/grpcio/setup.py", line 149, in check_linker_need_libatomic
cc_test = subprocess.Popen(['cc', '-x', 'c++', '-std=c++11', '-'],
File "/usr/lib/python3.8/subprocess.py", line 858, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'cc'

Build output:

[INFO] --- docker-maven-plugin:0.40.2:build (default) @ pulsar-docker-image ---
[INFO] Building tar: /Users/chenzili/Brittani/pulsar/docker/pulsar/target/docker/apachepulsar/pulsar/tmp/docker-build.tar
[INFO] DOCKER> [apachepulsar/pulsar:latest]: Created docker-build.tar in 2 seconds 
[INFO] DOCKER> Step 1/28 : FROM busybox as pulsar
[INFO] DOCKER> 
[INFO] DOCKER> ---> a5ab4ab35b15
[INFO] DOCKER> Step 2/28 : ARG PULSAR_TARBALL
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> f654625fc5f5
[INFO] DOCKER> Step 3/28 : ADD ${PULSAR_TARBALL} /
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 3a10a7f0f465
[INFO] DOCKER> Step 4/28 : RUN mv /apache-pulsar-* /pulsar
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 121b72edf26c
[INFO] DOCKER> Step 5/28 : COPY scripts/apply-config-from-env.py /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 29977ea123f6
[INFO] DOCKER> Step 6/28 : COPY scripts/apply-config-from-env-with-prefix.py /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> f3465cf9514c
[INFO] DOCKER> Step 7/28 : COPY scripts/gen-yml-from-env.py /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> f33d63ce245d
[INFO] DOCKER> Step 8/28 : COPY scripts/generate-zookeeper-config.sh /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> b824025f88bc
[INFO] DOCKER> Step 9/28 : COPY scripts/pulsar-zookeeper-ruok.sh /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> be232a3180e8
[INFO] DOCKER> Step 10/28 : COPY scripts/watch-znode.py /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> e1208c4fae8b
[INFO] DOCKER> Step 11/28 : COPY scripts/install-pulsar-client.sh /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 1de314e3f901
[INFO] DOCKER> Step 12/28 : RUN for SUBDIRECTORY in conf data download logs; do      [ -d /pulsar/$SUBDIRECTORY ] || mkdir /pulsar/$SUBDIRECTORY;      chmod -R g+w /pulsar/$SUBDIRECTORY;      done
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 1482c4534ac8
[INFO] DOCKER> Step 13/28 : RUN chmod g+w /pulsar/lib/presto
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> e7bf9d6530fc
[INFO] DOCKER> Step 14/28 : FROM ubuntu:20.04
[INFO] DOCKER> 
[INFO] DOCKER> ---> eb574ad4e57c
[INFO] DOCKER> Step 15/28 : ARG DEBIAN_FRONTEND=noninteractive
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 480737f2b5cc
[INFO] DOCKER> Step 16/28 : ARG UBUNTU_MIRROR=mirror://mirrors.ubuntu.com/mirrors.txt
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> b93da317a8e6
[INFO] DOCKER> Step 17/28 : RUN sed -i "s|http://archive\.ubuntu\.com/ubuntu/|${UBUNTU_MIRROR:-mirror://mirrors.ubuntu.com/mirrors.txt}|g" /etc/apt/sources.list      && echo 'Acquire::http::Timeout "30";\nAcquire::ftp::Timeout "30";\nAcquire::Retries "3";' > /etc/apt/apt.conf.d/99timeout_and_retries      && apt-get update      && apt-get -y dist-upgrade      && apt-get -y install --no-install-recommends openjdk-17-jdk-headless netcat dnsutils less procps iputils-ping                  python3 python3-kazoo python3-pip                  curl ca-certificates      && apt-get -y --purge autoremove      && apt-get autoclean      && apt-get clean      && rm -rf /var/lib/apt/lists/*
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 953397f00737
[INFO] DOCKER> Step 18/28 : RUN pip3 install pyyaml==5.4.1
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 0d5d392d05bb
[INFO] DOCKER> Step 19/28 : RUN mkdir /pulsar && chmod g+w /pulsar
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 31f147428bde
[INFO] DOCKER> Step 20/28 : ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> dba415d71afe
[INFO] DOCKER> Step 21/28 : RUN mkdir -p /usr/lib/jvm/java-17-openjdk-amd64/conf/security
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> d2cf7fef934e
[INFO] DOCKER> Step 22/28 : RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 83eaa4f453f4
[INFO] DOCKER> Step 23/28 : ADD target/python-client/ /pulsar/pulsar-client
[INFO] DOCKER> 
[INFO] DOCKER> ---> cf055b5ede4f
[INFO] DOCKER> Step 24/28 : ENV PULSAR_ROOT_LOGGER=INFO,CONSOLE
[INFO] DOCKER> 
[INFO] DOCKER> ---> Running in 2d90766ce2e2
[INFO] DOCKER> Removing intermediate container 2d90766ce2e2
[INFO] DOCKER> ---> 951081b9ab71
[INFO] DOCKER> Step 25/28 : COPY --from=pulsar /pulsar /pulsar
[INFO] DOCKER> 
[INFO] DOCKER> ---> b9e5fd50bebc
[INFO] DOCKER> Step 26/28 : WORKDIR /pulsar
[INFO] DOCKER> 
[INFO] DOCKER> ---> Running in 4a02a04acbe7
[INFO] DOCKER> Removing intermediate container 4a02a04acbe7
[INFO] DOCKER> ---> 01c16f5856d5
[INFO] DOCKER> Step 27/28 : RUN /pulsar/bin/install-pulsar-client.sh
[INFO] DOCKER> 
[INFO] DOCKER> ---> Running in 0e4ed0ece4bb
[INFO] DOCKER> [91m++ python3 -V
[INFO] DOCKER> [91m++ sed -E 's/.* ([[:digit:]]+).([[:digit:]]+).*/\1\2/'
[INFO] DOCKER> [91m+ PYTHON_MAJOR_MINOR=38
[INFO] DOCKER> [91m++ ls /pulsar/pulsar-client
[INFO] DOCKER> [91m++ grep cp38
[INFO] DOCKER> [91m+ WHEEL_FILE=pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

pip3 install '/pulsar/pulsar-client/pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl[all]'

[INFO] DOCKER> Processing ./pulsar-client/pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

[INFO] DOCKER> Collecting certifi

[INFO] DOCKER> Downloading certifi-2022.6.15-py3-none-any.whl (160 kB)

[INFO] DOCKER> Collecting apache-bookkeeper-client>=4.9.2; extra == "all"

[INFO] DOCKER> Downloading apache_bookkeeper_client-4.15.0-py2.py3-none-any.whl (70 kB)

[INFO] DOCKER> Collecting prometheus-client; extra == "all"

[INFO] DOCKER> Downloading prometheus_client-0.14.1-py3-none-any.whl (59 kB)

[INFO] DOCKER> Collecting ratelimit; extra == "all"

[INFO] DOCKER> Downloading ratelimit-2.2.1.tar.gz (5.3 kB)

[INFO] DOCKER> Collecting protobuf<=3.20.*,>=3.6.1; extra == "all"

[INFO] DOCKER> Downloading protobuf-3.19.4-cp38-cp38-manylinux2014_aarch64.whl (913 kB)

[INFO] DOCKER> Collecting grpcio<1.28,>=1.8.2; extra == "all"

[INFO] DOCKER> Downloading grpcio-1.27.2.tar.gz (16.9 MB)

[INFO] DOCKER> [91m    ERROR: Command errored out with exit status 1:

command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-od2wltzl/grpcio/setup.py'"'"'; file='"'"'/tmp/pip-install-od2wltzl/grpcio/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-od2wltzl/grpcio/pip-egg-info

cwd: /tmp/pip-install-od2wltzl/grpcio/

Complete output (13 lines):

/tmp/pip-install-od2wltzl/grpcio/src/python/grpcio/commands.py:102: SyntaxWarning: "is not" with a literal. Did you mean "!="?

if exit_code is not 0:

Traceback (most recent call last):

File "", line 1, in 

File "/tmp/pip-install-od2wltzl/grpcio/setup.py", line 191, in 

if check_linker_need_libatomic():

File "/tmp/pip-install-od2wltzl/grpcio/setup.py", line 149, in check_linker_need_libatomic

cc_test = subprocess.Popen(['cc', '-x', 'c++', '-std=c++11', '-'],

File "/usr/lib/python3.8/subprocess.py", line 858, in init

self._execute_child(args, executable, preexec_fn, close_fds,

File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child

raise child_exception_type(errno_num, err_msg, err_filename)

FileNotFoundError: [Errno 2] No such file or directory: 'cc'

----------------------------------------
[INFO] DOCKER> [91mERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
[INFO] DOCKER> Removing intermediate container 0e4ed0ece4bb

[ERROR] DOCKER> Unable to build image [apachepulsar/pulsar] : "The command '/bin/sh -c /pulsar/bin/install-pulsar-client.sh' returned a non-zero code: 1"  ["The command '/bin/sh -c /pulsar/bin/install-pulsar-client.sh' returned a non-zero code: 1" ]

tisonkun · 2022-08-19T08:57:38Z

It seems all the issues are included in #12944 (comment). I'll try it out.

tisonkun · 2022-09-20T09:27:54Z

With #17733 we should be able to build ARM-based docker image manually. Integrating it with automation or CI is another topic.

eolivelli · 2023-02-01T17:38:22Z

if you want to run pulsar standalone without docker you can simply add this to conf/standalone.conf, this will disable RocksDB

ledgerStorageClass=org.apache.bookkeeper.bookie.SortedLedgerStorage

merlimat · 2023-02-01T17:55:33Z

Since Pulsar 2.11.0, there is no problem in running on ARM64 with default configs

zhongduo · 2023-02-01T22:55:46Z

Any chance to support ARM64 docker in dockerhub?

merlimat · 2023-02-02T00:03:20Z

As of now, you can build ARM64 image, though we don't yet have multi-arch image. It needs some work on the maven-docker plugin to do so

jddCynet · 2023-04-13T12:30:17Z

How can we run it as a docker standalone ?
I tried this cmd
docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.11.0 bin/pulsar standalone

But I have this error: WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

onobc · 2023-04-15T20:39:11Z

But I have this error: WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

I see that warning too, but other than that it works like a charm for me.

tyler-boyd · 2023-05-01T21:23:58Z

I see this issue was closed (as a PR was merged), when should we expect an ARM image to be available on Dockerhub? We've found that running the amd64 image through qemu is too slow to be useful for our use cases, so are very much looking forward to an ARM image 👀 🙌

merlimat · 2023-05-01T21:45:13Z

Yes, this is going to be available in 3.0.0 which is being released this week.

vkuznecovas added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Nov 23, 2021

sijie mentioned this issue Nov 23, 2021

ISSUE-12944: ARM based docker image streamnative/pulsar-archived#3310

Open

codelipenghui assigned kezhenxu94 Dec 2, 2021

codelipenghui added the status/triaged label Dec 2, 2021

kezhenxu94 mentioned this issue Dec 30, 2021

Pulsar branch-2.9 build failure in auditwheel running build-wheel-file-within-docker.sh #13457

Closed

gromsterus mentioned this issue Feb 10, 2022

hub.docker.com apachepulsar/pulsar-standalone Apple Silicon arm64 M1 processor #14174

Closed

kezhenxu94 mentioned this issue Mar 10, 2022

Add arm64 Docker images #14005

Closed

4 tasks

github-actions bot added the lifecycle/stale label Mar 24, 2022

nodece mentioned this issue Apr 20, 2022

[branch-2.9][Broker] Fix call sync method in onPoliciesUpdate method #15227

Merged

1 task

github-actions bot removed the lifecycle/stale label Apr 21, 2022

alpreu mentioned this issue Aug 10, 2022

Tests are failing when running on M1 Mac spring-projects/spring-pulsar#57

Closed

michaeljmarshall mentioned this issue Aug 11, 2022

[2.10.0] ERROR org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Exception while instantiating ZooKeeper #15647

Closed

anthonywu-toast mentioned this issue Aug 12, 2022

CAMEL-18376: camel-pulsar: add redelivery backoff for ack timeout, nack apache/camel#8147

Merged

5 tasks

tisonkun mentioned this issue Aug 23, 2022

[improve][docker] Switch to Temurin JDK #17129

Merged

4 tasks

tisonkun mentioned this issue Sep 20, 2022

[feat][build] Support ARM64-based docker images #17733

Merged

4 tasks

Paraphraser mentioned this issue Sep 23, 2022

Request for Apache Pulsar SensorsIot/IOTstack#603

Open

tisonkun removed the status/triaged label Nov 21, 2022

merlimat mentioned this issue Feb 5, 2023

[improve] Allow to build and push multi-arch Docker images #19432

Merged

14 tasks

merlimat added this to the 3.0.0 milestone Feb 5, 2023

This was referenced Mar 7, 2023

[improve][ci] Addresses PR comments from #19432 merlimat/pulsar#5

Closed

[improve][ci] Docker multi arch - Addresses PR comments from #19432 #19766

Closed

cbornet modified the milestones: 3.0.0, 3.1.0 Apr 11, 2023

merlimat closed this as completed in #19432 Apr 22, 2023

merlimat modified the milestones: 3.1.0, 3.0.0 May 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARM based docker image #12944

ARM based docker image #12944

vkuznecovas commented Nov 23, 2021

michaeljmarshall commented Nov 24, 2021

kezhenxu94 commented Dec 1, 2021 •

edited

Loading

codelipenghui commented Dec 2, 2021

krissetto commented Dec 7, 2021

kezhenxu94 commented Dec 7, 2021

lorthirk commented Dec 9, 2021

jibbolo commented Dec 9, 2021

kezhenxu94 commented Dec 14, 2021 •

edited

Loading

vkuznecovas commented Dec 15, 2021

krissetto commented Dec 15, 2021

kezhenxu94 commented Dec 15, 2021

tdrozdowski commented Dec 29, 2021

kezhenxu94 commented Dec 29, 2021

mgagliardo91 commented Jan 27, 2022

kezhenxu94 commented Jan 27, 2022

kezhenxu94 commented Jan 28, 2022

krissetto commented Feb 21, 2022

github-actions bot commented Mar 24, 2022

heesung-sn commented Apr 20, 2022 •

edited

Loading

rhenwood-arm commented Apr 26, 2022

michaeljmarshall commented Jul 9, 2022

michaeljmarshall commented Jul 9, 2022

tisonkun commented Aug 18, 2022

tisonkun commented Aug 18, 2022 •

edited

Loading

tisonkun commented Aug 19, 2022

tisonkun commented Sep 20, 2022

eolivelli commented Feb 1, 2023

merlimat commented Feb 1, 2023

zhongduo commented Feb 1, 2023

merlimat commented Feb 2, 2023

jddCynet commented Apr 13, 2023

onobc commented Apr 15, 2023

tyler-boyd commented May 1, 2023

merlimat commented May 1, 2023

ARM based docker image #12944

ARM based docker image #12944

Comments

vkuznecovas commented Nov 23, 2021

michaeljmarshall commented Nov 24, 2021

kezhenxu94 commented Dec 1, 2021 • edited Loading

codelipenghui commented Dec 2, 2021

krissetto commented Dec 7, 2021

kezhenxu94 commented Dec 7, 2021

lorthirk commented Dec 9, 2021

jibbolo commented Dec 9, 2021

kezhenxu94 commented Dec 14, 2021 • edited Loading

vkuznecovas commented Dec 15, 2021

krissetto commented Dec 15, 2021

kezhenxu94 commented Dec 15, 2021

tdrozdowski commented Dec 29, 2021

kezhenxu94 commented Dec 29, 2021

mgagliardo91 commented Jan 27, 2022

kezhenxu94 commented Jan 27, 2022

kezhenxu94 commented Jan 28, 2022

krissetto commented Feb 21, 2022

github-actions bot commented Mar 24, 2022

heesung-sn commented Apr 20, 2022 • edited Loading

rhenwood-arm commented Apr 26, 2022

michaeljmarshall commented Jul 9, 2022

michaeljmarshall commented Jul 9, 2022

tisonkun commented Aug 18, 2022

tisonkun commented Aug 18, 2022 • edited Loading

tisonkun commented Aug 19, 2022

tisonkun commented Sep 20, 2022

eolivelli commented Feb 1, 2023

merlimat commented Feb 1, 2023

zhongduo commented Feb 1, 2023

merlimat commented Feb 2, 2023

jddCynet commented Apr 13, 2023

onobc commented Apr 15, 2023

tyler-boyd commented May 1, 2023

merlimat commented May 1, 2023

kezhenxu94 commented Dec 1, 2021 •

edited

Loading

kezhenxu94 commented Dec 14, 2021 •

edited

Loading

heesung-sn commented Apr 20, 2022 •

edited

Loading

tisonkun commented Aug 18, 2022 •

edited

Loading