Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat][build] Support ARM64-based docker images #17733

Merged
merged 8 commits into from
Sep 21, 2022
Merged

[feat][build] Support ARM64-based docker images #17733

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
6a2d129
detect arch for Golang installation
tisonkun Sep 20, 2022
e0d465a
detect arch for JDK installation
tisonkun Sep 20, 2022
b2d5605
fix python client build on ARM64
tisonkun Sep 20, 2022
9fe71d9
fix java_home path
tisonkun Sep 20, 2022
a13e8cb
fix resolve architecture
tisonkun Sep 20, 2022
1ae7709
revert chained run commands
tisonkun Sep 20, 2022
1909f25
architecture -> ARCH
tisonkun Sep 20, 2022
2793885
install build-essential python3-dev only for ARM64
tisonkun Sep 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions docker/pulsar/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,9 @@ RUN mkdir -p /etc/apt/keyrings \
&& echo "deb [signed-by=/etc/apt/keyrings/adoptium.asc] https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | tee /etc/apt/sources.list.d/adoptium.list \
&& apt-get update \
&& apt-get -y dist-upgrade \
&& apt-get -y install temurin-17-jdk
&& apt-get -y install temurin-17-jdk \
&& export ARCH=$(uname -m | sed -r 's/aarch64/arm64/g' | awk '!/arm64/{$0="amd64"}1') \
&& echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/temurin-17-jdk-$ARCH/conf/security/java.security \

# Cleanup apt
RUN apt-get -y --purge autoremove \
Expand All @@ -87,13 +89,10 @@ RUN pip3 install pyyaml==5.4.1
# 4. /pulsar - hadoop writes to this directory
RUN mkdir /pulsar && chmod g+w /pulsar

ENV JAVA_HOME /usr/lib/jvm/temurin-17-jdk-amd64
RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/temurin-17-jdk-amd64/conf/security/java.security
ADD target/python-client/ /pulsar/pulsar-client

ENV PULSAR_ROOT_LOGGER=INFO,CONSOLE


COPY --from=pulsar /pulsar /pulsar
WORKDIR /pulsar

Expand Down
3 changes: 2 additions & 1 deletion docker/pulsar/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
<packaging>pom</packaging>

<properties>
<pythonClientBuildArch>x86_64</pythonClientBuildArch>
<skipBuildPythonClient>false</skipBuildPythonClient>
<skipCopyPythonClients>false</skipCopyPythonClients>
</properties>
Expand Down Expand Up @@ -79,7 +80,7 @@
<executable>${project.basedir}/../../pulsar-client-cpp/docker/build-wheels.sh</executable>
<arguments>
<!-- build python 3.8 -->
<argument>3.8 cp38-cp38 manylinux2014 x86_64</argument>
<argument>3.8 cp38-cp38 manylinux2014 ${pythonClientBuildArch}</argument>
tisonkun marked this conversation as resolved.
Show resolved Hide resolved
</arguments>
</configuration>
</execution>
Expand Down
7 changes: 7 additions & 0 deletions docker/pulsar/scripts/install-pulsar-client.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,13 @@

set -x

# TODO: remove these lines once grpcio doesn't need to compile from source on ARM64 platform
ARCH=$(uname -m | sed -r 's/aarch64/arm64/g' | awk '!/arm64/{$0="amd64"}1')
if [ "${ARCH}" == "arm64" ]; then
apt update
apt -y install build-essential python3-dev
fi

PYTHON_MAJOR_MINOR=$(python3 -V | sed -E 's/.* ([[:digit:]]+)\.([[:digit:]]+).*/\1\2/')
WHEEL_FILE=$(ls /pulsar/pulsar-client | grep "cp${PYTHON_MAJOR_MINOR}")
pip3 install /pulsar/pulsar-client/${WHEEL_FILE}[all]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

[INFO] DOCKER> [91m++ python3 -V
++ sed -E 's/.* ([[:digit:]]+)\.([[:digit:]]+).*/\1\2/'

[INFO] DOCKER> [91m+ PYTHON_MAJOR_MINOR=38

[INFO] DOCKER> [91m++ ls /pulsar/pulsar-client
++ grep cp38

[INFO] DOCKER> [91m+ WHEEL_FILE='pulsar_client-2.11.0-cp38-cp38-manylinux1_x86_64.whl
pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl'

[INFO] DOCKER> [91m+ pip3 install /pulsar/pulsar-client/pulsar_client-2.11.0-cp38-cp38-manylinux1_x86_64.whl 'pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl[all]'

[INFO] DOCKER> [91mERROR: pulsar_client-2.11.0-cp38-cp38-manylinux1_x86_64.whl is not a supported wheel on this platform.

[INFO] DOCKER> Removing intermediate container a88eb585f4d1
[ERROR] DOCKER> Unable to build image [apachepulsar/pulsar] : "The command '/bin/sh -c /pulsar/bin/install-pulsar-client.sh' returned a non-zero code: 1"  ["The command '/bin/sh -c /pulsar/bin/install-pulsar-client.sh' returned a non-zero code: 1" ]

observed this error.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this problem will anyway go away once we make the changes for PIP-209 (moving C++ client out of repo)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@heesung-sn

mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests -DpythonClientBuildArch=aarch64

Be aware that you should pass -DpythonClientBuildArch=aarch64.

Copy link
Member Author

@tisonkun tisonkun Sep 26, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made this patch to support ARM64-based docker images while keeping the current workflow as is. That is, you should manually specify that you want to build an ARM64-based image.

About automatically detecting platforms, see also #17733 (comment) that we have some name matching problems.

And a clean solution would be, as @merlimat described, we publish C++/Python client alone and download them from the package manager so that leveraging the package manager's platform detect functionality.

Copy link
Contributor

@heesung-sn heesung-sn Sep 26, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I provided the pythonClientBuildArch option when I encountered the error.

mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests -DpythonClientBuildArch=aarch64

I tried to hard-code like the below, then it passes. I wonder if the client somehow output wheel files for both archs.

pip3 install /pulsar/pulsar-client/pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl[all]

7 changes: 3 additions & 4 deletions tests/docker-images/java-test-image/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,9 @@ RUN mkdir -p /etc/apt/keyrings \
&& echo "deb [signed-by=/etc/apt/keyrings/adoptium.asc] https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | tee /etc/apt/sources.list.d/adoptium.list \
&& apt-get update \
&& apt-get -y dist-upgrade \
&& apt-get -y install temurin-17-jdk

ENV JAVA_HOME /usr/lib/jvm/temurin-17-jdk-amd64
RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/temurin-17-jdk-amd64/conf/security/java.security
&& apt-get -y install temurin-17-jdk \
&& export ARCH=$(uname -m | sed -r 's/aarch64/arm64/g' | awk '!/arm64/{$0="amd64"}1') \
&& echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/temurin-17-jdk-$ARCH/conf/security/java.security

# /pulsar/bin/watch-znode.py requires python3-kazoo
# /pulsar/bin/pulsar-managed-ledger-admin requires python3-protobuf
Expand Down
9 changes: 2 additions & 7 deletions tests/docker-images/latest-version-image/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,8 @@ RUN apt-get install -y procps curl git build-essential

ENV GOLANG_VERSION 1.15.8

RUN curl -sSL https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz \
| tar -C /usr/local -xz

# RUN wget https://dl.google.com/go/go1.13.3.linux-amd64.tar.gz && tar -xvf go1.13.3.linux-amd64.tar.gz && mv go /usr/local
# RUN export GOROOT=/usr/local/go && export GOPATH=$HOME/go && export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
# RUN echo "export GOROOT=/usr/local/go" >> ~/.profile && echo "export GOPATH=$HOME/go" >> ~/.profile && echo "export PATH=$GOPATH/bin:$GOROOT/bin:$PATH" >> ~/.profile

RUN export ARCH=$(uname -m | sed -r 's/aarch64/arm64/g' | awk '!/arm64/{$0="amd64"}1') \
&& curl -sSL https://golang.org/dl/go$GOLANG_VERSION.linux-$ARCH.tar.gz | tar -C /usr/local -xz
ENV PATH /usr/local/go/bin:$PATH

RUN mkdir -p /go/src /go/bin && chmod -R 777 /go
Expand Down