Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve bazel builds #53

Merged
merged 21 commits into from
Jul 25, 2019
Merged

Improve bazel builds #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
2ec44ae
Fix quoting in travis scripts
tkilias Jul 11, 2019
517d3c8
Add essential udfclient sources to standard flavors
tkilias Jul 15, 2019
e3c6083
Remove debug output
tkilias Jul 16, 2019
f236294
Fix compiler flags for standard flavor
tkilias Jul 16, 2019
0c37e6f
Fix further compiler flags and don't use LIBEXAUDFLIB_PATH Environment
tkilias Jul 17, 2019
58e8656
test manual specification of compiler flags and remove all outputs to…
tkilias Jul 17, 2019
16cc735
bake paths into udfclient to avoid starter script
tkilias Jul 17, 2019
546ad31
Fix rpath for java to avoid shell script at startup
tkilias Jul 18, 2019
9dca467
align the compile flags to the compile flags before bazel
tkilias Jul 18, 2019
dd493f5
Use unusal optimizer flags
tkilias Jul 18, 2019
3dfb59f
try only -flto
tkilias Jul 18, 2019
23d1227
activate loop-unroll only for r and remove prints from exascript_pyth…
tkilias Jul 19, 2019
76075bb
Fix strange flag -py3 in swig call for python2
tkilias Jul 19, 2019
a93949d
Remove unneccessary java shared libraries
tkilias Jul 19, 2019
6999a95
add libjava
tkilias Jul 19, 2019
1e3860f
Clean up after solving the performance degradation
tkilias Jul 19, 2019
18ea28e
Move rpath workarround to java_repository.bzl
tkilias Jul 19, 2019
3d564aa
Fix linker/compiler problems with -flto, remove libjava from java_rep…
tkilias Jul 19, 2019
47247bc
Try to solve linking problems with scriptoptionlines
tkilias Jul 21, 2019
1feb4bd
Adapt base_test_build_run Dockerfiles to new build presets
tkilias Jul 22, 2019
76a0caa
Cleanup accidently added files
tkilias Jul 24, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions flavors/fancyr-EXASOL-6.0.0/flavor_base/base_test_build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c dbg --define r=true //:exaudfclient"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty -c dbg --config r --config slow-wrapper-py2"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient.sh

RUN mkdir /exasol_emulator
COPY emulator/ /exasol_emulator
COPY src/exaudflib/zmqcontainer.proto /exasol_emulator
Expand Down
3 changes: 2 additions & 1 deletion flavors/fancyr-EXASOL-6.0.0/flavor_base/build_deps/packages/apt_get_packages
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ openjdk-8-jdk
build-essential
locales
libpcre3-dev
protobuf-compiler
protobuf-compiler
chrpath
6 changes: 3 additions & 3 deletions flavors/fancyr-EXASOL-6.0.0/flavor_base/build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c opt --define r=true //:exaudfclient"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty --config optimize --config optimize-r --config fast-binary-py2"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient

COPY --from={{build_deps}} /build_info /build_info
6 changes: 3 additions & 3 deletions flavors/fancyr-EXASOL-6.1.0/flavor_base/base_test_build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c dbg --define r=true //:exaudfclient"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty -c dbg --config r --config slow-wrapper-py2"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient.sh

RUN mkdir /exasol_emulator
COPY emulator/ /exasol_emulator
COPY src/exaudflib/zmqcontainer.proto /exasol_emulator
Expand Down
3 changes: 2 additions & 1 deletion flavors/fancyr-EXASOL-6.1.0/flavor_base/build_deps/packages/apt_get_packages
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ openjdk-8-jdk
build-essential
locales
libpcre3-dev
protobuf-compiler
protobuf-compiler
chrpath
6 changes: 3 additions & 3 deletions flavors/fancyr-EXASOL-6.1.0/flavor_base/build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c opt --define r=true //:exaudfclient"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty --config optimize --config optimize-r --config fast-binary-py2"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient

COPY --from={{build_deps}} /build_info /build_info
6 changes: 3 additions & 3 deletions flavors/python3-ds-EXASOL-6.0.0/flavor_base/base_test_build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c dbg --define python=true //:exaudfclient_py3"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty -c dbg --config python --config slow-wrapper-py3"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient_py3.sh

RUN mkdir /exasol_emulator
COPY emulator/ /exasol_emulator
COPY src/exaudflib/zmqcontainer.proto /exasol_emulator
Expand Down
3 changes: 2 additions & 1 deletion flavors/python3-ds-EXASOL-6.0.0/flavor_base/build_deps/packages/apt_get_packages
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ openjdk-8-jdk
build-essential
locales
libpcre3-dev
protobuf-compiler
protobuf-compiler
chrpath
6 changes: 3 additions & 3 deletions flavors/python3-ds-EXASOL-6.0.0/flavor_base/build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c opt --define python=true //:exaudfclient_py3"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty --config optimize --config python --config fast-binary-py3"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient_py3

COPY --from={{build_deps}} /build_info /build_info
6 changes: 3 additions & 3 deletions flavors/python3-ds-EXASOL-6.1.0/flavor_base/base_test_build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c dbg --define python=true //:exaudfclient_py3"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty -c dbg --config python --config slow-wrapper-py3"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient_py3.sh

RUN mkdir /exasol_emulator
COPY emulator/ /exasol_emulator
COPY src/exaudflib/zmqcontainer.proto /exasol_emulator
Expand Down
3 changes: 2 additions & 1 deletion flavors/python3-ds-EXASOL-6.1.0/flavor_base/build_deps/packages/apt_get_packages
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ openjdk-8-jdk
build-essential
locales
libpcre3-dev
protobuf-compiler
protobuf-compiler
chrpath
6 changes: 3 additions & 3 deletions flavors/python3-ds-EXASOL-6.1.0/flavor_base/build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c opt --define python=true //:exaudfclient_py3"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty --config optimize --config python --config fast-binary-py3"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient_py3

COPY --from={{build_deps}} /build_info /build_info
6 changes: 3 additions & 3 deletions flavors/python3-ds-cuda-preview-EXASOL-6.1.0/flavor_base/base_test_build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c dbg --define python=true //:exaudfclient_py3"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty -c dbg --config python --config slow-wrapper-py3"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient_py3.sh

RUN mkdir /exasol_emulator
COPY emulator/ /exasol_emulator
COPY src/exaudflib/zmqcontainer.proto /exasol_emulator
Expand Down
3 changes: 2 additions & 1 deletion ...ors/python3-ds-cuda-preview-EXASOL-6.1.0/flavor_base/build_deps/packages/apt_get_packages
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ openjdk-8-jdk
build-essential
locales
libpcre3-dev
protobuf-compiler
protobuf-compiler
chrpath
6 changes: 3 additions & 3 deletions flavors/python3-ds-cuda-preview-EXASOL-6.1.0/flavor_base/build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c opt --define python=true //:exaudfclient_py3"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty --config optimize --config python --config fast-binary-py3"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient_py3

COPY --from={{build_deps}} /build_info /build_info
6 changes: 3 additions & 3 deletions flavors/standard-EXASOL-6.0.0/flavor_base/base_test_build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c dbg --define r=true --define python=true --define java=true //:exaudfclient"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty -c dbg --config python --config java --config r --config slow-wrapper-py2"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient.sh

RUN mkdir /exasol_emulator
COPY emulator/ /exasol_emulator
COPY src/exaudflib/zmqcontainer.proto /exasol_emulator
Expand Down
3 changes: 2 additions & 1 deletion flavors/standard-EXASOL-6.0.0/flavor_base/build_deps/packages/apt_get_packages
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ openjdk-8-jdk
build-essential
locales
libpcre3-dev
protobuf-compiler
protobuf-compiler
chrpath
11 changes: 8 additions & 3 deletions flavors/standard-EXASOL-6.0.0/flavor_base/build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,18 @@ COPY --from={{build_deps}} /env /env

RUN ldconfig

RUN mkdir /exaudf_src /exaudf
RUN mkdir /exaudf_src /exaudf /exaudf/src
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"
ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c opt --define r=true --define python=true --define java=true //:exaudfclient"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty --config optimize --config python --config java --config optimize-r --config fast-binary-py2"]
RUN cp -r -L bazel-bin/* /exaudf

# These source files is required for the building of the protegrity client
RUN cp debug_message.h exaudfclient.cc exaudflib/exaudflib.cc exaudflib/exaudflib.h exaudflib/zmqcontainer.proto /exaudf/src

RUN ./test_udfclient.sh /exaudf/exaudfclient

COPY --from={{build_deps}} /build_info /build_info
7 changes: 4 additions & 3 deletions flavors/standard-EXASOL-6.1.0/flavor_base/base_test_build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,13 @@ RUN ldconfig
RUN mkdir /exaudf_src /exaudf
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c dbg --define r=true --define python=true --define java=true //:exaudfclient //:exaudfclient_py3"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty -c dbg --config python --config java --config r --config slow-wrapper-both"]
RUN cp -r -L bazel-bin/* /exaudf

RUN ./test_udfclient.sh /exaudf/exaudfclient.sh
RUN ./test_udfclient.sh /exaudf/exaudfclient_py3.sh

RUN mkdir /exasol_emulator
COPY emulator/ /exasol_emulator
COPY src/exaudflib/zmqcontainer.proto /exasol_emulator
Expand Down
3 changes: 2 additions & 1 deletion flavors/standard-EXASOL-6.1.0/flavor_base/build_deps/packages/apt_get_packages
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ openjdk-8-jdk
build-essential
locales
libpcre3-dev
protobuf-compiler
protobuf-compiler
chrpath
12 changes: 8 additions & 4 deletions flavors/standard-EXASOL-6.1.0/flavor_base/build_run/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,17 @@ COPY --from={{build_deps}} /env /env

RUN ldconfig

RUN mkdir /exaudf_src /exaudf
RUN mkdir /exaudf_src /exaudf /exaudf/src
COPY src/ /exaudf_src/

#ENV VERBOSE_BUILD "--subcommands --verbose_failures"

WORKDIR /exaudf_src/
RUN ["/bin/bash", "-c", "source /env && bash build.sh -c opt --define r=true --define python=true --define java=true //:exaudfclient //:exaudfclient_py3"]
RUN ["/bin/bash", "-c", "source /env && bash build.sh --config no-tty --config optimize --config python --config java --config optimize-r --config fast-binary-both"]
RUN cp -r -L bazel-bin/* /exaudf

# These source files is required for the building of the protegrity client
RUN cp debug_message.h exaudfclient.cc exaudflib/exaudflib.cc exaudflib/exaudflib.h exaudflib/zmqcontainer.proto /exaudf/src

RUN ./test_udfclient.sh /exaudf/exaudfclient
RUN ./test_udfclient.sh /exaudf/exaudfclient_py3

COPY --from={{build_deps}} /build_info /build_info
21 changes: 18 additions & 3 deletions src/.bazelrc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
startup --expand_configs_in_place
build --force_pic --action_env=NUMPY_PREFIX --action_env=PYTHON2_PREFIX --action_env=PYTHON2_VERSION --action_env=PYTHON3_SYSPATH --action_env=PYTHON3_PREFIX --action_env=PYTHON3_VERSION --action_env=PYTHON2_SYSPATH --action_env=PROTOBUF_BIN --action_env=PROTOBUF_LIBRARY_PREFIX --action_env=PROTOBUF_INCLUDE_PREFIX --action_env=JAVA_PREFIX --cxxopt='-std=c++14'
build:opt --copt=-O3
build --copt='-std=c++14' --force_pic --action_env=PROTOBUF_BIN --action_env=PROTOBUF_LIBRARY_PREFIX --action_env=PROTOBUF_INCLUDE_PREFIX
# TODO add environment variables for R libraries
build:benchmark --define benchmark=true
build:r --define r=true
build:java --define java=true --action_env=JAVA_PREFIX
build:python --define python=true --action_env=PYTHON2_SYSPATH --action_env=PYTHON2_PREFIX --action_env=PYTHON2_VERSION --action_env=NUMPY_PREFIX --action_env=PYTHON3_SYSPATH --action_env=PYTHON3_PREFIX --action_env=PYTHON3_VERSION
build:fast-binary-py2 --copt='-DCUSTOM_LIBEXAUDFLIB_PATH="/exaudf/libexaudflib_complete.so"' //:exaudfclient
build:fast-binary-py3 --copt='-DCUSTOM_LIBEXAUDFLIB_PATH="/exaudf/libexaudflib_complete.so"' //:exaudfclient_py3
build:fast-binary-both --config=fast-binary-py2 --config=fast-binary-py3
build:slow-wrapper-py2 //:exaudfclient_py2_bash_wrapper
build:slow-wrapper-py3 //:exaudfclient_py3_bash_wrapper
build:slow-wrapper-both --config=slow-wrapper-py2 --config=slow-wrapper-py3
build:verbose --copt='-v' --subcommands --verbose_failures --announce_rc
#TODO test linkopts="-flto"
build:optimize --copt="-g0" --copt="-DNDEBUG" --copt=-fstack-protector-strong --copt=-fomit-frame-pointer --copt=-ffunction-sections --copt=-fdata-sections --copt="-O3" --copt="-U_FORTIFY_SOURCE" --copt="-flto"
build:optimize-r --define r-unroll-loops=true --config=r
build:no-tty --curses=no --color=no
build:debug-build --sandbox_debug --config=verbose
Loading