Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modernize wheel building job config #1783

Merged
merged 1 commit into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions .github/install_bazel.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,10 @@ if ! bazel version; then
if [ "$arch" == "aarch64" ]; then
arch="arm64"
fi
echo "Installing wget and downloading $arch Bazel binary from GitHub releases."
yum install -y wget
wget "https://github.com/bazelbuild/bazel/releases/download/6.4.0/bazel-6.4.0-linux-$arch" -O /usr/local/bin/bazel
chmod +x /usr/local/bin/bazel
echo "Downloading $arch Bazel binary from GitHub releases."
curl -L -o $HOME/bin/bazel --create-dirs "https://github.com/bazelbuild/bazel/releases/download/7.1.1/bazel-7.1.1-linux-$arch"
chmod +x $HOME/bin/bazel
else
# bazel is installed for the correct architecture
# Bazel is installed for the correct architecture
exit 0
fi
50 changes: 31 additions & 19 deletions .github/workflows/wheels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,24 +15,24 @@ jobs:
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install Python 3.11
- name: Install Python 3.12
uses: actions/setup-python@v5
with:
python-version: 3.11
python-version: 3.12
- run: python -m pip install build
- name: Build sdist
run: python -m build --sdist
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
with:
name: dist
name: dist-sdist
path: dist/*.tar.gz

build_wheels:
name: Build Google Benchmark wheels on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
os: [ubuntu-latest, macos-13, macos-14, windows-latest]
nicholasjng marked this conversation as resolved.
Show resolved Hide resolved

steps:
- name: Check out Google Benchmark
Expand All @@ -47,32 +47,44 @@ jobs:
platforms: all

- name: Build wheels on ${{ matrix.os }} using cibuildwheel
uses: pypa/cibuildwheel@v2.16.2
uses: pypa/cibuildwheel@v2.17
env:
CIBW_BUILD: 'cp38-* cp39-* cp310-* cp311-* cp312-*'
CIBW_BUILD: "cp38-* cp39-* cp310-* cp311-* cp312-*"
CIBW_SKIP: "*-musllinux_*"
CIBW_TEST_SKIP: "*-macosx_arm64"
CIBW_ARCHS_LINUX: x86_64 aarch64
CIBW_ARCHS_MACOS: x86_64 arm64
CIBW_ARCHS_WINDOWS: AMD64
CIBW_TEST_SKIP: "cp38-macosx_*:arm64"
CIBW_ARCHS_LINUX: auto64 aarch64
CIBW_ARCHS_WINDOWS: auto64
CIBW_BEFORE_ALL_LINUX: bash .github/install_bazel.sh
# Grab the rootless Bazel installation inside the container.
CIBW_ENVIRONMENT_LINUX: PATH=$PATH:$HOME/bin
CIBW_TEST_COMMAND: python {project}/bindings/python/google_benchmark/example.py

- name: Upload Google Benchmark ${{ matrix.os }} wheels
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: dist
name: dist-${{ matrix.os }}
path: wheelhouse/*.whl

merge_wheels:
name: Merge all built wheels into one artifact
runs-on: ubuntu-latest
needs: build_wheels
steps:
- name: Merge wheels
uses: actions/upload-artifact/merge@v4
with:
name: dist
pattern: dist-*
delete-merged: true

pypi_upload:
name: Publish google-benchmark wheels to PyPI
needs: [build_sdist, build_wheels]
needs: [merge_wheels]
runs-on: ubuntu-latest
permissions:
id-token: write
steps:
- uses: actions/download-artifact@v3
with:
name: dist
path: dist
- uses: pypa/[email protected]
- uses: actions/download-artifact@v4
with:
path: dist
- uses: pypa/gh-action-pypi-publish@v1
4 changes: 0 additions & 4 deletions MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,6 @@ python.toolchain(
is_default = True,
python_version = "3.12",
)
use_repo(
python,
python = "python_versions",
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip", dev_dependency = True)
pip.parse(
Expand Down
63 changes: 48 additions & 15 deletions setup.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,17 @@
import contextlib
import os
import platform
import re
import shutil
from pathlib import Path
from typing import Any
from typing import Any, Generator

import setuptools
from setuptools.command import build_ext

IS_WINDOWS = platform.system() == "Windows"
IS_MAC = platform.system() == "Darwin"
IS_LINUX = platform.system() == "Linux"

# hardcoded SABI-related options. Requires that each Python interpreter
# (hermetic or not) participating is of the same major-minor version.
Expand All @@ -17,6 +20,46 @@
options = {"bdist_wheel": {"py_limited_api": "cp312"}} if py_limited_api else {}


def is_cibuildwheel() -> bool:
return os.getenv("CIBUILDWHEEL") is not None


@contextlib.contextmanager
def _maybe_patch_toolchains() -> Generator[None, None, None]:
"""
Patch rules_python toolchains to ignore root user error
when run in a Docker container on Linux in cibuildwheel.
"""

def fmt_toolchain_args(matchobj):
suffix = "ignore_root_user_error = True"
callargs = matchobj.group(1)
# toolchain def is broken over multiple lines
if callargs.endswith("\n"):
callargs = callargs + " " + suffix + ",\n"
# toolchain def is on one line.
else:
callargs = callargs + ", " + suffix
return "python.toolchain(" + callargs + ")"

CIBW_LINUX = is_cibuildwheel() and IS_LINUX
try:
if CIBW_LINUX:
module_bazel = Path("MODULE.bazel")
content: str = module_bazel.read_text()
module_bazel.write_text(
re.sub(
r"python.toolchain\(([\w\"\s,.=]*)\)",
fmt_toolchain_args,
content,
)
)
yield
finally:
if CIBW_LINUX:
module_bazel.write_text(content)


class BazelExtension(setuptools.Extension):
"""A C/C++ extension that is defined as a Bazel BUILD target."""

Expand Down Expand Up @@ -73,21 +116,11 @@ def bazel_build(self, ext: BazelExtension) -> None:
for library_dir in self.library_dirs:
bazel_argv.append("--linkopt=/LIBPATH:" + library_dir)
elif IS_MAC:
if platform.machine() == "x86_64":
# C++17 needs macOS 10.14 at minimum
bazel_argv.append("--macos_minimum_os=10.14")

# cross-compilation for Mac ARM64 on GitHub Mac x86 runners.
# ARCHFLAGS is set by cibuildwheel before macOS wheel builds.
archflags = os.getenv("ARCHFLAGS", "")
if "arm64" in archflags:
bazel_argv.append("--cpu=darwin_arm64")
bazel_argv.append("--macos_cpus=arm64")

elif platform.machine() == "arm64":
bazel_argv.append("--macos_minimum_os=11.0")
# C++17 needs macOS 10.14 at minimum
bazel_argv.append("--macos_minimum_os=10.14")

self.spawn(bazel_argv)
with _maybe_patch_toolchains():
self.spawn(bazel_argv)

if IS_WINDOWS:
suffix = ".pyd"
Expand Down
Loading