Support Python container example on MacOS

[aherrmann]

Is your feature request related to a problem? Please describe.
The Python container example does not currently work if it is built on MacOS.:

  ERROR: /Users/runner/work/rules_nixpkgs/rules_nixpkgs/examples/python-container/BUILD:11:10: While resolving toolchains for target //:hello_image.binary: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.

The problem seems to be that rules_docker requires a CC toolchain for Python images that fulfills the run_in_container constraint. However, simply adding that execution constraint to the Nix provided CC toolchain seems to be insufficient.

Describe the solution you'd like
A setup of the Python container example that builds successfully on both a Linux and a MacOS host/exec OS.

[benradf]

Quick update with my findings so far. By changing the following:

1. Passing name = "nixpkgs_config_cc" to nixpkgs_cc_configure (instead of the default of "local_config_cc")
2. Updating rules_docker to version 0.25.0 to be able to pass --@io_bazel_rules_docker//transitions:enable=false to bazel build (as suggested here).
   I was able to get the cpp toolchain resolution error to go away. But it now complains about go instead:

ERROR: /private/var/tmp/_bazel_ben/5abc1b3a46cad12077f1f9cd1e1b30eb/external/io_bazel_rules_docker/container/go/cmd/digester/BUILD:31:10: While resolving toolchains for target @io_bazel_rules_docker//container/go/cmd/digester:digester: no matching toolchains found for types @io_bazel_rules_go//go:toolchain
ERROR: Analysis of target '//:hello_image' failed; build aborted:

Also I'm not sure the cpp issue has been properly resolved. I may have just masked it.

[benradf]

I managed to reproduce the same error on a minimal example without rules_nixpkgs so I think this is a more general problem with rules_docker. Taking a step back, I'm not even sure how this is meant to work without a cross-compiler. The execution platform is aarch64/osx but the target platform is a docker container (i.e. ???/linux).

I dug into the internals of Bazel a bit to build my understanding of how toolchain resolution works and confirm that the lack of a cross-compiler is the problem. The built-in debug info wasn't great so I added some extra print outs:

SingleToolchainResolutionFunction.checkConstraints(                                                                                                                                                                
    toolchainConstraints = <[@platforms//cpu:arm, @platforms//os:android]>                                                                                                                                         
    platformType = target                                                                                
    platform = PlatformInfo(@io_bazel_rules_docker//platforms:image_transition, constraints=<[@platforms//cpu:x86_64, @platforms//os:linux]>)
    toolchainTypeLabel = @bazel_tools//tools/cpp:toolchain_type                                                                                                                                                    
    toolchainLabel = @local_config_cc//:cc-compiler-armeabi-v7a                              
)                                                                                                        
SingleToolchainResolutionFunction.checkConstraints(                                                      
    toolchainConstraints = <[@platforms//cpu:aarch64, @platforms//os:osx]>                        
    platformType = target                                                                                                                                                                                          
    platform = PlatformInfo(@io_bazel_rules_docker//platforms:image_transition, constraints=<[@platforms//cpu:x86_64, @platforms//os:linux]>)
    toolchainTypeLabel = @bazel_tools//tools/cpp:toolchain_type                                       
    toolchainLabel = @local_config_cc//:cc-compiler-darwin_arm64                                    
)   

The above shows that two potential toolchains were checked and they targeted arm/android and aarch64/osx. Neither was selected because the desired target platform was x86_64/linux.

This could potentially be made to work with rules_nixpkgs if we can get Nix to build us an appropriate cross-compiler.

[aherrmann]

Taking a step back, I'm not even sure how this is meant to work without a cross-compiler. The execution platform is aarch64/osx but the target platform is a docker container (i.e. ???/linux).

Indeed, it looks like this requires a cross-compiler. That makes sense, as you describe.

This could potentially be made to work with rules_nixpkgs if we can get Nix to build us an appropriate cross-compiler.

True, I know nixpkgs has cross-compilation capabilities, but, I'm not sure how well MacOS is supported as a target. bazel-zig-cc can cross-compile to MacOS and the Zig compiler is based on LLVM/Clang. Perhaps a nixpkgs clang can be used for this purpose. If not, and if something like Zig is required instead, then this may well be considered out of scope for rules_nixpkgs.