builds with cgo appear to miss .S assembly files

[tals]

What version of rules_go are you using?

0.35.0

What version of gazelle are you using?

0.27.0

What version of Bazel are you using?

Build label: 5.4.0

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

$ lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

Any other potentially useful information about your toolchain?

What did you do?

I am attempting to build Go program using the popular zstd package from DataDog
This library vendors zstd, which includes accelerated operations in the .S file.

This works well with the vanilla go toolchain, however building this via rules_go fails to pick up an .S file for amd64, causing it to fail during link time

$ git clone https://github.com/tals/bazel-rulesgo-broken-asm-repro
$ cd bazel-rulesgo-broken-asm-repro
$ go run main.go
Hello, World!

$ bazel run  //:hello-go

INFO: Analyzed target //:hello-go (26 packages loaded, 195 targets configured).
INFO: Found 1 target...
ERROR: /home/tal/.cache/bazel/_bazel_tal/d6d42962320318bd35977430f4d78b15/external/com_github_datadog_zstd/BUILD.bazel:3:11: GoCompilePkg external/com_github_datadog_zstd/zstd.a failed: (Exit 1): builder failed: error executing command bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix linux_amd64 -src external/com_github_datadog_zstd/errors.go -src ... (remaining 211 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
/tmp/rules_go_work-585792782/cgo/github.com/DataDog/zstd/_x16.o:huf_decompress.c:function HUF_decompress4X1_usingDTable_internal_bmi2_asm: error: undefined reference to 'HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop'
/tmp/rules_go_work-585792782/cgo/github.com/DataDog/zstd/_x16.o:huf_decompress.c:function HUF_decompress4X2_usingDTable_internal_bmi2_asm: error: undefined reference to 'HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop'
collect2: error: ld returned 1 exit status
compilepkg: error running subcommand /usr/bin/gcc: exit status 1
Target //:hello-go failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 4.087s, Critical Path: 3.93s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

The Gazelle-generated project does include the .S file, however:

go_library(
    name = "zstd",
    srcs = [
        ...
        "huf_compress.c",
        "huf_decompress.c",
        "huf_decompress_amd64.S", # <---

What did you expect to see?

Hello, World!

What did you see instead?

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
/tmp/rules_go_work-585792782/cgo/github.com/DataDog/zstd/_x16.o:huf_decompress.c:function HUF_decompress4X1_usingDTable_internal_bmi2_asm: error: undefined reference to 'HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop'
/tmp/rules_go_work-585792782/cgo/github.com/DataDog/zstd/_x16.o:huf_decompress.c:function HUF_decompress4X2_usingDTable_internal_bmi2_asm: error: undefined reference to 'HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop'
collect2: error: ld returned 1 exit status
compilepkg: error running subcommand /usr/bin/gcc: exit status 1
Target //:hello-go failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 4.087s, Critical Path: 3.93s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

[fmeum]

Looks like an incarnation of #2006.
@jayconrod Do you happen to recall what exactly would be needed to support this situation and resolve #2006?

[tals]

I've also tried axing the asm build by passing in -DZSTD_DISABLE_ASM to copts, and now it blows up on portability_macros.h:42: unexpected token after '#': if (space between # and if ).

$ sed -n 40,45p portability_macros.h
#ifndef ZSTD_MEMORY_SANITIZER
#  if __has_feature(memory_sanitizer)
#    define ZSTD_MEMORY_SANITIZER 1
#  else
#    define ZSTD_MEMORY_SANITIZER 0

This also works correctly with the native golang toolchain.

[jsharpe]

I hit this when trying to build go-ethereum - this PR fixes this by patching the build files but would be good if this just worked! https://github.com/aspect-build/rules_sol/pull/8/files#diff-1f2c31b701f066691b4a4024644f86e092ab731eb81cc5d6282017fe485082ae

[evanj]

Having debugged this a bit: It seems to me like there are two problems:

1. rules_go is skipping this input .S, maybe because it is named the same as a GOARCH value, huf_decompress_amd64.S, which I think is a bug (see the Go build constraints documentation for details). I haven't been able to reproduce this separately yet.
2. rules_go tries to run the Go assembler, rather than using the C compiler. The go build tool makes this switch for CGO_ENABLED packages, but gazelle does not. Ideally the workaround should not be needed.

I added a small demo program here which reproduces this error in a simpler context: https://github.com/evanj/rulesgotest/tree/main/cgoexample_asm

bazelisk run //cgoexample_asm
ERROR: rulesgotest/cgoexample_asm/BUILD.bazel:17:10: GoCompilePkg cgoexample_asm/cgoexample_asm.a failed: (Exit 1): builder failed: error executing command (from target //cgoexample_asm:cgoexample_asm) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/go_sdk/builder_reset/builder compilepkg -sdk external/go_sdk -installsuffix darwin_arm64 -src cgoexample_asm/cgoexample_asm.go -src ... (remaining 39 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
cgoexample_asm/asm_arm64.S:3: unexpected token after '#': if

[fmeum]

@evanj Thanks for the detailed investigation!

Would you be interested in working on a fix for one or both of the issues you identified? I could support you along the way.

[evanj]

I'm going to investigate a bit in some spare time. Meanwhile: the patch approach is a viable workaround. I have an updated version of the patch from #3411 (comment) that works with the latest bazel/rules_go/zstd: https://gist.github.com/evanj/5f9619ace7a18a5acb8a1be9b9d4df23

[evanj]

I just confirmed with my independent reproduction that imports github.com/DataDog/zstd that this should work with the very latest version of rules_go! See https://github.com/evanj/ddzstdbazel

[fmeum]

Great work and thanks for the very instructive example!

[tals]

Awesome work @evanj !!