Update from Go 1.20 to 1.22 causes CodeQL to no longer detect that we built Go code

[dagood]

We use CodeQL through GitHub Actions and had an issue with CodeQL no longer finding Go code when we updated Go from 1.20 to 1.22. Our build actually builds the Go toolset itself, so it's probably a little unusual.

Broken run with debug logs: https://github.com/microsoft/go/actions/runs/10920221364/job/30391782040

CodeQL detected code written in Go but this run didn't build any of it

Example success with debug logs: https://github.com/microsoft/go/actions/runs/10934919945/job/30391982309

Both use:

  "productName" : "CodeQL",
  "vendor" : "GitHub",
  "version" : "2.18.4",
  "sha" : "9c5e434eb23805aa8a8574b87b24eae476ca8615",
  "branches" : [
    "codeql-cli-2.18.4"
  ],

The PR that triggered the broken run updated Go from 1.20.6 to 1.22.6. The example success may include some other changes, and I can produce a cleaner example if necessary, but I don't expect that anything important changed.

• Seemed related to this issue at first, but it was fixed and we still see this: Go 1.23 build and analyze failure in 2.18.3 codeql-action#2467

[mbg]

Hi again @dagood 👋🏻

Thanks for moving this to a separate issue and providing the logs with debugging enabled for both a failed and successful run! That's all very helpful in troubleshooting this.

For the failed run, CodeQL does not see any invocations of go build. For the successful run, it sees three go build calls. There are a couple of possible explanations for this:

1. Could you confirm whether your custom build script (eng/run.ps1 build) performs a fresh build of the sources that you want to analyse every time it is run? If there is any caching of build artifacts or build steps get skipped for other reasons because the source files have not been modified, then CodeQL will not be able to extract them.
2. We are aware of an issue with Go 1.21 onwards on Linux where CodeQL cannot correctly see invocations of calls to the go toolchain because the official releases are statically linked. (The same problem also exists if other, statically-linked programs sit in the call path to go build.) We have a workaround for the typical case that is implemented in the CodeQL Action (that you are using), but which may not work correctly here since you are seemingly using multiple (custom) builds of the toolchain.

[dagood]

1. Could you confirm whether your custom build script (eng/run.ps1 build) performs a fresh build of the sources that you want to analyse every time it is run?

This runs on a fresh VM each time and we don't e.g. run on a Docker image with anything already cached, so as far as I know nothing could be cached.

2. We are aware of an issue with Go 1.21 onwards on Linux where CodeQL cannot correctly see invocations of calls to the go toolchain because the official releases are statically linked. [...] We have a workaround [...] but which may not work correctly here since you are seemingly using multiple (custom) builds of the toolchain.

The Go build involves several bootstrap steps, which does sound like that. My understanding is that they aren't all fully pieced together toolchains, so there might be some nuance there.

Is it possible to turn off CodeQL to bootstrap, then turn it on, and then intentionally build the Go source again using the "final" Go toolchain? (Would we just do that before initializing CodeQL?) That seems like a way to avoid any more toolchain versions showing up. Or is it possible to improve the workaround?

[dagood]

For now, to get unblocked, we are stopping trying to analyze the Go source using CodeQL in GitHub Actions, instead focusing on the util code in the repo which doesn't do any bootstrapping. We run additional CodeQL scans internally. (It seems to me that those might end up being affected by the same issue--we'll see.)

[smowton]

If you move whatever steps are setting up the needed go binary (and add it to the PATH) above the github/codeql-action/init step and then perform only the final build below it, you should see the results you hope for.

[mbg]

To add to what @smowton wrote: I think concretely that means either breaking up your eng/run.ps1 build script or invoking it with different commands to run before initialising CodeQL and to run after it has been initialised.

Also, I had a look at your previously successful analysis runs and they only seemed to (partially) extract the utility code in any case.

[dagood]

We don't run the multiple-toolset bootstrap process in our own scripts, we just set up the environment and let the standard upstream process (https://go.dev/doc/install/source) do the rest. Letting it do the full build and then intentionally rebuilding bits and pieces under CodeQL supervision still sounds like the best option to me (if we end up needing this to work).

Also, I had a look at your previously successful analysis runs and they only seemed to (partially) extract the utility code in any case.

Yeah, we saw the same thing and that definitely made the unblock plan easier to justify. 😄