Newest macos-latest environment update not working for VS Code tests

[gcampbell-msft]

Description

My team maintains the CMake Tools repository: https://github.com/microsoft/vscode-cmake-tools.

In the past week, our Mac CI started failing. In our workflow, we download CMake and Ninja and add them to the path, and then our later unit tests expect them to be on PATH. However, after the most recent macos runner image update, it doesn't seem to work, and we haven't had any changes in that time.

Also, I have a PR here that succeeded our Mac CI before the update: https://github.com/microsoft/vscode-cmake-tools/actions/runs/10801962444/job/29963128203.
Today, after the update, I manually re-ran the CI on that same code, and it failed: https://github.com/microsoft/vscode-cmake-tools/actions/runs/10801962444

You will notice the two different runner versions in the pipelines.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Previous version that last worked:
Runner Image
Image: macos-14-arm64
Version: 20240903.5
Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/20240903.5/images/macos/macos-14-arm64-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20240903.5

Version that doesn't work:
Runner Image
Image: macos-14-arm64
Version: 20240911.3
Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/202409[11](https://github.com/microsoft/vscode-cmake-tools/actions/runs/10801962444/job/30207614436#step:1:12).3/images/macos/macos-14-arm64-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20240911.3

Is it regression?

Yes, 20240903.5, https://github.com/microsoft/vscode-cmake-tools/actions/runs/10801962444/job/29963128203#logs

Expected behavior

The CI to pass. Ninja should be on PATH when our tests run

Actual behavior

The CI is failing, Ninja isn't being found on path.

Specifically, it seems that VSCode is not started with the appropriate PATH. I tested a simple task run: ninja --version and it gets the version, but then VS Code doesn't. However, I've tested with macos-13 and everything passes and works as expected.

Repro steps

1. Install CMake and Ninja and add to path, we use the lukka/[email protected] GitHub Action.
2. Run our unit tests which expect Ninja on path. Any test or command that expects Ninja to be on PATH should work. See our failing unit tests to see which tests are impacted, see link above.

[hoxbro]

I think it is the same issue as #10624 (we filed the issue at the same time)

[erik-bershel]

Hey @gcampbell-msft!

I'll close this as a duplicate of #10624. I'm pretty sure you were struck by the same root cause.