OSX 10.15 images deprecated in azure, to be removed end of August

[h-vetinari]

Message started appearing in OSX jobs:

The macOS-10.15 environment is deprecated, consider switching to macos-11(macos-latest),
macos-12 instead. For more details see https://github.com/actions/virtual-environments/issues/5583

Quoting from the upstream issue:

Deprecation will begin on 5/31/2022 and the image will be fully unsupported by 8/30/2022

To raise awareness of the upcoming removal, we will temporarily fail jobs using macOS 10.15. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:

GitHub Actions:

• July 21, 12:00 UTC - July 22, 18:00 UTC
• July 27, 00:00 UTC - July 28, 00:00 UTC
• August 3, 00:00 UTC - August 4, 00:00 UTC
• August 15, 00:00 UTC - August 16, 00:00 UTC
• August 26, 00:00 UTC - August 27, 00:00 UTC
• August 29, 00:00 UTC - August 30, 00:00 UTC

Azure DevOps

• August 1, 12:00 UTC - August 1, 14:00 UTC
• August 9, 16:00 UTC - August 9, 18:00 UTC
• August 17, 14:00 UTC - August 17, 18:00 UTC
• August 25, 14:00 UTC - August 25, 20:00 UTC
• September 2, 12:00 UTC - September 2, 22:00 UTC
• September 8, 00:00 UTC - September 9, 00:00 UTC
• September 14, 00:00 UTC - September 15, 00:00 UTC
• September 20, 00:00 UTC - September 21, 00:00 UTC
• September 26, 00:00 UTC - September 27, 00:00 UTC

Those brownouts get pretty extensive - 24h for all but the first on GHA, and the same a bit later on Azure - so it sounds we need to move until end of July with this.

[jakirkham]

cc @conda-forge/core

[beckermr]

Is this for azure or GitHub actions or both?

[h-vetinari]

Is this for azure or GitHub actions or both?

The brownouts mentioned in actions/runner-images#5583 and the OP are for GHAs, and doesn't give a target date for azure, but the warning that caused me to open this issue was on azure

[ngam]

azure

It's the same for azure. Some builds will get this warnings, e.g. this one,

The macOS-10.15 environment is deprecated, consider switching to macos-11(macos-latest), macos-12 instead. For more details see actions/runner-images#5583

[h-vetinari]

I just checked, the oldest SDK that macos-11 comes with is 10.15, plus xcode 11.7. As far as I can tell, that version still supports deployment target 10.9 (actually 10.6-10.15), so it doesn't sound like the image change forces us to change anything substantial about MacOS support ¹

Footnotes

1. It's worth noting though that 10.9 is EOL for more than 5 years already, but moving on from that is a separate discussion. ↩

[beckermr]

Thank you!

@chrisburr had some nice stats about where we go next on osx pulled from pypi usage. I forget but moving to somewhere around 10.12 would be pretty non-destructive IIRC.

[jakirkham]

Would suggest we separate discussing the CI image change from updating the SDK used to build (if possible). Given CI is the primary driver of discussion here, let's focus on that. If we would like to discuss SDK changes, let's open a new issue (unless these are somehow intertwined).

In terms of CI image used (the primary concern here), moving macOS 11 is the next oldest thing we can use (as 10.15 is the last in the 10.x line).

[h-vetinari]

Would suggest we separate discussing the CI image change from updating the SDK used to build (if possible).

AFAIU, the SDK is not really the issue, the MACOS_DEPLOYMENT_TARGET specifies the compatibility. That said, I had assumed that we were using one of the SDK's in the current 10.15 image, but that's not the case, because we're still using the 10.9 SDK, but I couldn't easily find (e.g. in smithy) where that is injected.

So with that misunderstanding out of the way - fine by me to leave the SDK untouched (assuming the current SDK injection mechanism works in the macos-11 image).

[jakirkham]

All good. Just trying to help dial things in a bit so we can make decisions 🙂

[jaimergp]

So we have roughly a month to:

1. Update this block to macos-11 as soon as possible
2. Hope a month is enough for people to rerender in time
3. In September, remember users they still need to rerender to re-enable OSX builds

Does that sound correct?

[beckermr]

We have not historically worried about rerendering in these cases. People should rerender almost always when making new PRs.

[beckermr]

So we only need a PR to smithy.

[jaimergp]

And we don't have a reason to suspect that the current way we are installing SDKs on the image would happen to fail on MacOS 11, right? I guess we can only try... I'll open a PR on conda-smithy to see what happens.

[jaimergp]

Let's see: conda-forge/dav1d-feedstock#2

[jakirkham]

Don't think so, but agree a test is a better way to check

[jaimergp]

Everything looks ok:

   INFO: sysroot: '/Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/' files: '['usr/share/man/mann/zero.n', 'usr/share/man/mann/yencode.n', 'usr/share/man/mann/yaml.n', 'usr/share/man/mann/xsxp.n']'
   INFO (dav1d,bin/dav1d): lib/libdav1d.6.dylib found in this package
   INFO (dav1d,bin/dav1d): Needed DSO /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/libSystem.B.dylib found in $SYSROOT
   INFO (dav1d,lib/libdav1d.6.dylib): Needed DSO /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/libSystem.B.dylib found in $SYSROOT

[jaimergp]

[jakirkham]

Great! Thanks Jaime 😄

Sounds like we are ready for a conda-smithy PR & release. If we get it out today, we will have ~1 week to try things and catch/fix any issues 🙂

[jaimergp]

We have til August 30th, don't we? Not that I have anything against doing it soon, but it's not as rushed as a week :)

[h-vetinari]

We have til August 30th, don't we? Not that I have anything against doing it soon, but it's not as rushed as a week :)

Well the five 24h-long brown outs from July 27th until end of August sound pretty painful. During that time no OSX pipeline in conda-forge would run through.

[jaimergp]

Ah, hadn't considered the brown-outs. I'll add the PR to conda-smithy, then.

[jakirkham]

Ah sorry. Was thinking beginning of August for some reason 🤦‍♂️

Yeah brownouts are an issue too though

Thanks Jaime! 🙏

[jaimergp]

Just realized we also need to do this for staged-recipes. Submitted PR here: conda-forge/staged-recipes#19800

After that, I think we can consider this solved and closed.

[ngam]

And we don't have a reason to suspect that the current way we are installing SDKs on the image would happen to fail on MacOS 11, right? I guess we can only try... I'll open a PR on conda-smithy to see what happens.

No, as you figured (we also have used variations of macos-11 and macos-12 in different feedstocks, one could set this in conda-forge.yml, e.g. see this PR from a while back: #1638 and it works fines).

However, it may be worth considering trying to better understand the SDKs situation. The website/repo we get stuff from is no longer maintained, and it doesn't have many of the newer SDKs. Fortunately, for now, most of these newer SDKs come with the images anyway so it's not a big deal. However, it would likely be best to think this through again and decide how best to proceed (i.e. use another repo as shown here: conda-forge/conda-forge-ci-setup-feedstock#201 or rely entirely on what the vmImages offer for example this is what macos-11 has: https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md#installed-sdks)

[ngam]

(unless these are somehow intertwined).

They're somewhat intertwined because the images come preconfigured with certain SDKs and that's exactly what we relied on in the pytorch feedstock to support MPS (metal). But yes, this should be more thoroughly discussed elsewhere. Just wanted to point out this was on the radar before and highlight how the SDKs and vmImages relate to a degree.

After that, I think we can consider this solved and closed.

Yes, from my previous testing, this is solved and closed after the two PRs

[jaimergp]

Thanks @ngam!

[ngam]

Thank YOU for the PRs 😃

[ngam]

FYI @beckermr the change may take a bit to come online with conda-smithy (hence rerendering may not effect changes yet) because the CDN cloning has a "major outage"... https://conda-forge.org/status/

[ngam]

Also, before this is closed, we need to push a release of conda-smithy, I believe. Otherwise, the changes won't take place.

[h-vetinari]

We're now in the first brownout on azure (dates of which have been added to the upstream issue, as well as the OP).

Could we have a new smithy release?

[ngam]

Could we have a new smithy release?

@conda-forge/core @conda-forge/conda-smithy

[h-vetinari]

Felipe commented in conda-forge/conda-smithy#1645

I guess we need a new release with this ASAP.

To take the heat off a bit - it's not a crisis. The brownout is only two hours this time. But since that smithy PR has been merged (and I assume the repo is in a releasable state) it would still be good to get a new smithy release soon (i.e. before the next brownout in a week). It'll take a while anyway until feedstocks have been rerendered in large numbers.

[beckermr]

release is done - should be live in an hour or so

[h-vetinari]

With smithy & staged-recipes now done, I think we can close this. Thanks all! :)

[jakirkham]

JFYI we recently saw an (admittedly somewhat niche) issue with Python 3.7 on macOS 11 ( conda-forge/python-feedstock#575 ). Mentioning here merely for awareness.