Recent Ubuntu VM Ruby updates

[dentuzhik]

Hey guys!

Updates to Ubuntu VMs on 15.04.2021 broke our CI builds because suddenly a version of ruby which we relied on has been completely gone from the VMs. We are quite sure now that the latest version has been bumped to 2.7.3, without keeping 2.7.2 available.

This can be also somewhat seen here:
b20eda7#diff-6be9f9ce03ba57ca01e4ae937d5c2bb04893520fdf391ea87eb46acbc43ecdc5L202-R206

Now we know that "recommended" way of consuming ruby is to refer to 2.7 or 2.7.x version, to avoid locking on the patch, however in our place it's not practically possible, since we also have a fleet of self-hosted MacOS machines where we use rbenv, which does not really understand semver and requires us provide patches.

We could also keep precise numbers only on mac and rbenv, but send “open range” to GH, however this could create version mismatches, where ubuntu runners would have likely higher patch versions than MacOS.

So far it seems to us that occasional failures are better that version mismatches. This means that, unfortunately this issue will happen again for us, because we have no way to tell how and when do you deploy updates to virtual env.

There's an "alternative setup" which could provide us an "open version" on the self-hosted runners, using a fresh ruby/setup-ruby action, however it's considerably more cumbersome to setup and maintain comparing to rbenv (especialy considering combo of X86 and M1 runners).

---

This issue gave us roughly a 2 hour downtime on our CI to identify root cause of the issue and update all the consumers which rely on that version of ruby. I am closely watching this repository and I might have missed some announcement, but I honestly couldn't find anything post-factum related specifically to ruby, so I assume it was done just as "planned image maintenance".

One thing which could help us - is to reference exact snapshot of Ubuntu VM, without any eager updates being pushed (currently we use ubuntu-latest). At the time of writing, there's no real way we know of to provide such a strict lock. Neither docs refer that.
https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idruns-on

Considering that this is not the first time that eager updates to VMs break consumer CI (I have seen a few other discussions in this repo), we would be glad to an overhead of a full control on when/how to update for increased stability.
Most recent one we have been affected by is this: #2420

Do you have any suggestions for us on how to avoid such issues in the future?

---

One more thing which made this a little bit worse, that all the discussions and issue creation was temporarily locked on this repository. I don't know whether it was a mistake on the repo config side or a "planned lock", but without any possible place to report or get support, or get any way of realtime status of VM upgrades, it felt quite frustrating.

Or is there any way to monitor VM updates or contact GH in such cases without going through this repo?
We are on Github Enterpirse Cloud plan, if that makes a difference.

---

I hope it didn't sound too negative, we are just trying to mitigate future issues, so any advices are appreciated :)

[maxim-lobanov]

Hello @dentuzhik, thank you for reaching us!

1. Yes, bumping pre-installed Ruby on Hosted images is expected behavior. Our policy is installing latest patch for Ruby 2.5, 2.6, 2.7, 3.0. It is documented in actions/setup-ruby readme:

This action sets up a ruby environment for versions which are installed on the Actions Virtual Environments.
Virtual environments contain only one Ruby version within a 'major.minor' release, and are updated with new releases. Hence, a workflow should only be bound to minor versions.

Unfortunately, we can't bake every patch version into the images due to maintenance concerns.

2. The recommended solution is using ruby/setup-ruby action.
   actions/setup-ruby was deprecated some time ago in favor of ruby/setup-ruby because ruby/setup-ruby is maintained by Ruby community and provide better support / features.

I am absolutely agree that running against different Ruby versions on local box and in CI is not acceptable because of version mismatch. It is one more reason why we strongly recommend to switch to ruby/setup-ruby action. This action provides ability to install Ruby on-flight even it is not found locally on machine. It means that you can specify the exact Ruby version in ruby/setup-ruby like 2.7.2 and your builds will be never broken in future with such kind of changes.
While 2.7.2 is cached on images, it will be used from cache. When it is removed from images, action will download it in runtime and install it on-flight (usually, it takes ~2-3 seconds)

One more thing which made this a little bit worse, that all the discussions and issue creation was temporarily locked on this repository.

Yep, sorry for that. Repository was temporary blocked because of incorrect moderator configuration. Hope it will never happen again 😄

[dentuzhik]

Thanks for a reply 🙇🏼

We had a plan to migrate to ruby/setup-ruby, and if it provides an option to download / install ruby if cache is missing, it should technically eliminate future issues allowing us to gradually switch a version to the one available in the new image, thus completely avoiding downtimes.

---

While it technically solves the problem in case of ruby, this solution is heavily tool-dependent and relies on existence of an action which manages this case. This is not given that another dependency has the same mechanism available.

Relying on latest versions is nice for non-enterprise cases, since you get all greatest updates, but for enterprise use-cases, considering that on-prem infrastructure or a combination of runners is very likely, this problem will keep re-surfacing, because it's not realistically possible to protect against backwards incompatible changes or similar to our cases for the entire tooling stack.

We would really would prefer NOT to manage our own VMs for Linux machines and rely on infra provided by Github (and happily pay for this), however if it means that we have regular downtimes, eventually we will have to switch to something under our control.

It would be interesting to hear what is the rationale behind current design decision of not having some more versions of VM snapshots which do not change so frequently. Maybe somebody can state it better in README, if this is a decision which is not ever planned to be re-considered in the future. If it's already described, feel free to point to that documentation :)

[maxim-lobanov]

The main reasons:

• Applying specific image version on VM is not quick process so if we would apply image version on-demand, the build queue time would be tragically increased
• Having a set of machines for every image version will lead to maintenance hell where we will have hundreds image version with just a few customers on every and it won't be possible to maintain VM capacity for every image version properly.

Selecting the specific image version would be a great feature (a lot of customers asking about it) and it is something that we definitely consider for future updates but for now GitHub Actions doesn't provide such functionality.
There is an issue on GitHub roadmap github/roadmap#161 that will provide ability to run build on custom image (read "freeze any image version") and you won't need to maintain own VMs / infra. But unfortunately, there is no ETA on this issue.

[dentuzhik]

Thanks for the explanation, we will look forward to have custom images available 🙇🏼