Update and enhance the test environment #215

gildegoma · 2020-07-23T10:32:19Z

As a prerequisite to further works (e.g. rebasing #52), I propose here to refresh the test environment with following changes:

Replace outdated versions of CentOS, Ubuntu and Debian by new versions
Modify tests/assertions.yml to provide stronger validation of rvm behaviours
Integrate tests/root.yml in Travis CI build
Use latest versions of Ruby, Ansible and docker-compose in Travis CI build environment

pkuczynski · 2020-07-24T12:37:45Z

Looks good, but is it worth dropping old systems, instead of keeping them and adding newer ones?

gildegoma · 2020-07-25T05:52:56Z

is it worth dropping old systems [...] ?

Sorry, I forgot to give my rationale about this proposition of removals.

It is mainly motivated by the fact that these versions are (or will "soon" be) end of life (see summary and references below), and thus rvm1-ansible project may not want to support them any longer.
As a wild guess, I thought that the goal was here to test and support the two last major releases of each operating system platforms.

EOL Dates Summary:

CentOS 6: Full Updates: 20017-05-10 and Maintenance Updates: 2020-11-30
Debian 8: Full Updates: 2018-06-17 and Security Updates: 2020-06-30
Ubuntu 16.04: Standard Support: April 2021 and EOL: April 2024
Ubuntu 14.04: Standard Support: April 2019 and EOL: April 2022

Please let me know the list of systems you want to keep in the CI setup, and I'll update it.

References:

gildegoma · 2020-07-25T06:16:31Z

@pkuczynski I added c5594e1 as an example, but I'll adapt it as soon as you decide which versions you want to keep in the test matrix.

derekgottlieb · 2020-07-25T12:20:41Z

Is it overly aggressive to drop OS versions that are still supported upstream? Should we wait until they become EOL and put up quick PRs to remove them at that time?

gildegoma · 2020-07-25T13:39:29Z

As I said, your list will be my list 😉

PS: I am baking a next PR adding support for Arch Linux 😇.

pkuczynski · 2020-07-25T17:20:28Z

I would keep testing old systems even beyond the EOL as long as it's not too costly. There has been many cases in RVM where people took ownership of some very old project and they had to maintain it before they could upgrade it.

In this case, having the older systems does not cost much, so I would keep them in.

gildegoma · 2020-07-25T23:51:59Z

😒 second time that the Travis build fails due to "transient problems". See here an example of rate limiting issue on https://github.com.

I'll give a try to parallelise a little bit more. Not only via Ansible from the same run, but with the different travis jobs to try to avoid such problems, and speed up the build time.

gildegoma · 2020-07-26T04:56:21Z

I think that a1bd852 is a good move. This will

Speed up the CI process (especially if the project tests against many docker images)
Help to quickly identify the failure causes (which platform or playbook is failing)
Allow to rebuild only a portion of the total build (very useful at the moment, because of these transient problems, e.g. GPG key installation failure)

gildegoma · 2020-07-26T06:25:01Z

@pkuczynski I think this PR is now ready for your review.

pkuczynski

Overall it looks good to me, with few minor comments. @thbar what you think?

tests/docker-compose.yml

pkuczynski · 2020-07-27T18:47:56Z

tests/docker-compose.yml

+    build: ./dockerfiles/centos6
+    cap_add: [ALL]
+
+  debian10:


Same here, keep ASC version ordering

up to you, but please my comment below, and "double confirm" your decision: ASC (or DESC).

Thank you

Done in 6df0663. Please confirm it 😉

@pkuczynski we keep the Ascending order, right?

tests/inventory

CHANGELOG.md

pkuczynski · 2020-07-27T18:50:51Z

Possibly we could consider moving to Github Actions in case Travis is not being fast or stable...

gildegoma · 2020-07-27T20:05:28Z

@pkuczynski about ASC vs DESC ordering in docker-compose.yml and inventory files:

I intentionally reversed it, with the intention to put the most recent stuff on the top (as you typically do with release notes in a CHANGELOG file). At this occasion, the diff looks a little bit weird, but it won't be the case in the future (if we keep the convention). If you prefer the ASC sorting convention, no problem, but I just needed to give you these details.

gildegoma · 2020-07-27T20:23:35Z

Possibly we could consider moving to Github Actions in case Travis is not being fast or stable...

In this particular case, I don't think that Travis is performing bad at all (I faced similar problems on my local testing, but didn't want to tackle this problem in the scope of this PR).

That said, moving to GitHub Actions could be an interesting move, but certainly in a distinct pull request 😉

pkuczynski · 2020-07-30T13:22:17Z

Possibly we could consider moving to Github Actions in case Travis is not being fast or stable...

In this particular case, I don't think that Travis is performing bad at all (I faced similar problems on my local testing, but didn't want to tackle this problem in the scope of this PR).

That said, moving to GitHub Actions could be an interesting move, but certainly in a distinct pull request 😉

I did move some of my projects from Travis to GHA and it was pretty smooth yet seemed to be much faster than Travis. Should be done in a separate PR of course...

pkuczynski · 2020-07-30T13:23:41Z

@pkuczynski approved overall, yet the build is failing, so we will need presumably to work on this before merging!

Yeah, I agree. If this supposed to update the test env, it should also make sure that it's green. @gildegoma can you try to fix the failing build?

CHANGELOG.md

gildegoma · 2020-07-31T21:48:04Z

The only suggestion I would make (but that will be a good topic for a separate PR) is to have variables for the 2 versions of ruby installed (currently ruby-2.6.6 and ruby-2.7.1), so that we can update in DRY fashion.

I have prepared a patch for this, ~~and will submit it as a new PR, as soon as we get this one merged~~.

Edit: Finally, this will be part of this PR (was good to use to demonstrate the new caching setup). See f5d48d3

gildegoma · 2020-07-31T22:17:45Z

yet the build is failing, so we will need presumably to work on this before merging!

About the Travis CI failures... These are transient problems (that partially motivated me to split the jobs, so one can restart the failing parts without having to wait for a full rebuild. We'll come back onto this when we'll study a migration to GitHub Actions,...

For the records, I put here some comments about the two jobs that failed (#395.1 and #395.2).

395.1: The rvm-installer script couldn't be download (from raw.githubusercontent.com, which returned a 502 error).

I'll propose an until-retries-delay construct for the related get_url task (but again outside of the scope of this PR). Same fix could be applied elsewhere

395.2: This one is quite strange (never seen in previous builds). It looks like docker-compose detects a syntax error in the docker-compose.yml file, which is not the case at all.

gildegoma · 2020-07-31T22:18:08Z

Now, I will kick these two failed jobs, for the example...

and then... (still the same commit)

- Add more recent version of CentOS, Ubuntu and Debian - Integrate tests/root.yml in Travis CI build (not only user.yml) - Use latest versions of Ruby, Ansible and docker-compose in Travis CI build environment. This change fixes failing tests due to Bundler recent versions being incompatible with Ruby 2.2 and 2.3. - Minor changes in the tests playbooks (name plays and add tags)

The new assertion logic aims to validate rvm behaviours in a more abstract fashion, instead of directly hitting some installed files (which locations depends on RVM internals, i.e. current implementation).

gildegoma · 2020-08-01T05:10:16Z

⏫ rebased to amend my previous commits (I had GPG signing disabled on my current laptop).

As requested during pull request review.

gildegoma · 2020-08-01T07:06:32Z

@Kulgar @thbar @pkuczynski I think I have addressed the requested changes, and added a few more improvements.
I know, that it's getting too large to be easily reviewed, and that's why I would like to get it merged asap. It will unlock other PRs (having CI working), and allow to provide additional improvements for the test environments via future (smaller) PR iterations.

thbar

@gildegoma excellent! Indeed I definitely vote to merge as soon as possible, because it will have great impacts on the rest of the PR (testing etc). I had a quick look today and it looks good to me. Thanks for your work on this!

.travis.yml

thbar · 2020-08-01T09:37:49Z

I see that @pkuczynski already reviewed, so that's 3 pair of eyes, I'll go ahead and merge.

gildegoma requested a review from a team as a code owner July 23, 2020 10:32

gildegoma mentioned this pull request Jul 23, 2020

Improve Bundler setup to support RVM aliases #52

Open

pkuczynski added the enhancement label Jul 24, 2020

gildegoma added a commit to gildegoma/rvm1-ansible that referenced this pull request Jul 25, 2020

Update CHANGELOG (test environment update - rvm#215)

13cd233

gildegoma added a commit to gildegoma/rvm1-ansible that referenced this pull request Jul 25, 2020

Update CHANGELOG (test environment update - rvm#215)

70ed343

gildegoma force-pushed the bump-test-environment branch from 13cd233 to 70ed343 Compare July 25, 2020 06:22

gildegoma added a commit to gildegoma/rvm1-ansible that referenced this pull request Jul 25, 2020

Update CHANGELOG (test environment update - rvm#215)

dea7aae

gildegoma force-pushed the bump-test-environment branch from 70ed343 to dea7aae Compare July 25, 2020 22:56

mergeable bot assigned gildegoma Jul 25, 2020

gildegoma added a commit to gildegoma/rvm1-ansible that referenced this pull request Jul 25, 2020

Update CHANGELOG (test environment update - rvm#215)

00c1252

gildegoma force-pushed the bump-test-environment branch from dea7aae to 00c1252 Compare July 25, 2020 23:02

gildegoma force-pushed the bump-test-environment branch 2 times, most recently from d2fab66 to a1bd852 Compare July 26, 2020 00:35

gildegoma force-pushed the bump-test-environment branch 4 times, most recently from d5c230f to bed3733 Compare July 26, 2020 05:54

pkuczynski requested changes Jul 27, 2020

View reviewed changes

gildegoma commented Jul 31, 2020

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

gildegoma added 4 commits August 1, 2020 07:08

Add system-wide tests to CI and modify assertions

5c0f0f0

The new assertion logic aims to validate rvm behaviours in a more abstract fashion, instead of directly hitting some installed files (which locations depends on RVM internals, i.e. current implementation).

README: Update 'Requirements' section

b78e334

Update CHANGELOG (test environment update - rvm#215)

30c3321

gildegoma dismissed stale reviews from thbar and pkuczynski via 8ce4e84 August 1, 2020 05:08

gildegoma force-pushed the bump-test-environment branch from 9227e45 to 8ce4e84 Compare August 1, 2020 05:08

gildegoma force-pushed the bump-test-environment branch 2 times, most recently from a4b671a to 5dfdab6 Compare August 1, 2020 06:32

gildegoma and others added 5 commits August 1, 2020 08:40

Travis CI: Parallelize the test matrix

b58aaa9

Tests: Use variables for the Ruby versions to be installed

f5d48d3

Cleanup: Sort platform versions in ascending order

58e5fd4

As requested during pull request review.

Update CHANGELOG.md

ae327bd

Travis CI: Fail fast

943ff23

gildegoma force-pushed the bump-test-environment branch from 5dfdab6 to 943ff23 Compare August 1, 2020 06:50

thbar approved these changes Aug 1, 2020

View reviewed changes

.travis.yml Show resolved Hide resolved

thbar merged commit 3165650 into rvm:master Aug 1, 2020

gildegoma deleted the bump-test-environment branch August 1, 2020 10:01

sfgeorge added a commit to sfgeorge/rvm1-ansible that referenced this pull request Mar 23, 2022

[CI] Restore "Parallelize the test matrix" behavior from rvm#215

16b6359

sfgeorge added a commit to sfgeorge/rvm1-ansible that referenced this pull request Mar 24, 2022

[CI] Restore "Parallelize the test matrix" behavior from rvm#215

7938284

sfgeorge added a commit to sfgeorge/rvm1-ansible that referenced this pull request Mar 24, 2022

[CI] Restore "Parallelize the test matrix" behavior from rvm#215

23a9d8c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update and enhance the test environment #215

Update and enhance the test environment #215

gildegoma commented Jul 23, 2020

pkuczynski commented Jul 24, 2020

gildegoma commented Jul 25, 2020 •

edited

Loading

gildegoma commented Jul 25, 2020 •

edited

Loading

derekgottlieb commented Jul 25, 2020

gildegoma commented Jul 25, 2020

pkuczynski commented Jul 25, 2020

gildegoma commented Jul 25, 2020

gildegoma commented Jul 26, 2020

gildegoma commented Jul 26, 2020

pkuczynski left a comment

pkuczynski Jul 27, 2020

gildegoma Jul 27, 2020

gildegoma Jul 28, 2020

gildegoma Jul 31, 2020

pkuczynski commented Jul 27, 2020

gildegoma commented Jul 27, 2020

gildegoma commented Jul 27, 2020

pkuczynski commented Jul 30, 2020

pkuczynski commented Jul 30, 2020

gildegoma commented Jul 31, 2020 •

edited

Loading

gildegoma commented Jul 31, 2020

gildegoma commented Jul 31, 2020 •

edited

Loading

gildegoma commented Aug 1, 2020

gildegoma commented Aug 1, 2020 •

edited

Loading

thbar left a comment

thbar commented Aug 1, 2020

Update and enhance the test environment #215

Update and enhance the test environment #215

Conversation

gildegoma commented Jul 23, 2020

pkuczynski commented Jul 24, 2020

gildegoma commented Jul 25, 2020 • edited Loading

gildegoma commented Jul 25, 2020 • edited Loading

derekgottlieb commented Jul 25, 2020

gildegoma commented Jul 25, 2020

pkuczynski commented Jul 25, 2020

gildegoma commented Jul 25, 2020

gildegoma commented Jul 26, 2020

gildegoma commented Jul 26, 2020

pkuczynski left a comment

Choose a reason for hiding this comment

pkuczynski Jul 27, 2020

Choose a reason for hiding this comment

gildegoma Jul 27, 2020

Choose a reason for hiding this comment

gildegoma Jul 28, 2020

Choose a reason for hiding this comment

gildegoma Jul 31, 2020

Choose a reason for hiding this comment

pkuczynski commented Jul 27, 2020

gildegoma commented Jul 27, 2020

gildegoma commented Jul 27, 2020

pkuczynski commented Jul 30, 2020

pkuczynski commented Jul 30, 2020

gildegoma commented Jul 31, 2020 • edited Loading

gildegoma commented Jul 31, 2020

gildegoma commented Jul 31, 2020 • edited Loading

gildegoma commented Aug 1, 2020

gildegoma commented Aug 1, 2020 • edited Loading

thbar left a comment

Choose a reason for hiding this comment

thbar commented Aug 1, 2020

gildegoma commented Jul 25, 2020 •

edited

Loading

gildegoma commented Jul 25, 2020 •

edited

Loading

gildegoma commented Jul 31, 2020 •

edited

Loading

gildegoma commented Jul 31, 2020 •

edited

Loading

gildegoma commented Aug 1, 2020 •

edited

Loading