feat: installTools step

[v1v]

What does this PR do?

Provide a new step to install tools in the CI workers using a wrapper.

Why is it important?

Avoid the same scripts to be copied across different repos.

Related issues

Related to elastic/apm-agent-python#730
and elastic/apm-agent-nodejs#1393

Questions

@elastic/observablt-robots , I need your feedback for this particular approach and also the below questions:

• apm-agent-dotnet could use a similar approach.

Current pipeline does use a ps1 to install some required tools and the Jenkinsfile does call the ps1 file, see the below snippet:

### .ci/windows/msbuild-tools.ps1 looks like 
& choco install visualstudio2019buildtools -m -y --no-progress -r --version=16.4.0.0
& choco install visualstudio2019enterprise -m -y --no-progress -r --version=16.4.0.0 --package-parameters "--includeRecommended --includeOptional"
& choco install visualstudio2019-workload-netweb -m -y --no-progress -r --version=1.0.0

### Jenkinsfile looks like
...
powershell label: 'Install tools', script: '.ci\\windows\\msbuild-tools.ps1'
...

The proposal could be to delete the ps1 and put the logic in the call to the step in the Jenkinsfile, see the below snippet:

installTools([
  [ tool: 'visualstudio2019buildtools', version: '16.4.0.0' ], 
  [ tool: 'visualstudio2019enterprise', version: '16.4.0.0' ], 
  [ tool: 'visualstudio2019-workload-netweb', version: '1.0.0']
])
...

• nodejs versions might require a different way to order them in ps1. See https://github.com/elastic/apm-agent-nodejs/pull/1393/files#diff-860c4fe077b91b560f207db63e491a6aR9 (sort {[version] $_} )

• add a provider to enable to customise what we wanna do, for instance:

installTools([
  [ tool: 'foo', version: 'x.y.z' ], 
  [ tool: 'bar', version: 'z.y.x', provider: 'choco'],
  [ tool: 'abc', version: '1.2', provider: 'ps1']
])

## where `choco` means
choco install $env:TOOL --no-progress -y --version "$Version"

## where ps1 means 
.\\install-with-choco.ps1 ### or another filename... for the time being its the current filename

• add an extraArg to append to the choco provider more options:

If the provider parameter is enabled then we might need to use an extra flag, for instance the intallation in the apm-agent-dotnet of the VS2019-enterprise requrie some extra packages, so the below snippet is what it's done so far:

### From choco install
& choco install visualstudio2019enterprise -m -y --no-progress -r --version=16.4.0.0 --package-parameters "--includeRecommended --includeOptional"

The Jenkinsfile could do something like the below call:

### To a shared library step
installTools([
  [ tool: 'visualstudio2019enterprise', version: '16.4.0.0', provider: 'choco', extraArg: '-package-parameters "--includeRecommended --includeOptional"'
])

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 63746cd

[cachedout]

This looks like it is going to be very useful!

Do we have Ansible on these machines? I am wondering if instead of calling out to the specific package manager variants if we can just call Ansible and let it figure out the details of how to get the package installed.

[kuisathaverat]

probably we have Ansible, but in any case, it is Python so we have it (in most cases).
The thing is How much complexity use Ansible will add? and how slow it is?
Choco is pretty straight forward you can the command with a couple of params and that's it, it is true tat there is an Ansible module for Choco.

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 63746cd

[cachedout]

probably we have Ansible, but in any case, it is Python so we have it (in most cases).

Ah, so this is intended just to be used with Choco or is it a generic tool installer? From the name I thought it was generic but if it's just Choco then I agree that it is best to just call it directly.

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 63746cd

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 63746cd

[v1v]

so this is intended just to be used with Choco or is it a generic tool installer?

It is a generic tool installer from my point of view.

The initial approach was to be able to install the tools we do need independently of the OS, therefore we could say installTools([ [ tool: 'python', version: '3.1.1' ]])in any OS and then it will do the magic for us.

Then, I found we could even add more specific providers, such as apt, choco, brew and so on in case we wanted to get a more fine granularity to how and what to install., so something like that could also help with:

installTools([ [ tool: 'python', version: '3.1.1' , provider: 'choco' ]]) or installTools([ [ tool: 'python', version: '3.1.1' , provider: 'apt' ]])

In both cases, the entrypoint is installTools and its params can allow us to either use the default installation, therefore, the logic to detect what's the OS version and what's the provider to be used should be delegated to the step anyway, and if provider is used then we can force the type of installation.

Ansible could be potentially another provider to be added, there is no restriction regarding what to use and how to use it, but the default behavior is the one that might require some kind of agreement.

What do you think?

[kuisathaverat]

it makes sense.

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 63746cd

[cachedout]

Ansible could be potentially another provider to be added, there is no restriction regarding what to use and how to use it, but the default behavior is the one that might require some kind of agreement.

I thinks this strikes a good balance. We have the precision to call directly into package management systems if we want, or we can use Ansible if we want a higher degree of abstraction. 👍 to this approach.

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 63746cd

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 19a64e9

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: becc647

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: becc647

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 0a84138

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 0a84138

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 0a84138

[apmmachine]

💚 Build Succeeded

• pipeline
• Commit: 2b0581b