Implement policy to release stable releases

[vkuznet]

Impact of the new feature
On CMSWEB we should use images with -stable suffix to allow longer retention policies of images. The stable images will be available for 1 year in CERN registry.

Is your feature request related to a problem? Please describe.
the CMSWEB group suggest to use -stable images for all production images to keep it uniform across all services. Therefore, we should define proper policies how to create these images for all WMCore services.

Describe the solution you'd like
Write proper document how we can build and upload -stable images. This document can define all steps required and build stable images. For instance:

• we may change CI/CD to create stable images, or
• use manual step to create stable image once we perform integration tests

In this document we should define:

1. procedure to create stable images
2. time when they should be defined, e.g. as final release
3. how the integration tests on testbed should be accounted in this procedure.

Describe alternatives you've considered
None, as we can run as is using non-stable images on k8s

Additional context

[vkuznet]

@amaltaro , here is an dedicated issue about stable images.

[amaltaro]

From my side, I would say that we need to adapt the CD pipeline (github action) to - in addition to the usual docker image setup - labeling specific images with the -stable suffix.

Policy for what to tag as -stable is the following:

1. we are building docker images for a final release. Where final releases consist of 3 integer values separated by a period sign, e.g.: 2.2.4, 2.1.13, 3.0.0, 12.2.3, and so on.
2. we are building docker images for a patch release. Where patch releases consist of 4 integer values separated by a period sign, e.g.: 2.2.4.0, 2.2.4.1, 2.1.13.1, and so on.

Further information about tagging scheme in WMCore and definitions can be found in this wiki: https://github.com/dmwm/WMCore/wiki/TaggingAndReleasing#new-tagging-convention

[vkuznet]

I agree with proposed solution and can implement it. Said that I also in favor of creating separate (from workflows yaml file) script which will perform all necessary actions, like build, deploy, etc. For example:

# build new docker image
./scripts/docker.sh build $CERN_REGISTRY $SERVICE $TAG

# deploy new docker image
./scripts/docker.sh deploy $CERN_REGISTRY $SERVICE $TAG

The script will be wrapper around docker commands itself and can implement proper policies. Doing this within yaml file is not impossible but will complicate significantly its logic. While the script can be use within and outside yaml CICD files, being part of repository and managed separately. For instance, we can implement auto-deployment of services into dev cluster with every release candidate tag (as it is done for other services, like DBS/DAS) etc.

@amaltaro , let me know if it is acceptable with your vision.

[amaltaro]

Whenever people say less complication, I am all in for it!
If having a script will simplify and make this process more solid, then I have nothing against replacing the current logic in the CD pipeline with that.

For the deployment in kubernetes clusters - potentially to be used in the future - I would suggest not to spend any time with that at the moment

[vkuznet]

Alan, I prepared the script and you can view it here: https://gist.github.com/vkuznet/9d4956c9afe158757897ea8316b05085 So far I didn't put it anywhere in WMCore since I'm not sure if it belongs there. I would like to replace the CI/CD lines with this script starting from here https://github.com/dmwm/WMCore/blob/master/.github/workflows/docker_images_template.yaml#L49

I see few possibilities here:

1. put this script into CMSKubernetes such that within our CI/CD pipeline we'll get it directly from CMSKubernetes repo, see https://github.com/dmwm/WMCore/blob/master/.github/workflows/docker_images_template.yaml#L41
2. Keep it under WMCore/bin area, but this will require to adjust CI/CD yaml file to read it from local WMCore are during CI/CD execution

I would appreciate if you'll look into these details and let me know your opinion where it belongs that I can act accordingly.

[amaltaro]

Valentin, keeping it under the WMCore repository looks more manageable to me.

Regarding the script, to avoid extra complexity of finding out which image(s) have been built before we can push them upstream, I feel like the best would be to call that script only once per service, and in case it has to tag something with the stable suffix, it would basically:

• build tag
• push tag
• build tag-stable
• push tag-stable
  or whatever order we feel a best fit.

Another option would be to call this script twice for each service, e.g.:
docker.sh build reqmgr2 2.3.3
then
docker.sh push reqmgr2 2.3.3

But then we need to find out where we have or not a stable build. Which looks more inconvenient to me. HTH.

[anpicci]

Hi @amaltaro , I have pushed a commit to @vkuznet 's PR to implement some modifications needed according to some local tests.

For the time being, I would stick with the procedure suggested by Valentin, removing the remove and delete options for docker.sh, as for now they are not really needed for the CI/CD purposes. As a result, I implemented some code for the push case to check that the -stable build is present.

As it is used in the yaml routing, docker.sh build docker images as expected by the stable release policy, and then push them into the registry, with a sanity check on the existence of the -stable image.