Run WMCore central services validation for HG2203

[amaltaro]

Impact of the new feature
WMCore central services

Is your feature request related to a problem? Please describe.
Monthly task

Describe the solution you'd like
Validate central services in cmsweb-testbed (well, it might have to be in one of our VMs due to the current Rucio setup) and provide the final feedback by the March deadline specified by the CMSWEB team.

It also includes the creation of the service release notes and the validation check-list twiki.

Describe alternatives you've considered
none

Additional context
none

[todor-ivanov]

Just a NOTE:

For fixing some of the broken templates, we will need the patch the WMCore source with [1] before injecting.

[1]
#10864

[todor-ivanov]

Yet another problem to be solved, upon pymongo upgrade. With ms-output this time. The container fails to start, so no logs are preserved as usual in vocms0750 for this pod. Here is the log/output from the container itself at startup time:

MongoDB is not running, check in 18 sec
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/srv/HG2203c/sw/slc7_amd64_gcc630/external/py3-pymongo/4.0.1/lib/python3.8/site-packages/pymongo/__init__.py", line 66, in <module>
    from pymongo.collection import ReturnDocument
  File "/data/srv/HG2203c/sw/slc7_amd64_gcc630/external/py3-pymongo/4.0.1/lib/python3.8/site-packages/pymongo/collection.py", line 20, in <module>
    from collections import abc
ImportError: cannot import name abc
MongoDB is not running, check in 19 sec

[todor-ivanov]

Some more info here. I just managed to reproduce the error in my VM. I have deployed the reqmgr2ms spec from HG2203 and just run the environment setup and one import from pymongo:

$ .  ./sw/slc7_amd64_gcc630/cms/reqmgr2ms/1.0.1.pre5/etc/profile.d/init.sh 
$ python
Python 2.7.5 (default, Nov 16 2020, 22:23:17) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymongo import REturnDocument
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/srv/HG2203c/sw/slc7_amd64_gcc630/external/py3-pymongo/4.0.1/lib/python3.8/site-packages/pymongo/__init__.py", line 66, in <module>
    from pymongo.collection import ReturnDocument
  File "/data/srv/HG2203c/sw/slc7_amd64_gcc630/external/py3-pymongo/4.0.1/lib/python3.8/site-packages/pymongo/collection.py", line 20, in <module>
    from collections import abc
ImportError: cannot import name abc


>>> from pymongo import IndexModel
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/srv/HG2203c/sw/slc7_amd64_gcc630/external/py3-pymongo/4.0.1/lib/python3.8/site-packages/pymongo/__init__.py", line 66, in <module>
    from pymongo.collection import ReturnDocument
  File "/data/srv/HG2203c/sw/slc7_amd64_gcc630/external/py3-pymongo/4.0.1/lib/python3.8/site-packages/pymongo/collection.py", line 20, in <module>
    from collections import abc
ImportError: cannot import name abc


>>> from pymongo import errors
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/srv/HG2203c/sw/slc7_amd64_gcc630/external/py3-pymongo/4.0.1/lib/python3.8/site-packages/pymongo/__init__.py", line 66, in <module>
    from pymongo.collection import ReturnDocument
  File "/data/srv/HG2203c/sw/slc7_amd64_gcc630/external/py3-pymongo/4.0.1/lib/python3.8/site-packages/pymongo/collection.py", line 20, in <module>
    from collections import abc
ImportError: cannot import name abc

[todor-ivanov]

And here is the set of PRs which did solve the problems from above:

• Call python3 explicitely in check4db() at manage script for reqmgr2ms deployment#1132
• https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/132
• https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/133
• https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/135

[todor-ivanov]

And here are the validation results:
https://twiki.cern.ch/twiki/bin/view/CMS/WMAgentEndtoEndValidationHG2203

@amaltaro Could you please take a look, before we give the green light to Imran.

Thanks!

[todor-ivanov]

The services are now deployed in production, up and running.
I am closing this issue.