Update deployment script to accept python3 WMAgent service #10302

amaltaro · 2021-02-22T16:12:59Z

Status

tested

Description

Add a new argument to the deploy-wmagent script such that we can specify which RPM we would like to deploy:

wmagent (python2)
wmagentpy3 (python3 stack)

NOTE: when deploying a python3 agent, we need to update the env.sh script to point to:

export install=/data/srv/wmagent/current/install/wmagentpy3
export config=/data/srv/wmagent/current/config/wmagentpy3

instead of:

export install=/data/srv/wmagent/current/install/wmagent
export config=/data/srv/wmagent/current/config/wmagent

Is it backward compatible (if not, which system it affects?)

yes

Related PRs

none

External dependencies / deployment changes

Deployment changes: dmwm/deployment#1034

cmsdmwmbot · 2021-02-22T16:27:34Z

Jenkins results:

Unit tests: succeeded
- 3 tests no longer failing
Pylint check: succeeded
Pylint py3k check: succeeded
- 0 errors and warnings that should be fixed
- 0 warnings
- 0 comments to review
Pycodestyle check: succeeded
Python3 compatibility checks: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/11287/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2021-02-22T16:53:48Z

Jenkins results:

Unit tests: succeeded
- 3 tests no longer failing
Pylint check: succeeded
Pylint py3k check: succeeded
- 0 errors and warnings that should be fixed
- 0 warnings
- 0 comments to review
Pycodestyle check: succeeded
Python3 compatibility checks: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/11288/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2021-02-22T17:03:30Z

Jenkins results:

Unit tests: succeeded
- 3 tests no longer failing
Pylint check: succeeded
Pylint py3k check: succeeded
- 0 errors and warnings that should be fixed
- 0 warnings
- 0 comments to review
Pycodestyle check: succeeded
Python3 compatibility checks: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/11289/artifact/artifacts/PullRequestReport.html

amaltaro · 2021-02-23T07:44:19Z

With the proposed changes, WMAgent and all its dependencies can be installed. Unfortunately things fail to get initialized and the final agent deployment setup cannot be completed. This is the error message from the logs:

*** Initializing the agent ***
Initialising Agent...
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.4.6.pre7/sw/slc7_amd64_gcc630/cms/wmagentpy3/1.4.6.pre7/bin/wmagent-mod-config", line 14, in <module>
    standard_library.install_aliases()
  File "/data/srv/wmagent/v1.4.6.pre7/sw/slc7_amd64_gcc630/external/py3-future/0.18.2/lib/python3.8/site-packages/future/standard_library/__init__.py", line 453, in install_aliases
    __import__(newmodname)
ImportError: No module named reprlib
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.4.6.pre7/sw/slc7_amd64_gcc630/cms/wmagentpy3/1.4.6.pre7/bin/wmcore-db-init", line 19, in <module>
    from WMCore.Configuration import loadConfigurationFile
ImportError: No module named WMCore.Configuration
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.4.6.pre7/sw/slc7_amd64_gcc630/cms/wmagentpy3/1.4.6.pre7/bin/wmagent-couchapp-init", line 9, in <module>
    standard_library.install_aliases()
  File "/data/srv/wmagent/v1.4.6.pre7/sw/slc7_amd64_gcc630/external/py3-future/0.18.2/lib/python3.8/site-packages/future/standard_library/__init__.py", line 453, in install_aliases
    __import__(newmodname)
ImportError: No module named reprlib
Done!

and we are ending up with a mix of python2 and python3 libraries (we haven't yet migrated away of some python2-only libraries), examples are:

couchapp (there is no support to python3): https://github.com/cms-sw/cmsdist/blob/comp_gcc630/couchapp.spec
we might have to build condor python bindings under python3 (even better if we manage to build it via pip): https://github.com/cms-sw/cmsdist/blob/comp_gcc630/condor.spec
need a python3 package for pystack: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/pystack.spec
need a python3 package for cmsmonitoring: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/cmsmonitoring.spec
and need a python3 package for https://github.com/cms-sw/cmsdist/blob/comp_gcc630/dbs3-client.spec

I will make some new issues soon.

amaltaro · 2021-02-23T18:01:26Z

In terms of deployment changes, I think this covers what is needed. The whole wmagentpy3 deployment is not fully working yet because there are many other issues with RPMs and their dependencies.

todor-ivanov

@amaltaro I just made two comments inline. I hope you have already tested the current deployment script with the python3 version of the rpm package and it is working, but those lines that I am pointing to seem confusing. Could you please take a look.

todor-ivanov · 2021-02-24T07:36:02Z

deploy/deploy-wmagent.sh

@@ -285,12 +293,15 @@ echo -e "\n*** Applying agent patches ***"
 if [ "x$PATCHES" != "x" ]; then
  cd $CURRENT_DIR
  for pr in $PATCHES; do
-    wget -nv https://github.com/dmwm/WMCore/pull/$pr.patch -O - | patch -d apps/wmagent/lib/python2*/site-packages/ -p 3
+    wget -nv https://github.com/dmwm/WMCore/pull/$pr.patch -O - | patch -d apps/${RPM_NAME}/lib/python2*/site-packages/ -p 3


I am pretty sure you have already tested it, but just my curiosity: Why hard coded path leading to python2 libraries even though here we provide the option to deploy a python3 version of the rpm package

Good catch! This needs to be fixed.

todor-ivanov · 2021-02-24T07:45:15Z

deploy/deploy-wmagent.sh

 done
 set +e

+echo -e "\n*** Creating wmagent symlink ***"
+cd $CURRENT_DIR
+ln -s ../sw/${WMA_ARCH}/cms/${RPM_NAME}/${WMA_TAG} apps/wmagent


Here you seem to create the link name always as apps/wmagent, which to me is logical, but down bellow you refer to it as apps/${RPM_NAME} (line 296 in the current commit)

I had to create this symlink because some of those bin/* scripts look for files under apps/wmagent, which does not exist when we deploy wmagentpy3.
The Deploy script automatically creates a symlink according to the package name. So, for python2, it won't make any difference since the symlink will be already there. For python3, we will have both symlinks pointing to the same directory structure.

In short, I could simply use apps/wmagent below as well, but I tried to keep it as consistent as possible.

update MANAGE_DIR create wmagent symlink fix reference to python directory

amaltaro · 2021-02-24T08:57:06Z

Thanks for the review, Todor. I fixed one issue that you spotted.
Just wanted to mention that, ideally, I'd like to use this new RPM/package name (wmagentpy3) only for a transition period. Once WMCore/WMAgent is fully validated and functional in python3, we could likely rename it back to wmagent and drop the python2-based agent.

cmsdmwmbot · 2021-02-24T09:39:49Z

Jenkins results:

Unit tests: succeeded
- 4 tests no longer failing
- 1 changes in unstable tests
Pylint check: succeeded
Pylint py3k check: succeeded
- 0 errors and warnings that should be fixed
- 0 warnings
- 0 comments to review
Pycodestyle check: succeeded
Python3 compatibility checks: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/11299/artifact/artifacts/PullRequestReport.html

todor-ivanov

looks good to me

todor-ivanov · 2021-02-24T10:38:21Z

Thanks @amaltaro. the temporary name sounds logical to me too.

amaltaro · 2021-02-24T10:47:33Z

Thanks Todor.

amaltaro added the PR: Do not merge yet label Feb 22, 2021

amaltaro force-pushed the fix-10230 branch 2 times, most recently from c09de30 to 18edc37 Compare February 22, 2021 16:45

amaltaro added the PR: Work in progress label Feb 23, 2021

This was referenced Feb 23, 2021

CouchApp third-party library does not support python3 #10303

Closed

Build pystack DMWM package on python3 #10304

Closed

Build cmsmonitoring package on python3 #10305

Closed

Build dbs3-client/dbs3-pycurl-client on a python3 stack #10306

Closed

amaltaro removed the PR: Work in progress label Feb 23, 2021

amaltaro requested a review from todor-ivanov February 23, 2021 18:01

todor-ivanov reviewed Feb 24, 2021

View reviewed changes

Update deployment script to accept python3 WMAgent service

08c8db2

update MANAGE_DIR create wmagent symlink fix reference to python directory

amaltaro force-pushed the fix-10230 branch from 18edc37 to 08c8db2 Compare February 24, 2021 08:54

amaltaro requested a review from todor-ivanov February 24, 2021 09:44

todor-ivanov approved these changes Feb 24, 2021

View reviewed changes

amaltaro removed the PR: Do not merge yet label Feb 24, 2021

amaltaro merged commit 4f85ae1 into dmwm:master Feb 24, 2021

todor-ivanov mentioned this pull request Apr 16, 2021

Remove/replace MySQL-python library by something else that supports python3 #9805

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update deployment script to accept python3 WMAgent service #10302

Update deployment script to accept python3 WMAgent service #10302

amaltaro commented Feb 22, 2021 •

edited

Loading

cmsdmwmbot commented Feb 22, 2021

cmsdmwmbot commented Feb 22, 2021

cmsdmwmbot commented Feb 22, 2021

amaltaro commented Feb 23, 2021

amaltaro commented Feb 23, 2021

todor-ivanov left a comment

todor-ivanov Feb 24, 2021

amaltaro Feb 24, 2021

todor-ivanov Feb 24, 2021 •

edited

Loading

amaltaro Feb 24, 2021

amaltaro commented Feb 24, 2021

cmsdmwmbot commented Feb 24, 2021

todor-ivanov left a comment

todor-ivanov commented Feb 24, 2021

amaltaro commented Feb 24, 2021

Update deployment script to accept python3 WMAgent service #10302

Update deployment script to accept python3 WMAgent service #10302

Conversation

amaltaro commented Feb 22, 2021 • edited Loading

Status

Description

Is it backward compatible (if not, which system it affects?)

Related PRs

External dependencies / deployment changes

cmsdmwmbot commented Feb 22, 2021

cmsdmwmbot commented Feb 22, 2021

cmsdmwmbot commented Feb 22, 2021

amaltaro commented Feb 23, 2021

amaltaro commented Feb 23, 2021

todor-ivanov left a comment

Choose a reason for hiding this comment

todor-ivanov Feb 24, 2021

Choose a reason for hiding this comment

amaltaro Feb 24, 2021

Choose a reason for hiding this comment

todor-ivanov Feb 24, 2021 • edited Loading

Choose a reason for hiding this comment

amaltaro Feb 24, 2021

Choose a reason for hiding this comment

amaltaro commented Feb 24, 2021

cmsdmwmbot commented Feb 24, 2021

todor-ivanov left a comment

Choose a reason for hiding this comment

todor-ivanov commented Feb 24, 2021

amaltaro commented Feb 24, 2021

amaltaro commented Feb 22, 2021 •

edited

Loading

todor-ivanov Feb 24, 2021 •

edited

Loading