Allow list of multiple architectures #11228

khurtado · 2022-07-21T17:18:32Z

Fixes #10674

Status

In development

Description

Modify condor requirements to allow multiple architectures
Modify Scram class so that executors match the target architecture

Is it backward compatible (if not, which system it affects?)

YES

cmsdmwmbot · 2022-07-21T17:41:11Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 1 tests no longer failing
- 1 changes in unstable tests
Python3 Pylint check: failed
- 7 warnings and errors that must be fixed
- 4 warnings
- 53 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 21 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13427/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-07-21T23:09:34Z

Jenkins results:

Python3 Unit tests: failed
- 3 new failures
- 1 tests no longer failing
- 1 changes in unstable tests
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 6 warnings
- 99 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 28 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13428/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-07-21T23:11:58Z

Jenkins results:

Python3 Unit tests: failed
- 3 new failures
- 1 tests no longer failing
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 6 warnings
- 99 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 28 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13429/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-07-28T15:34:31Z

Jenkins results:

Python3 Unit tests: failed
- 137 new failures
- 226 tests deleted
- 4 changes in unstable tests
Python3 Pylint check: failed
- 10 warnings and errors that must be fixed
- 4 warnings
- 53 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 31 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13457/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-07-28T15:51:50Z

Jenkins results:

Python3 Unit tests: failed
- 3 new failures
- 1 changes in unstable tests
Python3 Pylint check: failed
- 12 warnings and errors that must be fixed
- 6 warnings
- 100 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 30 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13459/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-07-28T16:17:20Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 2 changes in unstable tests
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 6 warnings
- 100 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 30 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13460/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-07-28T16:46:31Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 1 changes in unstable tests
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 6 warnings
- 103 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 32 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13461/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-07-28T17:32:39Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 2 changes in unstable tests
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 6 warnings
- 103 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 34 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13463/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-07-28T18:21:32Z

Jenkins results:

Python3 Unit tests: succeeded
- 3 changes in unstable tests
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 6 warnings
- 103 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 34 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13464/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-07-31T11:41:30Z

Jenkins results:

Python3 Unit tests: succeeded
- 3 changes in unstable tests
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 6 warnings
- 103 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 34 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13466/artifact/artifacts/PullRequestReport.html

khurtado · 2022-08-01T14:59:25Z

CMSSW and Python look healthy:

Local time: Sun Jul 31 16:29:24 CEST 2022
Hostname:   r234n01.m100.cineca.it
System:     Linux r234n01 4.18.0-147.13.2.el8_1.ppc64le #1 SMP Wed May 13 15:23:36 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux
Arguments:  spiga_TC_SLC7_Marconi_TB_CMS_Marconi_220731_160802_6847-Sandbox.tar.bz2 977 0

2022-07-31 16:29:26,695:INFO:Startup:Python path: ['/m100_scratch/userexternal/dspiga00/glidein_work_area_nogpu/glide_bcIp0L/execute/dir_87019/job', '/cvmfs/cms.cern.ch/slc7_ppc64le_gcc820/external/py2-future/0.18.2/lib/python3.8/site-packages', '/m100_scratch/userexternal/dspiga00/glidein_work_area_nogpu/glide_bcIp0L/execute/dir_87019/job/WMCore.zip', '/m100_scratch/userexternal/dspiga00/glidein_work_area_nogpu/glide_bcIp0L/execute/dir_87019/job', '/cvmfs/cms.cern.ch/slc7_ppc64le_gcc820/external/python3/3.8.2/lib/python38.zip', '/cvmfs/cms.cern.ch/slc7_ppc64le_gcc820/external/python3/3.8.2/lib/python3.8', '/cvmfs/cms.cern.ch/slc7_ppc64le_gcc820/external/python3/3.8.2/lib/python3.8/lib-dynload', '/cvmfs/cms.cern.ch/slc7_ppc64le_gcc820/external/python3/3.8.2/lib/python3.8/site-packages']


2022-07-31 16:29:31,069:INFO:CMSSW:    Invoking command:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH
/cvmfs/cms.cern.ch/slc7_ppc64le_gcc820/external/python3/3.8.2/bin/python3 -m WMCore.WMRuntime.ScriptInvoke WMTaskSpace.cmsRun1 SetupCMSSWPset

I have squashed the commits and I think this is ready for review

cmsdmwmbot · 2022-08-01T15:12:16Z

Jenkins results:

Python3 Unit tests: succeeded
- 1 tests no longer failing
- 3 changes in unstable tests
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 6 warnings
- 103 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 34 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13467/artifact/artifacts/PullRequestReport.html

amaltaro

Thanks Kenyi, it's looking good for me. I have left a few comments along the code for your consideration as well. In addition to those, can you please check the pycodestyle report for test/python/WMCore_t/BossAir_t/BasePlugin_t.py (there are a few new items reported here).

src/python/WMCore/WMRuntime/Tools/Scram.py

amaltaro · 2022-08-01T15:37:34Z

test/python/WMCore_t/BossAir_t/BasePlugin_t.py

-        self.assertEqual(bp.scramArchtoRequiredArch(
-                ['slc7_amd64_gcc10', 'slc7_aarch64_gcc700', 'slc7_ppc64le_gcc9']), 'X86_64')
-        self.assertEqual(bp.scramArchtoRequiredArch(['slc7_aarch64_gcc700', 'slc7_ppc64le_gcc9']), 'ppc64le')
+        test = (bp.scramArchtoRequiredArch(['slc7_amd64_gcc10', 'slc7_aarch64_gcc700'])).split(',')


I think one pair of parenthesis here is not really needed (the first one).

I think I got an error with .spllit() without the parenthesis

cmsdmwmbot · 2022-08-01T18:34:06Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 1 tests no longer failing
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 6 warnings
- 102 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 34 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13469/artifact/artifacts/PullRequestReport.html

amaltaro

It looks good to me, thanks Kenyi.

@belforte FYI, in case you are using one of these APIs mapping ScramArch to OS/Arch.

khurtado added PR: Work in progress PR: Do not merge yet labels Jul 21, 2022

khurtado force-pushed the archs branch from b684b22 to 193f6fa Compare July 21, 2022 23:00

mapellidario mentioned this pull request Jul 22, 2022

review new WMCore Runtim Doc and update code as needed dmwm/CRABServer#7084

Closed

khurtado force-pushed the archs branch from 193f6fa to 1661a4f Compare July 28, 2022 15:23

khurtado force-pushed the archs branch from 1661a4f to 4a3a4bc Compare July 28, 2022 15:38

khurtado force-pushed the archs branch from 4a3a4bc to 36e5b74 Compare July 28, 2022 16:02

khurtado force-pushed the archs branch from 9addace to e477673 Compare July 28, 2022 17:16

khurtado force-pushed the archs branch from e477673 to d06f34d Compare July 28, 2022 18:08

khurtado force-pushed the archs branch from 4a238db to c2b45ec Compare August 1, 2022 14:58

khurtado requested a review from amaltaro August 1, 2022 14:59

amaltaro requested changes Aug 1, 2022

View reviewed changes

khurtado added 2 commits August 1, 2022 14:24

Allow list of multiple architectures

6e870eb

Fix unit test

654e13e

khurtado force-pushed the archs branch from c2b45ec to 654e13e Compare August 1, 2022 18:24

khurtado requested a review from amaltaro August 1, 2022 18:26

amaltaro removed the PR: Do not merge yet label Aug 2, 2022

amaltaro removed the PR: Work in progress label Aug 2, 2022

amaltaro approved these changes Aug 2, 2022

View reviewed changes

amaltaro merged commit 1b8d18e into dmwm:master Aug 2, 2022

mapellidario mentioned this pull request Feb 26, 2023

Support jobs on non-x86 architectures dmwm/CRABServer#7208

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow list of multiple architectures #11228

Allow list of multiple architectures #11228

khurtado commented Jul 21, 2022 •

edited

Loading

cmsdmwmbot commented Jul 21, 2022

cmsdmwmbot commented Jul 21, 2022

cmsdmwmbot commented Jul 21, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 31, 2022

khurtado commented Aug 1, 2022 •

edited

Loading

cmsdmwmbot commented Aug 1, 2022

amaltaro left a comment

amaltaro Aug 1, 2022

khurtado Aug 1, 2022

cmsdmwmbot commented Aug 1, 2022

amaltaro left a comment

Allow list of multiple architectures #11228

Allow list of multiple architectures #11228

Conversation

khurtado commented Jul 21, 2022 • edited Loading

Status

Description

Is it backward compatible (if not, which system it affects?)

cmsdmwmbot commented Jul 21, 2022

cmsdmwmbot commented Jul 21, 2022

cmsdmwmbot commented Jul 21, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 28, 2022

cmsdmwmbot commented Jul 31, 2022

khurtado commented Aug 1, 2022 • edited Loading

cmsdmwmbot commented Aug 1, 2022

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro Aug 1, 2022

Choose a reason for hiding this comment

khurtado Aug 1, 2022

Choose a reason for hiding this comment

cmsdmwmbot commented Aug 1, 2022

amaltaro left a comment

Choose a reason for hiding this comment

khurtado commented Jul 21, 2022 •

edited

Loading

khurtado commented Aug 1, 2022 •

edited

Loading