Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updating workflow json templates; adding working PY3 templates #10761

Merged
merged 1 commit into from
Oct 7, 2021
Merged

Updating workflow json templates; adding working PY3 templates #10761

merged 1 commit into from
Oct 7, 2021

Conversation

amaltaro
Copy link
Contributor

@amaltaro amaltaro commented Aug 19, 2021 •
edited
Loading

Fixes #10654

Status

In development

Description

Another run at our json workflow templates. Changes are so far:

  • ReRecoSkim removed, since it relies on slc5_* scramArch, which is no longer supported by the WM system/cmssw-wm-tools
  • TaskChain Python3 from scratch and without pileup (using CMSSW_12_0_0_pre6)
  • TaskChain Python3 with input and pileup data (using CMSSW_12_0_0_pre6)
  • StepChain Python3 with input and pileup data (using CMSSW_12_0_0_pre6)
  • New input dataset for a couple of python3 workflows
  • Basic GPU ReReco and TaskChain templates

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

None

External dependencies / deployment changes

None

@amaltaro amaltaro changed the title Fix 10654 Updating workflow json templates; adding working PY3 templates Aug 19, 2021
@cmsdmwmbot
Copy link

Jenkins results:

  • Python2 Unit tests: succeeded
  • Python3 Unit tests: succeeded
    • 1 changes in unstable tests
  • Python2 Pylint check: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12376/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python2 Unit tests: succeeded
  • Python3 Unit tests: succeeded
    • 1 changes in unstable tests
  • Python2 Pylint check: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12382/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python2 Unit tests: succeeded
    • 1 changes in unstable tests
  • Python3 Unit tests: succeeded
  • Python2 Pylint check: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12389/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python2 Unit tests: succeeded
  • Python3 Unit tests: succeeded
    • 1 changes in unstable tests
  • Python2 Pylint check: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12437/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor Author

The following requests are using an input dataset that is no longer available in the storage (maybe invalidated):

2021-09-15 02:04:51,547:ERROR:PycurlRucio: Failure in getBlocksAndSizeRucio function for container /RelValTTbar_14TeV/CMSSW_12_0_0_pre4-PU_120X_mcRun3_2021_realistic_v2-v1/GEN-SIM-DIGI-RAW. Response: {'url': 'http://cms-rucio.cern.ch/dids/cms/dids/search?type=dataset&long=True&name=/RelValTTbar_14TeV/CMSSW_12_0_0_pre4-PU_120X_mcRun3_2021_realistic_v2-v1/GEN-SIM-DIGI-RAW%23%2A', 'data': '', 'headers': 'HTTP/1.1 200 OK\r\nServer: nginx/1.17.10\r\nDate: Wed, 15 Sep 2021 00:04:51 GMT\r\nContent-Type: application/x-json-stream\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nAccess-Control-Allow-Origin: None\r\nAccess-Control-Allow-Headers: None\r\nAccess-Control-Allow-Methods: *\r\nAccess-Control-Allow-Credentials: true\r\nCache-Control: no-cache, no-store, max-age=0, must-revalidate, post-check=0, pre-check=0\r\nPragma: no-cache\r\nX-Rucio-Host: cms-rucio.cern.ch\r\n\r\n'}
2021-09-15 02:04:51,548:WARNING:RequestInfo: Removing workflow that failed processing in MSTransferor: amaltaro_SC_PY3_PURecyc_Sept2021_Val_210914_214038_4739
2021-09-15 02:04:51,548:WARNING:RequestInfo: Removing workflow that failed processing in MSTransferor: amaltaro_TC_PY3_TTbarPU_Sept2021_Val_210914_214047_4235

I am pushing in two replacements for them.

@cmsdmwmbot
Copy link

Jenkins results:

  • Python2 Unit tests: succeeded
  • Python3 Unit tests: succeeded
    • 2 changes in unstable tests
  • Python2 Pylint check: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12447/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python2 Unit tests: succeeded
    • 8 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Unit tests: succeeded
    • 4 tests no longer failing
    • 2 changes in unstable tests
  • Python2 Pylint check: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12464/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python2 Unit tests: succeeded
    • 8 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Unit tests: succeeded
    • 4 tests no longer failing
    • 3 changes in unstable tests
  • Python2 Pylint check: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12468/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python2 Unit tests: failed
    • 10 new failures
  • Python3 Unit tests: failed
    • 10 new failures
    • 1 changes in unstable tests
  • Python2 Pylint check: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12474/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python2 Unit tests: succeeded
  • Python3 Unit tests: succeeded
    • 1 changes in unstable tests
  • Python2 Pylint check: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12481/artifact/artifacts/PullRequestReport.html

@todor-ivanov todor-ivanov mentioned this pull request Oct 1, 2021
Closed
@amaltaro
Updating workflow json templates; adding working PY3 templates
b55b863 
update StepChain Py3 template

fixing campaign

performance tweaks for the SC template

remove Requestor key/value pair

Update two python3 workflows with non-existent input dataset

New templates with GPU requirements

trying a new relval pileup for SC_PY3 template

another attempt for Py3 SC with input and pileup

correct Memory requirements for SC_PY3
@cmsdmwmbot
Copy link

Jenkins results:

  • Python2 Unit tests: failed
    • 3 new failures
    • 15 tests no longer failing
  • Python3 Unit tests: failed
    • 3 new failures
    • 13 tests no longer failing
    • 1 changes in unstable tests
  • Python2 Pylint check: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12502/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor Author

amaltaro commented Oct 7, 2021

The 3 failing unit tests are a consequence of #10855 and an likely an issue with the host certificate (Eric has already been informed).

We also had many workflows failing, and the reason is that they had files marked as invalid in the dbs k8s int database. I fixed all of those by setting those files to valid and now all the workflows are running properly. This is ready to go.

@amaltaro amaltaro merged commit e075ed8 into dmwm:master Oct 7, 2021
@amaltaro amaltaro mentioned this pull request Oct 12, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Update workflow json templates
2 participants
@amaltaro @cmsdmwmbot