Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MS: Update MSOutput record schema #9901

Merged
merged 2 commits into from
Sep 10, 2020
Merged

MS: Update MSOutput record schema #9901

merged 2 commits into from
Sep 10, 2020

Conversation

amaltaro
Copy link
Contributor

@amaltaro amaltaro commented Sep 9, 2020

Fixes #9797
Fixes #9610

Status

tested

Description

This pull request reflects all the discussions we had over the last few months on which mongodb record schema we should use for MSOutput documents.

In short, changes are:

  • adopted capital camelCase for the msoutput attributes
  • removed isTaken and isTakenBy
    • removed that logic from MSOutput as well, I don't expect having multiple threads consuming documents from MongoDB
  • updated some default values
  • RelVal requests are identified from within MSOutputTemplate (instead of using the pipeline for that)
  • validate the TransferStatus value, only pending | done is allowed
  • merged RelVal and NonRelVal Producer pipeline into a single one
  • updated Consumer query to only look for records in pending status
  • fetch mongodb records outside of the pipeline

Is it backward compatible (if not, which system it affects?)

no, schema changes. It is not in production though, so we can drop the mongo database.

Related PRs

none

External dependencies / deployment changes

none

@cmsdmwmbot
Copy link

Jenkins results:

  • Unit tests: succeeded
    • 5 tests added
  • Pylint check: failed
    • 5 warnings and errors that must be fixed
    • 12 warnings
    • 32 comments to review
  • Pycodestyle check: succeeded
    • 5 comments to review
  • Python3 compatibility checks: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/10402/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Unit tests: succeeded
    • 5 tests added
  • Pylint check: failed
    • 5 warnings and errors that must be fixed
    • 12 warnings
    • 32 comments to review
  • Pycodestyle check: succeeded
    • 4 comments to review
  • Python3 compatibility checks: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/10403/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Unit tests: succeeded
    • 5 tests added
  • Pylint check: failed
    • 6 warnings and errors that must be fixed
    • 12 warnings
    • 32 comments to review
  • Pycodestyle check: succeeded
    • 4 comments to review
  • Python3 compatibility checks: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/10404/artifact/artifacts/PullRequestReport.html

few bugfixes for MSOutput

change Consumer pipeline; fixed isRelval setting

make sure TransferStatus gets validated before updating it
@cmsdmwmbot
Copy link

Jenkins results:

  • Unit tests: succeeded
    • 5 tests added
  • Pylint check: failed
    • 6 warnings and errors that must be fixed
    • 12 warnings
    • 31 comments to review
  • Pycodestyle check: succeeded
    • 4 comments to review
  • Python3 compatibility checks: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/10405/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor Author

This has been tested enough and it's good to go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants