Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove block open logic from DBS3Reader and WorkQueue #11123

Merged
merged 2 commits into from
May 2, 2022

Conversation

amaltaro
Copy link
Contributor

@amaltaro amaltaro commented May 1, 2022

Fixes #11120
Partial fix for #11048

Status

not-tested

Description

As described in the issue, this feature was needed back in the days of DBS2.

This PR removes DBS/WorkQueue logic related to:

  • DBS blocks that could be open (onlyClosedBlocks parameter, or open_for_writing, or OpenForWriting)
  • DBS3Reader APIs will no longer return a key/value pair for IsOpen
  • DBS3Reader method blockIsOpen has been removed
  • DBS3Reader method listOpenFileBlocks has been removed (use listFileBlocks instead)
  • (unused?) DBSWriterObjects function getDBSFileBlock has been removed
  • WorkQueue Block start policy now defaults OpenForNewData to False
  • WorkQueue method addNewFilesToOpenSubscriptions has been removed, since we no longer add new files to open subscriptions (To be tested: what happens when new elements are pulled down?)
  • updated emulators code

Is it backward compatible (if not, which system it affects?)

NO (in the sense that it removes a feature)

Related PRs

PR #11119 will have to be rebased once this one goes through.

External dependencies / deployment changes

None

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 1 tests deleted
    • 7 tests added
  • Python3 Pylint check: failed
    • 35 warnings and errors that must be fixed
    • 12 warnings
    • 176 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13111/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 2 tests deleted
    • 7 tests added
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 34 warnings and errors that must be fixed
    • 12 warnings
    • 176 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13112/artifact/artifacts/PullRequestReport.html

@amaltaro amaltaro requested review from vkuznet and todor-ivanov May 1, 2022 13:43
@amaltaro
Copy link
Contributor Author

amaltaro commented May 1, 2022

FYI @belforte Stefano, I removed another DBS3Reader method that wasn't mentioned in the initial GH issue. Please check out the first description of this PR with a summary to make sure it won't catch you by surprise ;)

@belforte
Copy link
Member

belforte commented May 1, 2022

thanks @amaltaro . No problem for CRAB, none of those was used.

Copy link
Contributor

@todor-ivanov todor-ivanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @amaltaro
Looking good to me. Please give us further info upon testing.


logging.warning("Open FileBlock located at PNN: %s to use is FileBlock: %s ", pnn, blockRef['Name'])
return blockRef
return DbsFileBlock( Name = blockName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, add the new line the end of the file :)

amaltaro added 2 commits May 2, 2022 06:42
more OpenForWriting removals

fix DataBlockGenerator

removed DBS3Reader listOpenFileBlocks method

new line at the end of the module
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 2 tests deleted
    • 7 tests added
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 34 warnings and errors that must be fixed
    • 12 warnings
    • 175 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13113/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor Author

amaltaro commented May 2, 2022

Thanks for the review over the weekend, Stefano and Todor.
Now that we merge it, I can resume my work on #11119

@amaltaro amaltaro merged commit a260091 into dmwm:master May 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove legacy concept of open blocks from DBS
4 participants