WMCore 2.0.3 central services production release

This release brings in many awesome features, such as: some initial support to IAM tokens; full support to workflows with growing input dataset; new policy for Release Validation output data placement, with a distinction of pre-releases from the other samples.
It also includes a few bug fixes required for the DBSServer migration to the Go implementation. Finally, it comes with many enhancements to the code and documentation.

Release date: 7 April 2022.
Changes since release: 2.0.2.

Central services

Software stack

• Bring dbs3-client version to 4.0.8 (Alan Malta Rodrigues) #11138
• Removed pep8; added pycodestyle 2.8.0; updated pylint to 2.13.5 (Alan Malta Rodrigues) cms-sw/cmsdist#7846

Features and/or feature changes

• Add bin and dependencies to wmcore PyPI package (Erik Gough) #11103
• Remove block open logic from DBS3Reader and WorkQueue (Alan Malta Rodrigues) #11123
• Properly handle OpenRunningTimeout in WorkQueue (Alan Malta Rodrigues) #11119 #11135
• Initial script to create IAM token (Valentin Kuznetsov) #11093 #11134
• speed up listDatasetFileDetails API (Valentin Kuznetsov) #11099
• [MSTransferor] Lock the whole input container for growing workflows (Alan Malta Rodrigues) #11141
• [MSOutput] Apply different rule lifetime for RelVal pre-release samples (Alan Malta Rodrigues) #11139
• Add support for IAM token in the underlying pycurl_manager module (Valentin Kuznetsov) #10939
• Add support to OpenRunningTimeout for TaskChain/StepChain (Alan Malta Rodrigues) #11135

Bug Fixes

• Update ReqMgr2 CP thread to properly deal with aborted/force-complete workflows (Alan Malta Rodrigues) #11113

Enhancements

• move ckey/cert functions to Utils.CertTools (Valentin Kuznetsov) #11101
• Remove no longer needed code in the StatusChangeTasks ReqMgr2 thread (Alan Malta Rodrigues) #11095
• Create script to cancel GQEs (Alan Malta Rodrigues) #11113
• Switch from pep8 to pycodestyle (Erik Gough) #11128

WMAgent

Features and/or feature changes

• When ScramArch is empty str/list/None, return any as required_os (Alan Malta Rodrigues) #11083
• If job requires rhel8 container, use slc7 python libraries (Alan Malta Rodrigues) #11077
• Support adding diskless resources into the database (Alan Malta Rodrigues) #11097
• Remove PFNs from a final document we send to WMArchive (Valentin Kuznetsov) #10998
• change fallback for missing system xrootd-client (Dirk Hufnagel) #11117
• Remove support for Unpacking user tarballs. No longer used by CRAB. (khurtado) #11114
• Enforce dataset_lifetime column to be integer and not null (Alan Malta Rodrigues) #11115
• speed up listDatasetFileDetails API (Valentin Kuznetsov) #11099
• Add support for IAM token in the underlying pycurl_manager module (Valentin Kuznetsov) #10939

Bug Fixes

• Assign the correct activity for Tape_Test RSEs (Jhonatan Amado) #11067
• [Tier0] Check CMSSW version before getting sim-datatier map (germanfgv) #11110

Enhancements

• Enhance logic to map ScramArch to OS (Alan Malta Rodrigues) #11088
• [PY2] Remove cPickle support (Erik Gough) #11096
• Update wmagent deploy script to add opportunistic resources one by one (Alan Malta Rodrigues) #11097
• Minor test json template updates (Alan Malta Rodrigues) #10864
• Add T3_US_Lancium. (Todor Ivanov) #11126
• Add buildAllDep option && Fix package verstion estimation for ParseSpecCmsswdist. (Todor Ivanov) #11132

WMAgent patch release: 2.0.2.patch1

Release date: 12 April 2022.
Changes since release: 2.0.2.

In addition to the release notes provided under within the tag 2.0.2, this patch release brings in the following.

WMAgent

Features and/or feature changes

• Default job wrapper to use rhel7 python libraries, otherwise use the same as CMSSW (Alan Malta Rodrigues) #11090

Bug Fixes

• Assign the correct activity for Tape_Test RSEs (Jhonatan Amado) #11090
• When ScramArch is empty str/list/None, return any as required_os (Alan Malta Rodrigues) #11090

Enhancements

• Enhance logic to map ScramArch to OS (Alan Malta Rodrigues) #11090

WMCore 2.0.2 central services and WMAgent production release

This release has support to compressed data in HTTP requests, relying on the automatic
decoding algorithms built with libcurl. For WMAgent, it has a database schema change for
the processing version; and it has full support to EL8 workflows.
Other than that, many enhancements to the WMCore workflow, documentation and overall
improvements have been made.

Release date: 7 April 2022.
Changes since release: 2.0.1.
WMAgent changes since release: 1.5.7

Central services

Software stack

Features and/or feature changes

• Define gzip encoding by default in the headers; properly decompress it (Valentin Kuznetsov) #11041
• Add usage of gzip encoding (Valentin Kuznetsov) #11056
• Use Content-Encoding to automatically decode body in RequestHandler (Alan Malta Rodrigues) #11069 #11070
• [MSUnmerged] Fetch protectedLFNs for T0 (Todor Ivanov) #11050

Bug Fixes

• [MSTransferor] do not bypass Pileup rules in INJECT/REPLICATING state (Alan Malta Rodrigues) #11061

Enhancements

• Document the use of docstrings and best practices in the project (Alan Malta Rodrigues) #11052
• Document Services cache deletion on exit (Alan Malta Rodrigues) #10915

WMAgent

Features and/or feature changes

• Define gzip encoding by default in the headers; properly decompress it (Valentin Kuznetsov) #11041
• Adjust processing version to be integer (Valentin Kuznetsov) #11043 #11059
• [CouchDB] Do not sanitize couch url at the component layer (Alan Malta Rodrigues) #11045
• Add el8 flavor to the list of rhel8 container jobs (Alan Malta Rodrigues) #11060 #11062
• Add usage of gzip encoding (Valentin Kuznetsov) #11056
• Use Content-Encoding to automatically decode body in RequestHandler (Alan Malta Rodrigues) #11069 #11070
• Add parameters to rucio datasets in RucioInjector (Jhonatan Amado) #10989
• Add lifetime to RucioContainers shipping to disk sites (Jhonatan Amado) #10950
• make sure proxy is 2048 bit. Fix #11005 (belforte) #11006
• Explicitly create proxies with 2048 bits (Alan Malta Rodrigues) #11008

Bug Fixes

• Prevent excessive calls for agent config via ReqMgrAux (Erik Gough) #11033
• Fix a few DBS3Reader APIs, likely unused ones (Alan Malta Rodrigues) #10990
• DBS3Upload: check error message from the response body attribute (Alan Malta Rodrigues) #10961

Enhancements

• Better logging for DBS block insertion in DBS3Upload (Alan Malta Rodrigues) #11058
• Download script and insert US opportunistic resources during deployment (Alan Malta Rodrigues) #11047
• Update mocked DBS APIs and data (Alan Malta Rodrigues) #10990
• Enhancements to the GH issues and PRs workflows (Alan Malta Rodrigues) #11021
• Update deploy-wmagent for 1.5.7.patch1; remove deprecated py2 option (Alan Malta Rodrigues) #10966
• Update error message for jobs killed from site state transition (Alan Malta Rodrigues) #10958

WMCore 2.0.1 central services production release

This release brings in many new features and feature changes to the microservices. It also includes a few bug fixes required for the DBSServer migration to the Go implementation. Lastly, usual code and documentation enhancements have been provided.

Release date: 10 March 2022.
Changes since release: 2.0.0.

Central services

Software stack

• Bump dbs3-client to 4.0.7 (Alan Malta Rodrigues) cms-sw/cmsdist#7624
• Updated pymongo library to 4.0.1 (Todor Ivanov) cms-sw/cmsdist#7654

Features and/or feature changes

• Update requirements.txt and PyPI build scripts for py3 stack (Erik Gough) #10976
• Add parameters to rucio datasets (Jhonatan Amado) #10989
• [MSTransferor] Send alerts to prometheus when a workflow fails to be processed. Rename notifyLargeData method. Fixes #10215 (Kenyi Hurtado) #10959
• [MSUnmerged] Add rucioConMonErrors counter. (Todor Ivanov) #10993
• [MSUnmerged] Simplify counters logic && Substitute rucioConMonErrors with rucioConMonStatus. (Todor Ivanov) #10993
• [MSUnmerged] Swap NotMasterError with NotPrimaryError. (Todor Ivanov) #11016
• [MSUnmerged] Add rse=ALL_DOCS to the MSUnmerged info REST API. (Todor Ivanov) #11025
• [MSOutput] Make MSOutput Disk expression configurable (Alan Malta Rodrigues) #11018
• [MSOutput] MSOutput: read RelVal output data policy from configuration (Alan Malta Rodrigues) #11024

Bug Fixes

• Fix a few DBS3Reader APIs, likely unused ones (Alan Malta Rodrigues) #10990
• [MSUnmerged] Remove files from mongoProjection (Todor Ivanov) #10987
• [MSUnmerged] Allow MongoDB to be instantiated without mongomock library (Alan Malta Rodrigues) #11023
• [MSUnmerged] Reorder RSE's status and record age checks from RucioConMon. (Todor Ivanov) #11029
• [MSTransferor] Fix MSTransferor typo calling getName method (Alan Malta Rodrigues) #11031
• [MSTransferor] MSTransferor: fix positional and kwargs passed to sendAlert API (Alan Malta Rodrigues) #11032

Enhancements

• Update mocked DBS APIs and data (Alan Malta Rodrigues) #10990
• Remove leftover requirements_py3.txt; update contribution doc (Alan Malta Rodrigues) #10997
• fix uninitialized templates variable (Dirk Hufnagel) #11004
• Add mockMongoDB to MSUnmerged and MongoDB Class init options (Todor Ivanov) #11007
• Enhancements to the GH issues and PRs workflows (Alan Malta Rodrigues) #11021

WMAgent

Features and/or feature changes

• Add parameters to rucio datasets in RucioInjector (Jhonatan Amado) #10989
• Add lifetime to RucioContainers shipping to disk sites (Jhonatan Amado) #10950
• make sure proxy is 2048 bit. Fix #11005 (belforte) #11006
• Explicitly create proxies with 2048 bits (Alan Malta Rodrigues) #11008

Bug Fixes

• Fix a few DBS3Reader APIs, likely unused ones (Alan Malta Rodrigues) #10990

Enhancements

• Update mocked DBS APIs and data (Alan Malta Rodrigues) #10990
• Enhancements to the GH issues and PRs workflows (Alan Malta Rodrigues) #11021

WMCore 2.0.0 central services production release

This is our first WMCore release which no longer supports Python2 stack, and we will keep gradually removing python2 from our baseline code (including the future library).
This release has a major bug fix for MSUnmerged as well as other bug-fixes and enhancements to the overall system.

Release date: 8 February 2022.
Changes since release: 1.5.7.

Central services

Software stack

• Update Python3 pycurl library from 7.43.0.3 to 7.43.0.6 (Alan Malta Rodrigues) cms-sw/cmsdist#7600

Features and/or feature changes

• Aggregate DBS results from Go server (Valentin Kuznetsov) #10963

Bug Fixes

• Fix MSUnmerged logic and MSUnmergedRSE format to account for deletion errors. (Todor Ivanov) #10687

Enhancements

• Check that sdir exists on a file system (Valentin Kuznetsov) #10938
• Added MSUnmerged to the WM schematic (Alan Malta Rodrigues) #10974
• Update diagrams and contribution documentation for end of support for Py2 (Alan Malta Rodrigues) #10968

WMAgent

Features and/or feature changes

Bug Fixes

• DBS3Upload: check error message from the response body attribute (Alan Malta Rodrigues) #10961

Enhancements

• Update deploy-wmagent for 1.5.7.patch1; remove deprecated py2 option (Alan Malta Rodrigues) #10966
• Update error message for jobs killed from site state transition (Alan Malta Rodrigues) #10958

Python3 related changes

WMCore 1.5.7.patch1 WMAgent production release

This is the last WMAgent stable release supporting both Python2 and Python3. This release brings in some important new features and bug fixes, in addition to the continuous minor enhancements. It also comes with a few very important software stack changes.

Release date: 27 January 2022.
Changes since release: 1.5.4.

WMAgent

Software stack

• Upgrade MariaDB spec from 10.1.21 to 10.6.5 (Alan Malta Rodrigues) #7205
• Update pylint version required from 2.6.0 to 2.7.0 (Alan Malta Rodrigues) #10913
• Bump dbs3-client to 4.0.5 (Alan Malta Rodrigues) #10931
• Bump dbs3-pycurl to 3.17.7 (Alan Malta Rodrigues) #10931
• Update CMSCouchapp version for python3 stack (Alan Malta Rodrigues) #10910
• Update python3 pip to 20.3.3 and setuptools to 51.3.3 (Alan Malta Rodrigues) #7418

Features and/or feature changes

• Remove trailing slash from DBS URL (Alan Malta Rodrigues) #10930
• Add two new CMSSW exit code: 8034 and 8501 (Stefano Belforte) #10850
• Remove PhEDEx library and any other use of PhEDEx (Alan Malta Rodrigues) #10856
• Stop using password for MariaDB localhost users (Alan Malta Rodrigues) #10932
• Use siteLocalConfig to get LogCollect stage out params (germanfgv) #10905
• Add --deep option to getReplicaInfoForBlocks. Add slicing to deleteBlocks (germanfgv) #10902
• Set request_GPUs classad according to RequiresGPU (Alan Malta Rodrigues) #10849
• Support different architectures during job submission and runtime (Alan Malta Rodrigues) #10853

Bug Fixes

• DBS3Upload: check error message from the response body attribute #10961
• Properly update workqueue elements from JobUpdater (Alan Malta Rodrigues) #10944
• Skip ErrorHandler cycle if agent configuration fails to be retrieved - take2 (Alan Malta Rodrigues) #10852
• Remove leading ../../ from logArchive tarballs (Alan Malta Rodrigues) #10868
• Check whether WMStep object has gpu attribute (Alan Malta Rodrigues) #10866
• Use the right COMP scramArch for PowerPC workflows (Alan Malta Rodrigues) #10871
• CredentialProxy - convert to bytes the input of hashlib.sha1() (Dario Mapelli) #10918
• REST/Server.py - encode to bytes input of hashlib.sha1 (Dario Mapelli) #10917
• Fix path for Py2/Py3 WMAgent logs (Alan Malta Rodrigues) #10897

Enhancements

• Update error message for jobs killed from site state transition #10958
• Create the MariaDB wmcore unittest database with the current unix user (Alan Malta Rodrigues) #10934
• fix DbsUrl in test modules and templates (Alan Malta Rodrigues) #10930
• Fix python version in subprocess call for inject-test-wfs (Alan Malta Rodrigues) #10847
• Update Rucio Int url (Alan Malta Rodrigues) #10855
• Update job priority only for processing and production task types (Kenyi Hurtado) #10812
• Updating workflow json templates; adding working PY3 templates (Alan Malta Rodrigues) #10761
• Update wmagent-deploy script to deploy Py3 WMAgent by default; update to cycle 1.5.4 (Alan Malta Rodrigues) #10859
• Update campaigns used in our test setup (Alan Malta Rodrigues) #10859

WMCore 1.5.7 central services and WMAgent production release

This release brings in the latest dbs3-client version, compatible with both the Python and Golang server implementations.
The software stack also had a major MariaDB upgrade (from 10.1.21 to 10.6.5).
It also has a few bug fixes and improvements for central services and WMAgent.

Release date: 18 January 2021.
Changes since release: 1.5.6.

Central services

Software stack

• Upgrade MariaDB spec from 10.1.21 to 10.6.5 (Alan Malta Rodrigues) #7205
• Update pylint version required from 2.6.0 to 2.7.0 (Alan Malta Rodrigues) #10913
• Bump dbs3-client to 4.0.5 (Alan Malta Rodrigues) #10931
• Bump dbs3-pycurl to 3.17.7 (Alan Malta Rodrigues) #10931

Features and/or feature changes

• Remove trailing slash from DBS URL (Alan Malta Rodrigues) #10930

Bug Fixes

• [MSTransferor] Fix logic which drops input blocks to be transferred (Alan Malta Rodrigues) #10928
• [MSUnmerged] REST APIs to fetch RSE data from MongoDB. (Todor Ivanov) #10923

Enhancements

• fix DbsUrl in test modules and templates (Alan Malta Rodrigues) #10930

WMAgent

Features and/or feature changes

• Remove trailing slash from DBS URL (Alan Malta Rodrigues) #10930
• Stop using password for MariaDB localhost users (Alan Malta Rodrigues) #10932
• Create the MariaDB wmcore unittest database with the current unix user (Alan Malta Rodrigues) #10934

Bug Fixes

• Properly update workqueue elements from JobUpdater (Alan Malta Rodrigues) #10944

Enhancements

• Update job priority only for processing and production task types, rather than every single subtask name, to reduce condor_qedit load. (Kenyi Hurtado) #10812

Python3 related changes

• CredentialProxy - convert to bytes the input of hashlib.sha1() (Dario Mapelli) #10918
• REST/Server.py - encode to bytes input of hashlib.sha1 (Dario Mapelli) #10917

WMCore 1.5.6 central services production release

First version of MSUnmerged with a MongoDB database backend.

Release date: 10 November 2021.
Changes since release: 1.5.5.

Central services

Software stack

• Update CMSCouchapp version for python3 stack (Alan Malta Rodrigues) #10910

Features and/or feature changes

• [MSUnmerged] Initial integration of MongoDB. (Todor Ivanov) #10895
• [MSUnmerged] Add replicaset opotion to the MongoDB wrapper. (Todor Ivanov) #10908
• [MSUnmerged] Adapt MSUnmerged to sync RSE data with MongoDB properly (Todor Ivanov) #10906

Bug Fixes

Enhancements

• Update request state diagram with archival done by MS (Alan Malta Rodrigues) #10899

WMAgent

Features and/or feature changes

• Use siteLocalConfig to get LogCollect stage out params (germanfgv) #10905
• Add --deep option to getReplicaInfoForBlocks. Add slicing to deleteBlocks (germanfgv) #10902

Bug Fixes

Enhancements

Python3 related changes

• Fix path for Py2/Py3 WMAgent logs (Alan Malta Rodrigues) #10897

WMCore 1.5.5 central services production release

This release no longer ships any of the PhEDEx libraries. The running-open and running-close statuses have been slightly redesigned.
Last but not least, some important bug fixes and enhancements have been made.

Release date: 10 November 2021.
Changes since release: 1.5.4.

Central services

Software stack

• Update dbs3-client to 3.17.6 and dbs3-pycurl 3.17.4
• Update python3 pip to 20.3.3 and setuptools to 51.3.3

Features and/or feature changes

• add two new CMSSW exit code: 8034 and 8501 (Stefano Belforte) #10850
• Remove PhEDEx library and any other use of PhEDEx (Alan Malta Rodrigues) #10856
• [MSTransferor] Remove unused Unified/MSTransferor code (Alan Malta Rodrigues) #10876
• Add validation to the SubRequestType parameter (Alan Malta Rodrigues) #10890
• Refactored conditions for running-open/running-closed workflow statuses (Alan Malta Rodrigues) #10693
• Temporary fix for the StatusChangeTasks ReqMgr2 thread (Alan Malta Rodrigues) #10693

Bug Fixes

• [MSUnmerged] Remove check of RSE dump root_failed attr in MSUnmerged (Alan Malta Rodrigues) #10894

Enhancements

• Fix python version in subprocess call for inject-test-wfs (Alan Malta Rodrigues) #10847

WMAgent

Features and/or feature changes

• Set request_GPUs classad according to RequiresGPU (Alan Malta Rodrigues) #10849
• Update Rucio Int url (Alan Malta Rodrigues) #10855
• Support different architectures during job submission and runtime (Alan Malta Rodrigues) #10853

Bug Fixes

• Skip ErrorHandler cycle if agent configuration fails to be retrieved - take2 (Alan Malta Rodrigues) #10852
• Remove leading ../../ from logArchive tarballs (Alan Malta Rodrigues) #10868
• Check whether WMStep object has gpu attribute (Alan Malta Rodrigues) #10866
• Use the right COMP scramArch for PowerPC workflows (Alan Malta Rodrigues) #10871

Enhancements

• Updating workflow json templates; adding working PY3 templates (Alan Malta Rodrigues) #10761
• Update wmagent-deploy script to deploy Py3 WMAgent by default; update to cycle 1.5.4 (Alan Malta Rodrigues) #10859
• update campaigns used in our test setup (Alan Malta Rodrigues) #10859

Python3 related changes

WMCore 1.5.4.patch2 WMAgent production release

This is the latest stable WMAgent branch, which already includes to patch releases on top of it.
The highlight of this release is the support to GPU and PowerPC workflows/jobs. In terms of the software stack,
it brings in a new oracle client version 19, and HTCondor python bindings version 9.2.0.

Release date: 12 October 2021.
Changes since release: 1.5.2.

WMAgent

Software stack

• Oracle client version 19.11
• Update Python3 requirements list with Sphinx and Jinja2 #10794
• Update py3-dbs3-client/dbs3-pycurl to 3.16.2001 cms-sw/cmsdist#7384
• New HTCondor specs from PyPi; update version to 9.2.0 cms-sw/cmsdist#7383

Features and/or feature changes

• Support different architectures during job submission and runtime (Alan Malta Rodrigues) #10863
• Update Rucio Int url (Alan Malta Rodrigues) #10855
• Add two new CMSSW exit code: 8034 and 8501 (Stefano Belforte) #10850
• Set request_GPUs classad according to RequiresGPU (Alan Malta Rodrigues) #10849
• Propagate GPU requirements to job creation and submission (Alan Malta Rodrigues) #10811
• Provide request type via job classad (Alan Malta Rodrigues) #10821
• Permanently stop staging out log tarball to CASTOR (Alan Malta Rodrigues) #10712
• CouchDB 3: Add new Error HTTP Status Codes to CMSCouch.py (phenomax) #10785

Bug Fixes

• JobAccountant workaround for StepChain jobs with duplicate files - wmagent branch (Alan Malta Rodrigues) #10863
• [ReqMgr2/Tier0] PhysicsSkim issue (Jhonatan Amado) #10809
• Skip ErrorHandler cycle if agent configuration fails to be retrieved (Alan Malta Rodrigues) #10817 #10852
• Fix missing parenthesis during PSet tweak (Kenyi Hurtado) #10771

Enhancements

• Updating workflow json templates; adding working PY3 templates (Alan Malta Rodrigues) #10761
• Fix python version in subprocess call for inject-test-wfs (Alan Malta Rodrigues) #10847
• Define pylint message template in the pylintrc file (Alan Malta Rodrigues) #10776

Python3 related changes

• [py2py3] modernize WMCore_t/REST_t/Api_t.py (Dario Mapelli) #10757
• #10349 - JobSubmitter_t and JobSubmitterCaching_t - test (Dario Mapelli) #10608
• #10371 check - fix JobCreator_t.py - test (Dario Mapelli) #10724
• Modernization of leftover files - src (Dario Mapelli) #10768
• RESTFormatter.to_string returns 'native str' (Dario Mapelli) #10753