Improve py2 compatibility of WMException init #9868

mapellidario · 2020-08-12T16:58:34Z

Fixes #9862

Status

ready

Description

Small change to WMException.__init__ in order to improve its handling of unicode error messages in python2

Is it backward compatible (if not, which system it affects?)

yes

External dependencies / deployment changes

This makes WMException depend on python future

cmsdmwmbot · 2020-08-12T17:21:35Z

Jenkins results:

Unit tests: failed
- 2 new failures
- 1 tests no longer failing
- 2 changes in unstable tests
Pylint check: succeeded
- 2 warnings
- 4 comments to review
Pycodestyle check: succeeded
- 2 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/10302/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2020-08-12T17:27:45Z

Jenkins results:

Unit tests: succeeded
- 2 tests no longer failing
- 1 changes in unstable tests
Pylint check: succeeded
- 2 warnings
- 4 comments to review
Pycodestyle check: succeeded
- 2 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/10303/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2020-08-13T11:51:15Z

Jenkins results:

Unit tests: failed
- 3 new failures
- 1 tests added
- 5 changes in unstable tests
Pylint check: failed
- 6 warnings and errors that must be fixed
- 2 warnings
- 9 comments to review
Pycodestyle check: succeeded
- 7 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/10309/artifact/artifacts/PullRequestReport.html

amaltaro · 2020-08-13T12:29:24Z

Dario, can you please apply the suggested "python3 idioms"?

mapellidario · 2020-08-13T14:53:56Z

I did not do it because it can cause trouble such as in #9852, but I agree that in this case it should cause any problems using isinstance instead of checking the typed directly

amaltaro · 2020-08-20T05:41:13Z

@mapellidario could you please fix the conflicts here and fire another review request once you're done with your changes? Thanks

cmsdmwmbot · 2020-08-20T09:01:29Z

Jenkins results:

Unit tests: failed
- 1 new failures
- 1 tests added
- 1 changes in unstable tests
Pylint check: failed
- 7 warnings and errors that must be fixed
- 2 warnings
- 14 comments to review
Pycodestyle check: succeeded
- 7 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/10338/artifact/artifacts/PullRequestReport.html

This solves the issue dmwm#9862, The rationale for this change is described in the issue comments

New unit test in WMException_t.py to check if WMException handles correctly a message string that contains unicode characters

cmsdmwmbot · 2020-08-20T11:32:31Z

Jenkins results:

Unit tests: succeeded
- 1 tests added
Pylint check: failed
- 6 warnings and errors that must be fixed
- 2 warnings
- 13 comments to review
Pycodestyle check: succeeded
- 7 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/10340/artifact/artifacts/PullRequestReport.html

amaltaro

Thanks Dario!

amaltaro · 2020-09-02T07:02:36Z

I was looking at the backfill agent (vocms0285), and I've spotted a few jobs failing with errors like (wmagentJob.log):

2020-09-01 23:18:55,930:INFO:PerformanceMonitor:PSS: 133864; RSS: 133860; PCPU: 0.0; PMEM: 0.0
2020-09-01 23:23:44,021:CRITICAL:CMSSW:Error running cmsRun
{'arguments': ['/bin/bash', '/srv/job/WMTaskSpace/cmsRun1/cmsRun1-main.sh', '', u'slc7_amd64_gcc700', 'scramv1', 'CMSSW', 'CMSSW_10_6_9', 'FrameworkJobReport.xml', 'cmsRun', 'PSet.py', '', '', '']}
Linux Return code: 135

2020-09-01 23:23:44,021:CRITICAL:CMSSW:Error message: None
2020-09-01 23:23:44,352:ERROR:ExecuteMaster:Exception occured when executing step
2020-09-01 23:23:44,352:ERROR:ExecuteMaster:Exception is unknown encoding: utf-8
2020-09-01 23:23:44,353:ERROR:ExecuteMaster:Traceback:
2020-09-01 23:23:44,354:ERROR:ExecuteMaster:Traceback (most recent call last):
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/ExecuteMaster.py", line 139, in doExecution
    executionObject.execute()
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Executors/CMSSW.py", line 284, in execute
    raise WMExecutionFailure(returnCode, "CmsRunFailure", msg)
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/WMExecutionFailure.py", line 18, in __init__
    WMException.__init__(self, detail, code)
  File "/srv/job/WMCore.zip/WMCore/WMException.py", line 33, in __init__
    message = message.decode('utf-8', 'ignore')
LookupError: unknown encoding: utf-8

2020-09-01 23:23:44,354:CRITICAL:CMSSW:CMSDefaultHandler Diagnostic Handler invoked
2020-09-01 23:23:44,356:ERROR:ExecuteMaster:Encountered error while running ExecuteMaster:
unknown encoding: utf-8
Traceback (most recent call last):
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/ExecuteMaster.py", line 73, in __call__
    result = self.doExecution(executor, step, wmbsJob)
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/ExecuteMaster.py", line 149, in doExecution
    executor.diagnostic(99109, executor, ExceptionInstance=ex)
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Diagnostic.py", line 87, in __call__
    handler(errCode, executor, **args)
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Diagnostics/CMSSW.py", line 102, in __call__
    self.parse(executor, jobRepXml)
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Diagnostic.py", line 42, in parse
    executorInstance.report.parse(jobRepXml, executorInstance.stepName)
  File "/srv/job/WMCore.zip/WMCore/FwkJobReport/Report.py", line 156, in parse
    raise FwkJobReportException(msg)
  File "/srv/job/WMCore.zip/WMCore/WMException.py", line 33, in __init__
    message = message.decode('utf-8', 'ignore')
LookupError: unknown encoding: utf-8


2020-09-01 23:23:44,357:INFO:Watchdog:MonitorThread: JobEnded

@mapellidario can I ask you to look into this PR and try to backport it to the branch 1.3.6_wmagent? That branch does not have any of these code modernization, so it might not be very straight forward...
Once you have it ready, please make a PR against the 1.3.6_wmagent branch and we can start testing your changes in the backfill agent itself (I'll take care of the testing). Thanks

amaltaro · 2020-09-02T07:03:36Z

Oh, if you consider it to be a new issue, then go ahead and file a new GH issue and start working on a bug fix please.

mapellidario force-pushed the issue9862 branch from b98ee30 to 237e497 Compare August 12, 2020 17:03

amaltaro added the PR: Do not merge yet label Aug 13, 2020

mapellidario force-pushed the issue9862 branch from b4957dd to a57add6 Compare August 20, 2020 08:19

mapellidario added 2 commits August 20, 2020 13:06

Improve py2 compatibility of WMException __init__

fd6449c

This solves the issue dmwm#9862, The rationale for this change is described in the issue comments

Improve py2 compatibility of WMException __init__ - unit test

ec818ee

New unit test in WMException_t.py to check if WMException handles correctly a message string that contains unicode characters

mapellidario force-pushed the issue9862 branch from a57add6 to ec818ee Compare August 20, 2020 11:07

mapellidario requested a review from amaltaro August 20, 2020 11:44

amaltaro approved these changes Aug 21, 2020

View reviewed changes

amaltaro merged commit 36ce39b into dmwm:master Aug 21, 2020

amaltaro assigned mapellidario Sep 2, 2020

This was referenced Sep 2, 2020

WMAgent - LookupError: unknown encoding: utf-8 - branch 1.3.6_wmagent #9885

Closed

Make WMException handle unicode messages in 1.3.6_wmagent #9886

Closed

This was referenced Dec 12, 2020

[py2py3] Fix potential bug in WMException #10173

Closed

[py2py3] Fix potential bug in WMException #10174

Merged

mapellidario deleted the issue9862 branch September 18, 2024 14:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve py2 compatibility of WMException init #9868

Improve py2 compatibility of WMException init #9868

mapellidario commented Aug 12, 2020

cmsdmwmbot commented Aug 12, 2020

cmsdmwmbot commented Aug 12, 2020

cmsdmwmbot commented Aug 13, 2020

amaltaro commented Aug 13, 2020

mapellidario commented Aug 13, 2020

amaltaro commented Aug 20, 2020

cmsdmwmbot commented Aug 20, 2020

cmsdmwmbot commented Aug 20, 2020

amaltaro left a comment

amaltaro commented Sep 2, 2020

amaltaro commented Sep 2, 2020

Improve py2 compatibility of WMException __init__ #9868

Improve py2 compatibility of WMException __init__ #9868

Conversation

mapellidario commented Aug 12, 2020

Status

Description

Is it backward compatible (if not, which system it affects?)

External dependencies / deployment changes

cmsdmwmbot commented Aug 12, 2020

cmsdmwmbot commented Aug 12, 2020

cmsdmwmbot commented Aug 13, 2020

amaltaro commented Aug 13, 2020

mapellidario commented Aug 13, 2020

amaltaro commented Aug 20, 2020

cmsdmwmbot commented Aug 20, 2020

cmsdmwmbot commented Aug 20, 2020

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro commented Sep 2, 2020

amaltaro commented Sep 2, 2020

Improve py2 compatibility of WMException init #9868

Improve py2 compatibility of WMException init #9868