TypeError: cannot pickle '_io.TextIOWrapper' object #961

lorddaedra · 2020-02-14T20:42:18Z

gsutil -m -h "Cache-Control: public, max-age=31536000" cp -r test/** gs://some-bucket
Traceback (most recent call last):
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
    gsutil.RunMain()
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gsutil.py", line 124, in RunMain
    sys.exit(gslib.__main__.main())
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 424, in main
    return _RunNamedCommandAndHandleExceptions(
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 762, in _RunNamedCommandAndHandleExceptions
    _HandleUnknownFailure(e)
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 620, in _RunNamedCommandAndHandleExceptions
    return command_runner.RunNamedCommand(command_name,
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 411, in RunNamedCommand
    return_code = command_inst.RunCommand()
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1201, in RunCommand
    self.Apply(_CopyFuncWrapper,
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1499, in Apply
    self._ParallelApply(
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1719, in _ParallelApply
    self._CreateNewConsumerPool(process_count, thread_count,
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1384, in _CreateNewConsumerPool
    p.start()
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object

gsutil version: 4.47

The text was updated successfully, but these errors were encountered:

crwilcox · 2020-02-21T19:08:44Z

I ran into the same issue. If you use another interpreter (python 3.7 for instance) all is well. This is a problem specifically with Python 3.8

Google Cloud SDK 281.0.0
beta 2019.05.17
bq 2.0.53
cloud-firestore-emulator 1.10.4
core 2020.02.14
gsutil 4.47

lorddaedra · 2020-03-05T07:34:38Z

Bug is still there

Google Cloud SDK 283.0.0
alpha 2019.05.17
app-engine-python 1.9.88
beta 2019.05.17
bq 2.0.54
cloud-datastore-emulator 2.1.0
core 2020.02.28
gsutil 4.48

calebmoss · 2020-03-25T14:07:56Z

Still exists now, only on multiprocessing flag, runs fine without -m :

Google Cloud SDK 286.0.0
bq 2.0.55
core 2020.03.24
gsutil 4.48

calebmoss · 2020-03-26T19:49:25Z

Tracking this down, this error comes from a change in Python 3.8 in the multiprocessing library:

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

Spawn is being run for those using MacOs and Python 3.8+ by default since nothing is explicitly set either through get_context or set_start_method.

caizixian · 2020-07-23T11:06:34Z

The issue still presents with Cloud SDK 302.0.0 (gsutil 4.52), on macOS 10.15.6 with Python 3.8.5 installed from homebrew

caizixian · 2020-07-23T11:18:48Z

One workaround is to use the Python 3 interpreter shipped with macOS /usr/bin/python3 by setting the Cloud SDK interpreter path https://cloud.google.com/sdk/gcloud/reference/topic/startup

aleb · 2020-07-24T14:27:09Z

gsutil does not work with python 3.8, force it to use python 3.7 with something like

export CLOUDSDK_PYTHON=/usr/bin/python3     # on mac
export CLOUDSDK_PYTHON=/usr/bin/python3.7   # on linux

dbrookeUAB · 2020-07-29T11:14:02Z

@aleb I'm not sure if this is specific to Mac Mojave, but the path for python3 for me was /usr/local/bin/python3. I couldn't get it to work with python3 anyways, but forcing it to use 2.7 worked like a charm.

export CLOUDSDK_PYTHON=/usr/local/bin/python3      # did not work
export CLOUDSDK_PYTHON=/usr/bin/python2.7          # worked

From the link @caizixian provided,

Python 3 is preferred over Python 2. Note that gcloud requires Python version 2.7.x or 3.5 and up. Other Python tools shipped in the Cloud SDK do not support Python 3 and require Python 2.7.x,

dinvlad · 2020-09-01T06:08:30Z

Another workaround on macOS is to

brew install [email protected]
export CLOUDSDK_PYTHON=/usr/local/opt/[email protected]/bin/python3

pcantalupo · 2020-09-03T13:36:47Z

@dinvlad That worked for me! Thank you so much

rogerthomas84 · 2020-09-21T08:05:11Z

@dinvlad Thank you! Works perfectly!

wayjake · 2020-09-21T17:43:24Z

With such a strange "pickle" error, I didn't expect to find my resolution so quickly. Thank you, @dinvlad!!

Catherine19950122 · 2020-09-30T14:33:12Z

export CLOUDSDK_PYTHON=/usr/bin/python2.7 will work ! export CLOUDSDK_PYTHON=/usr/bin/python3 or export CLOUDSDK_PYTHON=path/for/python3.7 will solve the current issue but will run into module 'sys' has no attribute 'maxint' error.

max-sixty · 2020-10-21T18:50:10Z

While I recognize comments like "Is tHiS fiXEd??" are not helpful — would it be possible for someone on the Google side to acknowledge this is a bug in gsutil and plan to resolve?

Currently, IIUC, gsutil breaks on python 3.8 — a version released a year ago, and the default brew version. Workarounds like installing another version of python are not small adjustments, and difficult for less technical colleagues. There are 49 👍s on the issue.

dilipped · 2020-10-21T19:57:31Z

Sorry for the delay. We are aware of this bug and we are working on releasing this workaround soon #1107

dilipped · 2020-10-21T21:46:28Z

Another workaround would be to disable multiprocessing altogether when using Python 3.8. This can be done either by setting the parallel_process_count=1 in the boto config file or by passing the option from the command line like this

gsutil -o "GSUtil:parallel_process_count=1" -m cp .....

This will be relatively slow as it will be using a single process, however, multithreading will be still ON.

max-sixty · 2020-10-23T21:22:19Z

That's an excellent workaround, thanks @dilipped !

tartavull · 2020-11-09T12:42:53Z

updating gsutil solved the issue with python3.8

cuonghuynh · 2021-01-25T09:23:44Z

gsutil does not work with python 3.8, force it to use python 3.7 with something like
export CLOUDSDK_PYTHON=/usr/bin/python3     # on mac
export CLOUDSDK_PYTHON=/usr/bin/python3.7   # on linux

It works fine to me

chollinger93 mentioned this issue Jun 25, 2020

gsutil breaks after updating to SDK 298 on OS X #1052

Closed

Nic-Ma mentioned this issue Jun 28, 2020

629 Update progress_bar and replace tqdm in notebook Project-MONAI/MONAI#633

Merged

4 tasks

christopherowen mentioned this issue Jul 15, 2020

TypeError: cannot pickle '_thread.lock' object aboul3la/Sublist3r#236

Closed

dilipped mentioned this issue Aug 27, 2020

macOS: crcmod not installed? (gsutil rsync) #1073

Open

This was referenced Sep 9, 2020

gsutil flag '-m' exception with Python 3.8 #1085

Open

'ForkAwareLocal' object has no attribute 'connection' for multithreaded cp #1100

Open

thomasmaclean mentioned this issue Sep 23, 2020

Explicitly set multiprocessing start method to 'fork' #1107

Merged

ibressler mentioned this issue Oct 23, 2020

[doit] missing task for building the website with BAMresearch/Reproducible-Science#12

Open

amotl mentioned this issue Mar 16, 2021

Running the tests raise "TypeError: cannot pickle '_io.TextIOWrapper' object" crate/crate-python#398

Closed

philoel mentioned this issue May 26, 2021

"TypeError: cannot pickle '_io.TextIOWrapper' object" issue blocking final step; python version issue? davidemms/OrthoFinder#570

Closed

kidmose mentioned this issue Aug 18, 2021

cannot pickle '_io.TextIOWrapper' object on python 3.9 saeeddhqan/Maryam#246

Closed

mikeurbanski1 mentioned this issue Sep 28, 2021

use fork for multiprocessing bridgecrewio/checkov#1669

Merged

davidemms mentioned this issue Feb 9, 2022

OrthoFinder generates BLAST files with NUL characters and is unhappy davidemms/OrthoFinder#663

Closed

WolfgangFahl mentioned this issue Aug 30, 2022

shutdown for unit tests using psutil justpy-org/justpy#474

Closed

haessar mentioned this issue Jan 20, 2023

Example run failure haessar/peaks2utr#2

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TypeError: cannot pickle '_io.TextIOWrapper' object #961

TypeError: cannot pickle '_io.TextIOWrapper' object #961

lorddaedra commented Feb 14, 2020

crwilcox commented Feb 21, 2020

lorddaedra commented Mar 5, 2020

calebmoss commented Mar 25, 2020

calebmoss commented Mar 26, 2020

caizixian commented Jul 23, 2020

caizixian commented Jul 23, 2020

aleb commented Jul 24, 2020

dbrookeUAB commented Jul 29, 2020 •

edited

Loading

dinvlad commented Sep 1, 2020

pcantalupo commented Sep 3, 2020

rogerthomas84 commented Sep 21, 2020

wayjake commented Sep 21, 2020

Catherine19950122 commented Sep 30, 2020

max-sixty commented Oct 21, 2020

dilipped commented Oct 21, 2020

dilipped commented Oct 21, 2020 •

edited

Loading

max-sixty commented Oct 23, 2020

tartavull commented Nov 9, 2020

cuonghuynh commented Jan 25, 2021

TypeError: cannot pickle '_io.TextIOWrapper' object #961

TypeError: cannot pickle '_io.TextIOWrapper' object #961

Comments

lorddaedra commented Feb 14, 2020

crwilcox commented Feb 21, 2020

lorddaedra commented Mar 5, 2020

calebmoss commented Mar 25, 2020

calebmoss commented Mar 26, 2020

caizixian commented Jul 23, 2020

caizixian commented Jul 23, 2020

aleb commented Jul 24, 2020

dbrookeUAB commented Jul 29, 2020 • edited Loading

dinvlad commented Sep 1, 2020

pcantalupo commented Sep 3, 2020

rogerthomas84 commented Sep 21, 2020

wayjake commented Sep 21, 2020

Catherine19950122 commented Sep 30, 2020

max-sixty commented Oct 21, 2020

dilipped commented Oct 21, 2020

dilipped commented Oct 21, 2020 • edited Loading

max-sixty commented Oct 23, 2020

tartavull commented Nov 9, 2020

cuonghuynh commented Jan 25, 2021

dbrookeUAB commented Jul 29, 2020 •

edited

Loading

dilipped commented Oct 21, 2020 •

edited

Loading