Preprocessing crashes: A process in the process pool was terminated abruptly while the future was running or pending.

[schmidt12335]

Hi. Im encountering this error while running the preprocessing. Im running the latest rabies version in docker on windows 10. It seems, no crash file is generated. It worked with the example dataset. I also had this issue with another atlas but somehow it was resolved after removing STC and commonspace registration. This atlas was missing CSF and WM masks, so i am now doing it with the Sigma Whistar atlas.

I tried several things, including allocating more resources and trying different rabies and atlas versions. I saw that MultiProc may cause such issues but it was not further specified. I did try using the linear processing but it took more than 7 hours, after which i terminated the process manually.

My specs are:
Intel i7 14700K
32Gb RAM
Geforce RTX 4060Ti

So i dont see why this would be a resource issue.

Any help is appreciated and i am happy to provide any files / infos that might be helpfull.

Thanks in advance.

RABIES call:

PS C:\Users\Anonym> docker run -it --rm --platform linux/amd64 `

-v "D:/rabies/new_test/fmri_data:/input_BIDS:ro" -v "D:/rabies/new_test/prepro_out:/preprocess_outputs/"
-v "D:/rabies/new_test/atlas/sigma_1_2:/atlas/" `
ghcr.io/cobralab/rabies:latest -p MultiProc preprocess /input_BIDS/ /preprocess_outputs/ --anat_template /atlas/SIGMA_InVivo_Brain_Template.nii --brain_mask /atlas/SIGMA_InVivo_Brain_Mask.nii --WM_mask /atlas/SIGMA_InVivo_WM_mask_binary.nii --CSF_mask /atlas/SIGMA_InVivo_CSF_mask_binary.nii --vascular_mask /atlas/SIGMA_InVivo_CSF_mask_binary.nii --labels /atlas/SIGMA_InVivo_Anatomical_Brain_Atlas.nii

log file:

241108-11:12:30,431 nipype.workflow INFO:
Running RABIES - version: 0.5.1
241108-11:12:30,433 nipype.workflow INFO:
CLI INPUTS:
-> rabies_stage = preprocess
-> inclusion_ids = ['all']
-> exclusion_ids = ['none']
-> plugin = MultiProc
-> local_threads = 28
-> scale_min_memory = 1.0
-> min_proc = 1
-> figure_format = png
-> verbose = 1
-> force = False
-> bids_dir = /input_BIDS
-> output_dir = /preprocess_outputs
-> bids_filter = {'func': {'suffix': ['bold', 'cbv']}, 'anat': {'suffix': ['T1w', 'T2w']}}
-> bold_only = False
-> anat_autobox = False
-> bold_autobox = False
-> oblique2card = none
-> apply_despiking = False
-> HMC_option = optim
-> isotropic_HMC = False
-> voxelwise_motion = False
-> apply_slice_mc = False
-> detect_dummy = False
-> data_type = float32
-> anat_inho_cor = {'method': 'SyN', 'otsu_thresh': '2', 'multiotsu': False}
-> anat_robust_inho_cor = {'apply': False, 'masking': False, 'brain_extraction': False, 'template_registration': 'SyN'}
-> bold_inho_cor = {'method': 'Rigid', 'otsu_thresh': '2', 'multiotsu': False}
-> bold_robust_inho_cor = {'apply': False, 'masking': False, 'brain_extraction': False, 'template_registration': 'SyN'}
-> commonspace_reg = {'masking': False, 'brain_extraction': False, 'template_registration': 'SyN', 'fast_commonspace': False}
-> inherit_unbiased_template = none
-> bold2anat_coreg = {'masking': False, 'brain_extraction': False, 'registration': 'SyN'}
-> nativespace_resampling = inputs_defined
-> commonspace_resampling = inputs_defined
-> anatomical_resampling = inputs_defined
-> apply_STC = False
-> TR = auto
-> tpattern = alt-z
-> stc_axis = Y
-> interp_method = fourier
-> anat_template = /atlas/SIGMA_InVivo_Brain_Template.nii
-> brain_mask = /atlas/SIGMA_InVivo_Brain_Mask.nii
-> WM_mask = /atlas/SIGMA_InVivo_WM_mask_binary.nii
-> CSF_mask = /atlas/SIGMA_InVivo_CSF_mask_binary.nii
-> vascular_mask = /atlas/SIGMA_InVivo_CSF_mask_binary.nii
-> labels = /atlas/SIGMA_InVivo_Anatomical_Brain_Atlas.nii

241108-11:12:30,444 nipype.workflow INFO:
INPUT BIDS DATASET:
input_BIDS/
sub-01/
anat/
sub-PHG001_ses-3_acq-RARE_T2w.json
sub-PHG001_ses-3_acq-RARE_T2w.nii
func/
sub-PHG001_ses-3_task-rest_acq-EPI_run-1_bold.json
sub-PHG001_ses-3_task-rest_acq-EPI_run-1_bold.nii

241108-11:12:31,895 nipype.utils WARNING:
A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.1
/opt/conda/lib/python3.9/site-packages/bids/config.py:38: FutureWarning: Setting 'extension_initial_dot' will be removed in pybids 0.16.
warnings.warn("Setting 'extension_initial_dot' will be removed in pybids 0.16.",
241108-11:12:32,585 nipype.workflow WARNING:
The BIDS compliance failed: 'dataset_description.json' is missing from project root. Every valid BIDS dataset must have this file.
Example contents of 'dataset_description.json':
{"Name": "Example dataset", "BIDSVersion": "1.0.2"}

RABIES will run anyway; double-check that the right files were picked up for processing.

241108-11:12:32,987 nipype.workflow INFO:
Running workflow with MultiProc plugin.
241108-11:12:33,450 nipype.workflow INFO:
Workflow main_wf settings: ['check', 'execution', 'logging', 'monitoring']
241108-11:12:33,538 nipype.workflow INFO:
Running in parallel.
241108-11:12:33,583 nipype.workflow INFO:
[MultiProc] Running 0 tasks, and 3 jobs ready. Free memory (GB): 13.96/13.96, Free processors: 28/28.
241108-11:12:33,671 nipype.workflow INFO:
[Node] Setting-up "main_wf.resample_template" in "/preprocess_outputs/main_wf/resample_template".
241108-11:12:33,697 nipype.workflow INFO:
[Node] Running "resample_template" ("nipype.interfaces.utility.wrappers.Function")
...
...
...
241108-11:50:43,968 nipype.workflow INFO:
[Node] Running "estimate_motion_node" ("rabies.preprocess_pkg.hmc.EstimateMotionParams")
241108-11:50:43,984 nipype.workflow INFO:
[Node] Running "bold_transform" ("rabies.utils.slice_applyTransforms")
241108-11:50:43,991 nipype.workflow INFO:
[Node] Running "bold_transform" ("rabies.utils.slice_applyTransforms")
241108-11:50:45,796 nipype.workflow INFO:
[MultiProc] Running 3 tasks, and 0 jobs ready. Free memory (GB): 9.66/13.96, Free processors: 25/28.
Currently running:
* main_wf.bold_main_wf.estimate_motion_node
* main_wf.bold_main_wf.bold_commonspace_trans_wf.bold_transform
* main_wf.bold_main_wf.bold_native_trans_wf.bold_transform
exception calling callback for <Future at 0x7fe73b920640 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 330, in _invoke_callbacks
callback(self)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x7fe73b909190 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 330, in _invoke_callbacks
callback(self)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 330, in _invoke_callbacks
callback(self)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x7fe73b9209d0 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 330, in _invoke_callbacks
callback(self)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 330, in _invoke_callbacks
callback(self)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

[Gab-D-G]

The resources are not well managed through MultiProc. How about you try again with a lower number of local_threads?