Error during sampling

[yingyuctw]

Summary:

Please provide a short couple sentence summary.
I try to run cmdstanpy. But it keep showing error.

Description:

Describe the issue as clearly as possible.
I am using pycharm to run the code. I check error code on stackflow. In docker, it is issue about c++. So, I try to update the c++ version. It does not work.

Additional Information:

Provide any additional information here.
raise RuntimeError(msg)
`RuntimeError: Error during sampling:

Command and output files: RunSet: chains=4, chain_ids=[1, 2, 3, 4], num_processes=4
cmd (chain 1): ['D:\xxx\Dropbox\Fall2022\dotstudy-master\data\dotstudy\lnrm2_v2.exe', 'id=1', 'random', 'seed=98393', 'data', 'file=C:\\Users\\xxx\\AppData\\Local\\Temp\\tmp36qi7n31\\_mweuysg.json', 'init=C:\\Users\\lalor\\AppData\\Local\\Temp\tmp36qi7n31\l4eb0hu8.json', 'output', 'file=C:\Users\lalor\AppData\Local\Temp\tmp36qi7n31\lnrm2_v2zw2ep36n\\lnrm2_v2-20230130162342_1.csv', 'method=sample', 'algorithm=hmc', 'adapt', 'engaged=1']
retcodes=[3221225781, 3221225781, 3221225781, 3221225781]
per-chain output files (showing chain 1 only):
csv_file:
C:\Users\xxx\AppData\Local\Temp\tmp36qi7n31\lnrm2_v2zw2ep36n\lnrm2_v2-20230130162342_1.csv
console_msgs (if any):
C:\Users\xxx\AppData\Local\Temp\tmp36qi7n31\lnrm2_v2zw2ep36n\lnrm2_v2-20230130162342_0-stdout.txt
Consider re-running with show_console=True if the above output is unclear!

Current Version:

Please include the output of import cmdstanpy; cmdstanpy.show_versions(), or
at least the cmdstan and cmdstanpy versions used.
1.0.8

[WardBrian]

Can you run the code with 'show_console=True` and paste that output?

Are you able to share your Stan code?

[yingyuctw]

11:59:00 - cmdstanpy - INFO - Chain [1] start processing
11:59:00 - cmdstanpy - INFO - Chain [2] start processing
11:59:00 - cmdstanpy - INFO - Chain [3] start processing
11:59:00 - cmdstanpy - INFO - Chain [4] start processing
11:59:00 - cmdstanpy - INFO - Chain [1] done processing
11:59:00 - cmdstanpy - ERROR - Chain [1] error: terminated by signal 3221225653
11:59:00 - cmdstanpy - INFO - Chain [2] done processing
11:59:00 - cmdstanpy - ERROR - Chain [2] error: terminated by signal 3221225653
11:59:00 - cmdstanpy - INFO - Chain [4] done processing
11:59:00 - cmdstanpy - ERROR - Chain [4] error: terminated by signal 3221225653
11:59:00 - cmdstanpy - INFO - Chain [3] done processing
11:59:00 - cmdstanpy - ERROR - Chain [3] error: terminated by signal 3221225653
Traceback (most recent call last):
File "D:\xxx\Dropbox\Fall2022\dotstudy-master\data\dotstudy\dotstudy_ver4(5).py", line 759, in
prx_levels = find_salience(dat_prx, 3, 1)
File "D:\xxx\Dropbox\Fall2022\dotstudy-master\data\dotstudy\adaptive_SFT.py", line 198, in find_salience
Chain [2]
Chain [2]
Chain [2]
Chain [2] Chain [1]
Chain [1]

Chain [2]
Chain [1]
Chain [1]
Chain [1]
Chain [1] Chain [2]

Chain [1]
Chain [2]
Chain [1]
Chain [1]
Chain [2]
Chain [2]
Chain [1]
Chain [2]
Chain [2]
Chain [2]
Chain [2]
Chain [2]
Chain [2]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [4]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
Chain [3]
fit_model = sm.sample(data=dat, inits=init_dict,show_console=True)
File "D:\xxx\Dropbox\Fall2022\venv\lib\site-packages\cmdstanpy\model.py", line 1188, in sample
raise RuntimeError(msg)
RuntimeError: Error during sampling:

Command and output files:
RunSet: chains=4, chain_ids=[1, 2, 3, 4], num_processes=4
cmd (chain 1):
['D:\xxx\Dropbox\Fall2022\dotstudy-master\data\dotstudy\lnrm2_v2.exe', 'id=1', 'random', 'seed=86162', 'data', 'file=C:\Users\lalor\AppData\Local\Temp\tmpjq2dphzl\6273ssf8.json', 'init=C:\Users\lalor\AppData\Local\Temp\tmpjq2dphzl\efcagbg7.json', 'output', 'file=C:\Users\lalor\AppData\Local\Temp\tmpjq2dphzl\lnrm2_v2k4xkqkgr\lnrm2_v2-20230206115900_1.csv', 'method=sample', 'algorithm=hmc', 'adapt', 'engaged=1']
retcodes=[3221225781, 3221225781, 3221225781, 3221225781]
per-chain output files (showing chain 1 only):
csv_file:
C:\Users\xxx\AppData\Local\Temp\tmpjq2dphzl\lnrm2_v2k4xkqkgr\lnrm2_v2-20230206115900_1.csv
console_msgs (if any):
C:\Users\xxx\AppData\Local\Temp\tmpjq2dphzl\lnrm2_v2k4xkqkgr\lnrm2_v2-20230206115900_0-stdout.txt
Consider re-running with show_console=True if the above output is unclear!

[bob-carpenter]

Hi, @yingyuctw. Can you try running it without dropbox and with argument show_console=True? The latter should actually provide the content of the error messages.

You can try looking in the temp directory specified to see if the output is actually there.

It looks like it's failing at the end while manipulating files of draws so it may just be a write permission failure.

[WardBrian]

3221225653 seems like it is sometimes used as the error code for device IO timeout, which certainly could be dropbox/permissions related.

[yingyuctw]

I don't know why it generate lnrm2_v2.exe in dropbox. I move the folder to different space. It still generates in dropbox. Anyway to reset it?

Traceback (most recent call last):
File "D:\lalor\Documents\WORK\dotstudy_ver4(5).py", line 759, in
prx_levels = find_salience(dat_prx, 3, 1)
File "D:\lalor\Documents\WORK\adaptive_SFT.py", line 198, in find_salience
fit_model = sm.sample(data=dat, inits=init_dict,show_console=True)
File "C:\Program Files\PsychoPy\lib\site-packages\cmdstanpy\model.py", line 1188, in sample
raise RuntimeError(msg)
RuntimeError: Error during sampling:

Command and output files:
RunSet: chains=4, chain_ids=[1, 2, 3, 4], num_processes=4
cmd (chain 1):
['D:\lalor\Dropbox\Fall2022\dotstudy-master\data\dotstudy\lnrm2_v2.exe', 'id=1', 'random', 'seed=37954', 'data', 'file=C:\Users\lalor\AppData\Local\Temp\tmp_k5lnenz\njipmno6.json', 'init=C:\Users\lalor\AppData\Local\Temp\tmp_k5lnenz\bnax8cq1.json', 'output', 'file=C:\Users\lalor\AppData\Local\Temp\tmp_k5lnenz\lnrm2_v2z72ov65v\lnrm2_v2-20230206195238_1.csv', 'method=sample', 'algorithm=hmc', 'adapt', 'engaged=1']
retcodes=[-1, -1, -1, -1]
per-chain output files (showing chain 1 only):
csv_file:
C:\Users\lalor\AppData\Local\Temp\tmp_k5lnenz\lnrm2_v2z72ov65v\lnrm2_v2-20230206195238_1.csv
console_msgs (if any):
C:\Users\lalor\AppData\Local\Temp\tmp_k5lnenz\lnrm2_v2z72ov65v\lnrm2_v2-20230206195238_0-stdout.txt
Consider re-running with show_console=True if the above output is unclear!

[yingyuctw]

I upload the code I use to gist.
I hope it help.
https://gist.github.com/yingyuctw/fa3e85c3219f85244fee7888b922f07c
https://gist.github.com/yingyuctw/a7e585955c4a1c04d6872d49c01083e2
https://gist.github.com/yingyuctw/143e86ff59982a7fd9978cfbd72b857e

[yingyuctw]

I try to update the code.

sm =CmdStanModel(stan_file='D:\xxx\Documents\WORK\lnrm2_v2.stan',exe_file='D:\xxx\Documents\WORK')

It still generate error. I remove the folder from dropbox, It still show same error

Command and output files:
RunSet: chains=4, chain_ids=[1, 2, 3, 4], num_processes=4
cmd (chain 1):
['D:\lalor\Dropbox\Fall2022\dotstudy-master\data\dotstudy\lnrm2_v2.exe', 'id=1', 'random', 'seed=28425', 'data', 'file=C:\Users\lalor\AppData\Local\Temp\tmpmm_alrxz\yrqwbvyb.json', 'init=C:\Users\lalor\AppData\Local\Temp\tmpmm_alrxz\0cwc3j2t.json', 'output', 'file=C:\Users\lalor\AppData\Local\Temp\tmpmm_alrxz\lnrm2_v2wkejlf4h\lnrm2_v2-20230211205639_1.csv', 'method=sample', 'algorithm=hmc', 'adapt', 'engaged=1']
retcodes=[-1, -1, -1, -1]
per-chain output files (showing chain 1 only):
csv_file:
C:\Users\lalor\AppData\Local\Temp\tmpmm_alrxz\lnrm2_v2wkejlf4h\lnrm2_v2-20230211205639_1.csv
console_msgs (if any):
C:\Users\lalor\AppData\Local\Temp\tmpmm_alrxz\lnrm2_v2wkejlf4h\lnrm2_v2-20230211205639_0-stdout.txt

[ahartikainen]

Can you share the code you run? Do you use any caching modules?

[yingyuctw]

@ahartikainen
https://gist.github.com/yingyuctw/fa3e85c3219f85244fee7888b922f07c
https://gist.github.com/yingyuctw/a7e585955c4a1c04d6872d49c01083e2
https://gist.github.com/yingyuctw/143e86ff59982a7fd9978cfbd72b857e

The code I am using. Dotstudy call adapt_SFT run cmdstanpy
The Stan model is lnrm2_v2.stan.

[ahartikainen]

Maybe your code is still using the pickled model?
'I recommend trying to create a simple script that just compiles the model and then samples.

Now there is a lot of going on. e.g. by default .exe is created to same folder as .stan I think I saw dropbox folder there.

Then another script first checks if pickled model is found (in what folder, same as in python file, because that calls chdir?).

Also, what is the purpose pickling?

[yingyuctw]

Now there is a lot of going on. e.g. by default .exe is created to same folder as .stan I think I saw dropbox folder there. @ahartikainen

Can you tell me which line? I think I removed everything.

[ahartikainen]

Not sure
https://gist.github.com/yingyuctw/fa3e85c3219f85244fee7888b922f07c#file-dotstudy-py-L73

Any case, can you create a minimal example where you have data as json and stan file etc in a same folder as your cmdstanpy script. Don't use chdir etc commands.

[yingyuctw]

@ahartikainen I change the path you mention. It still the same.
I will try to rewrite the code. It was written by other people. I am not fully understand how the code work, including pickel model. Thanks for your advice.