-
-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add timeout parameter. #621
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a nice idea, thanks for looking into it.
Do you have the black
formatter installed? Some of the changes look like they need to be formatted
Codecov Report
@@ Coverage Diff @@
## develop #621 +/- ##
===========================================
+ Coverage 80.16% 80.31% +0.15%
===========================================
Files 69 69
Lines 10338 10386 +48
===========================================
+ Hits 8287 8342 +55
+ Misses 2051 2044 -7
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
I just raised |
That would certainly be clearer, but I assume it has some difficulties with our |
I'm currently manually patching the return code to 60 as |
Sounds good to me |
I suppose the other thing you could do besides patch the return code is have the Timer function both terminate the proc and indicate it did this by setting an attribute in runset. I'm mostly indifferent, especially if we think that |
I've tried it with storing state in |
Alright, these seem to be passing now. As a timing-related thing, what are your thoughts on adding a |
Seems reasonable, but looking at that code it seems like it will do the wrong thing if more than one line of output is buffered at a time (which is bad)? So we would also definitely want to do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me. Do you want to look at the proc.poll issue in a different PR? If so I will merge this
Makes sense; maybe I'll have a look at that in a separate PR. |
As a follow up, the issue of busy-waiting is not an issue: |
Submission Checklist
Summary
This PR adds a
timeout
parameter tooptimize
,sample
,generate_quantities
, andvariational
including unit tests. I was running hyperparameter sweeps for a model and some configurations took much longer than others. Setting a timeout means that "bad" configurations can be aborted if they take too long.I used a
threading.Timer
to terminate the cmdstan subprocess. Checking the elapsed time in_run_cmdstan
didn't work because thereadline
call blocks until the next line ofstdout
becomes available (which can take a while if the model is sampling inefficiently).Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company):
Harvard University
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses: