Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce number of DBS concurrent requests in MSTransferor pycurl #9752

Closed
amaltaro opened this issue Jun 17, 2020 · 0 comments · Fixed by #9785
Closed

Reduce number of DBS concurrent requests in MSTransferor pycurl #9752

amaltaro opened this issue Jun 17, 2020 · 0 comments · Fixed by #9785

Comments

@amaltaro
Copy link
Contributor

amaltaro commented Jun 17, 2020

Impact of the new feature
MSTransferor

Is your feature request related to a problem? Please describe.
DBS is tightening up the throttling limits in DBSReader Global (decreasing it from 10 to 5 requests per backend/DN now):
dmwm/deployment#919

and we better decrease the number of concurrent requests made against DBS as well, to avoid getting the service in an endless failure mode created by the concurrency/throttling in place.

Describe the solution you'd like
Ideally, the number of concurrent requests should be made as a configuration parameter, which would propagate all the way down to the method actually creating the pycurl requests.

Extra: we should print the amount of requests we have to make and the number of concurrent requests getting created.

Describe alternatives you've considered
If too much work to make it configurable, I think it's okay to have those limits hardwired in the source code.

Additional context
none

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant