-
Notifications
You must be signed in to change notification settings - Fork 641
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DynamoDB: Fix parallelism #876 #877
Conversation
This PR fixes the parallelism in DynamoDB so that it is applied to the HTTP connection pool. I upped the default parallelism to 32 as this is the nearest power of 2 below 50, which is the default that [AWS used in their own SDK before deprecating the methods](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/AmazonDynamoDBAsyncClient.html). I've run up to 128 connections in production (by setting the akka-http config propertis) without any problems so 32 seems perfectly safe.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense to me.
Where does the need for "power of two" come from?
The power of two is from the akka-http requirement for max open requests: # Must be a power of 2 and > 0!
max-open-requests = 32 It's an intriguing requirement but I haven't dug into the code enough to see why it's implemented that way. |
One thing that occurs to me is that with this change For example, we use 128 connections from our production servers but don't need anything like that parallelism for Any ideas? Perhaps leaving them the same is fine as it'll never get that high, or we could compute a value for it that seems more reasonable such as parallelism/8. |
I think its OK to use the same high value for the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Thank you for this improvement. Good to see you funnelling quite some data through it! |
This PR fixes #876 by changing the parallelism in DynamoDB so that it is applied to the HTTP connection pool.
I upped the default parallelism to 32 as this is the nearest power of 2 below 50, which is the default that AWS used in their own SDK before deprecating the methods. I've run up to 128 connections in production (by setting the akka-http config properties) without any problems so 32 seems perfectly safe.