client: Cache small data chunks before PUTing them #650
Labels
client
Issue related to the client
enhancement
Improving existing functionality
I3
Minimal impact
performance
More of something per second
S4
Routine
U4
Nothing urgent
Is your feature request related to a problem? Please describe.
https://pkg.go.dev/github.com/nspcc-dev/neofs-sdk-go/client#DefaultObjectWriter.Write sends at least one message to the underlying stream when chunk is non-empty regardless of its size. Each message requires preparation incl. pretty heavyweight cryptography. The server also verifies each message. Even besides app processing, in general, the more messages the slower the stream
thus, unintentional transition to granular chunks (e.g. when using
io.Copy
) can significantly slow down the processDescribe the solution you'd like
bufferize smaller chunks, and flush the buffer to the stream when it's full. This would minimize the number of messages
the disadvantage of this approach will be an additional buffer for each operation. Diff with the current approach will require careful benchmark testing
there is also a nuance with dynamic buffer allocation: how to know if the intended data volume (unknown in general) is less than the limit so to not overallocated the buffer? In special cases, we can make a hypothesis:
Describe alternatives you've considered
Additional context
Client
testingThe text was updated successfully, but these errors were encountered: