Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

client: Cache small data chunks before PUTing them #650

Open
cthulhu-rider opened this issue Dec 5, 2024 · 0 comments
Open

client: Cache small data chunks before PUTing them #650

cthulhu-rider opened this issue Dec 5, 2024 · 0 comments
Labels
client Issue related to the client enhancement Improving existing functionality I3 Minimal impact performance More of something per second S4 Routine U4 Nothing urgent

Comments

@cthulhu-rider
Copy link
Contributor

cthulhu-rider commented Dec 5, 2024

Is your feature request related to a problem? Please describe.

https://pkg.go.dev/github.com/nspcc-dev/neofs-sdk-go/client#DefaultObjectWriter.Write sends at least one message to the underlying stream when chunk is non-empty regardless of its size. Each message requires preparation incl. pretty heavyweight cryptography. The server also verifies each message. Even besides app processing, in general, the more messages the slower the stream

thus, unintentional transition to granular chunks (e.g. when using io.Copy) can significantly slow down the process

Describe the solution you'd like

bufferize smaller chunks, and flush the buffer to the stream when it's full. This would minimize the number of messages

the disadvantage of this approach will be an additional buffer for each operation. Diff with the current approach will require careful benchmark testing

there is also a nuance with dynamic buffer allocation: how to know if the intended data volume (unknown in general) is less than the limit so to not overallocated the buffer? In special cases, we can make a hypothesis:

  • if the object is sewn, look at the payload length
  • add an initialization option (which ofc will not be used by everyone)

Describe alternatives you've considered

  1. keep as is
  2. leave the number of messages, but remove the signing of chunk requests. This will improve performance, but will require an alternative integrity check (e.g. checksum)
  3. expose recommended buffer size for users so some of them (but not all ofc) could stream their data the most optimal way

Additional context

Client testing

@cthulhu-rider cthulhu-rider added enhancement Improving existing functionality client Issue related to the client performance More of something per second labels Dec 5, 2024
@roman-khimov roman-khimov added U4 Nothing urgent S4 Routine I3 Minimal impact labels Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
client Issue related to the client enhancement Improving existing functionality I3 Minimal impact performance More of something per second S4 Routine U4 Nothing urgent
Projects
None yet
Development

No branches or pull requests

2 participants