Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance degradation on python producer.send with large number of records #258

Open
AlexMikhalev opened this issue May 23, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@AlexMikhalev
Copy link

AlexMikhalev commented May 23, 2023

slack thread: https://infinyon.slack.com/archives/CQ8UX6SBH/p1684349700779719

simple producer in python:

#!/usr/bin/env python3
from datetime import datetime
from fluvio import Fluvio
import time 

PARTITION = 0

if __name__ == "__main__":
   # Connect to cluster
   fluvio = Fluvio.connect()

   producer = fluvio.topic_producer("simple")

   start_time = time.time()

   for x in range(10000):
       producer.send_string("{}: timestamp: {}".format(x, datetime.now()))

   # Flush the last entry
   producer.flush()
   print("completed in  {} seconds", (time.time()-start_time))

10K records pushed in 1.2153611183166504 seconds, when if you increase the range to 100000
python3 producer100K.py
The same code takes 64.67606687545776 to complete.
Rust producer doesn't have such behaviour.

@digikata digikata added the bug Something isn't working label Sep 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants