Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change how "bytes per second" is calculated #13

Open
hariso opened this issue Sep 6, 2022 · 0 comments
Open

Change how "bytes per second" is calculated #13

hariso opened this issue Sep 6, 2022 · 0 comments
Labels
feature New feature or request

Comments

@hariso
Copy link
Contributor

hariso commented Sep 6, 2022

Feature description

There are two important time related metrics we measure for records:

  1. the time a record spends in Conduit only (from the time it's read till the time it's ack'd) and,
  2. the time a record spends in the whole pipeline, on average

The first is provided by the built-in metrics, and its measured for each record individually. The second one is calculated (by dividing the pipeline run time with the number of records). The first one should be used carefully in test interpretations, since there's some parallelism in Conduit, due to which the the time Conduit spends handling a record is not always (inversely) proportionate with the average pipeline rate.

The "bytes per second" we have currently is using the first time duration, and as such may be confusing and not always indicative of Conduit's performance. We should change it to be "total bytes ÷ pipeline run time".

@hariso hariso added feature New feature or request triage Needs to be triaged labels Sep 6, 2022
@meroxa-machine meroxa-machine moved this to Triage in Conduit Main Sep 6, 2022
@lovromazgon lovromazgon removed the status in Conduit Main Oct 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
Status: No status
Development

No branches or pull requests

1 participant