Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](write)add metrics for flink load data #287

Merged
merged 17 commits into from
Jan 5, 2024

Conversation

vinlee19
Copy link
Contributor

@vinlee19 vinlee19 commented Jan 5, 2024

Proposed changes

When you import data to Apache Doris in real time , you can observe the import of some metrics in Flink webui metrics. The metrics are a concatenation of table, and Flink metrics, e.g. table_1_totalFlushLoadBytes. Below is a description of how to use several metrics:

Name Metric Type Description
totalFlushLoadBytes Counter Number of bytes imported.
flushTotalNumberRows Counter Number of rows imported for total processing
totalFlushLoadedRows Counter Number of rows successfully imported.
totalFlushTimeMs Counter Number of Import completion time. Unit milli
totalFlushSucceededNumber Counter Number of times that the data-batch been successfully imported.
totalFlushFailedNumber Counter Number of times that the data-batch been failed.
totalFlushFilteredRows Counter Number of rows that do not qualify for data quality flushed
totalFlushUnselectedRows Counter Number of rows filtered by where condition flushed
beginTxnTimeMs Histogram The time cost for RPC to Fe to begin a transaction, Unit milliseconds.
putDataTimeMs Histogram The time cost for RPC to Fe to get a stream load plan, Unit milliseconds.
readDataTimeMs Histogram Read data time, Unit milliseconds.
writeDataTimeMs Histogram Write data time, Unit milliseconds.
commitAndPublishTimeMs Histogram The time cost for RPC to Fe to commit and publish a transaction, Unit milliseconds.
loadTimeMs Histogram Import completion time
The monitoring metrics for Flink write are shown below: 截屏2024-01-05 15 37 00 截屏2024-01-05 15 47 45

Problem Summary:

Describe the overview of changes.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

String.format(
METRIC_NAME_FORMAT,
tableIdentifier,
subTaskId,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add subTaskId? It seems to be there by default?

.ifPresent(totalFlushNumberTotalRows::inc);
Optional.ofNullable(responseContent.getNumberFilteredRows())
.ifPresent(totalFlushFilteredRows::inc);
Optional.ofNullable(responseContent.getLoadTimeMs()).ifPresent(totalFlushTimeMs::inc);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is getLoadTimeMs also a counter-type metric?

Copy link
Contributor Author

@vinlee19 vinlee19 Jan 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two metric type for LoadTimeMs: Counter and Historgram ,

@JNSimba
Copy link
Member

JNSimba commented Jan 5, 2024

Thanks for the @vinlee19 . Can you post some final monitoring screenshots?

@vinlee19
Copy link
Contributor Author

vinlee19 commented Jan 5, 2024

Thanks for the @vinlee19 . Can you post some final monitoring screenshots?

Yeah,I will post the final monitoring screenshots later.

Copy link
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JNSimba JNSimba merged commit dc1f0ec into apache:master Jan 5, 2024
8 checks passed
@JNSimba JNSimba mentioned this pull request Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants