Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support collect routine load latency #52890

Open
hoffermei opened this issue Nov 14, 2024 · 0 comments · May be fixed by #52878
Open

support collect routine load latency #52890

hoffermei opened this issue Nov 14, 2024 · 0 comments · May be fixed by #52878

Comments

@hoffermei
Copy link
Contributor

在实时数据要求非常高的场景,当前无法观测数据延迟,不能直观看出参数调优的效果。
en:
In scenarios with very high real-time data requirements, it is currently impossible to observe data delay and the effect of parameter tuning cannot be intuitively seen

Feature request

暴露延迟时间指标给prometheus, 方便通过grafana等工具查看延迟变化
en:
Expose latency metrics to prometheus, making it easier to view latency changes through tools such as grafana

Implement plan

获取每批次第一条数据写入到消息队列的时间戳,计算与starrocks事务提交时间戳之间的差值,即可得出这个批次的最大延迟时间。延迟数据暴露通过先写入到histogram metric,在每次查询metric到时候返回给prometheus agent。

en:
Get the timestamp of the first data in each batch written to the message queue, calculate the difference between it and the timestamp of the starrocks transaction submission, and you can get the maximum delay time of this batch. The delay data is exposed by first writing it to the histogram metric and returning it to the prometheus agent every time the metric is queried.

@hoffermei hoffermei linked a pull request Nov 14, 2024 that will close this issue
24 tasks
hoffermei added a commit to hoffermei/starrocks that referenced this issue Nov 14, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this issue Nov 20, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this issue Nov 20, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this issue Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant