Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metrics to track in-flight bytes at OTel-Arrow receiver #148

Closed
jmacd opened this issue Feb 2, 2024 · 0 comments · Fixed by #150
Closed

Add metrics to track in-flight bytes at OTel-Arrow receiver #148

jmacd opened this issue Feb 2, 2024 · 0 comments · Fixed by #150

Comments

@jmacd
Copy link
Contributor

jmacd commented Feb 2, 2024

The concurrent batch processor exposes its in-flight count, but we know that memory can pile up in the receiver waiting to be admitted by the concurrent batch processor (CBP). We'd benefit from being able to know how much memory is tied up in active requests that is stalled by the CBP's in-flight limit.

This functionality makes sense for all receivers; the community has discussed integrating memory limiter behavior as an extension, in which case the number of bytes in flight at the receiver would be used to fast-fail requests w/ RESOURCE_EXHAUSTED. For now, we'd like to track this quantity using an UpDownCounter.

Since items of telemetry vary in size, I think it would be useful in connection to know how many items are represented by the quantity of in-flight bytes. This suggests two new metrics, one for bytes and one for items, both counting in-flight numbers.

@jmacd jmacd closed this as completed in #150 Feb 9, 2024
jmacd pushed a commit that referenced this issue Feb 9, 2024
… items pending (#150)

Resolves #148

This PR adds two metrics to track the number of items and bytes that are
pending in the receiver. Specifically when used with the
`concurrentbatchprocessor` requests can be have a long lifetime in
receiver due to the in flight byte limiter in the
`concurrentbatchprocessor`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant