OTLP/GRPC exporter implementation - Missing retries on Exporter errors

[bobstrecansky]

We're missing retries on Exporter errors: https://github.com/open-telemetry/opentelemetry-specification/blob/318745aaf6fb173eb5cbc887ab76796bd817b3d6/specification/protocol/exporter.md#retry

[Grunet]

Here are some notes from my quick look at the current state of other client SDK implementations around this:

• Go is the only one I see evidence of having tried implementing this, but possibly only partially (i.e. maybe just for gRPC/http and traces?)
• Java is waiting on further clarification around the spec before starting, in particular around how the backoff parameters should be made adjustable by consumers
  • Side note: per that spec thread, gRPC maybe has built-in capacities for backoff, but I'm not sure if Go or any others are currently relying on that
• .NET's issue for this only seems to discuss gRPC support

And for PHP, I'm not sure how backoff strategies typically get implemented (in particular for a library that needs to be very generically reusable in various different frameworks, e.g. the reactive ones I've heard about) so if anyone happens to know of a really good example of this from somewhere else on the internet, feel free to link to it here

[tidal]

This is not feasible as a generic feature of the exporter, as it pretty much depends on the runtime, if this will create problems or not. So if anything this feature has to be turned on explicitly by the user/instrumentation.
The problems can occure mainly (but not only) in PHP-FPM and traditional web servers, as there PHP is normally executed in a separate single-threaded, share-nothing process, and this process is expected be quick & dirty clean. Or in other words request termination (RSHUTDOWN) should not be defered (especially not because of cross-cutting concerns) , as this will keep the connection to the client open. In the best case this results in the loading spinner in the browser keep spinning while the page is already "loaded", but in the worst case the process may exeed the configured max-execution-time and create an error with possible side-effects.

For non-traditional runtimes this is less of a problems, in particular not for the event-based ones. Especially Swoole and Swoft can handle back-offs with ease, as they can be executed in coroutines, while pure event-based runtimes (ReactPHP, AMP, etc.) can only defer execution like JavaScript.

To cut a long story short. To have a safe implementation of this feature for all runtimes, the retry must not happen in the same place, where the initial try happened. So retries have to go into a queue upon which a daemon or agent will act. Since classic queues can also possible fail because of networks errors, etc, ZeroMQ could be used as a broker-less buffered transport mechanism to guarantee delivery.

btw: I think the exporter is blocking on the grpc request. Is there any way to change this to be asynchronous?

[tidal]

@bobstrecansky
Could we settle on making retries a configurable option (default=off)? As outlined above, I don't see how we can implement this otherwise without creating complaints/bug-tickets of SDK users in the future.
On the other hand we could use this setting for other exporters as well.

[bobstrecansky]

After discussing this in the SIG meeting: Yes, this sounds reasonable to me.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity, but it can be reopened. Thank you for your contributions.

[sanketmehta28]

Hi @tidal and @bobstrecansky : Can you assign this to me?