export and shutdown timeouts for all OTLP exporters

[Arnatious]

Description

This is a solution to several issues related to the current synchronous OTLP exporters.

Currently, OTLP exporters have a couple of pain points

• OTLP exporters don't respect timeout #2663 Exporter timeouts are mostly ignored or are unintuitively applied
  • In places where they do apply, they apply to individual attempts and not to retries
• Exporters shutdown takes longer then a minute when failing to send metrics/traces #3309
  • Exporters persist for a minimum of 63 seconds after program shutdown if a transient error is ongoing
• Process never exits due to hang in shutdown #2284
  • Shutdown timeouts are ignored if exports are ongoing, and exporters have no way to cancel ongoing exports when shutdown is requested.
• Exponential backoff does not include jitter
• Timeouts are not configurable

This PR implements a new utility class, opentelemetry.exporter.otlp.proto.common.RetryingExporter, that fixes the above issues. It also significantly refactors the existing OTLP exporters to use this, and extracts retry related logic from their test suites.

Attempts were made to maintain the call signature of public APIs, though in several cases **kwargs were added to ensure future proofing, and positional arguments were renamed to create a consistent interface.

OTLP exporters will create a RetryingExporter, passing in a function performing a single export attempt as well as the OTLPExporter's timeout and export result type.

Example

from opentelemetry.exporter.otlp.proto.common import RetryingExporter, RetryableExportError

class OTLPSpanExporter(SpanExporter):
  def __init__(self, ...):
    self._exporter = RetryingExporter(self._export, SpanExportResult, self._timeout)

  def _export(self, timeout_s: float, serialized_data: bytes) -> SpanExportResult:
    result = ...

    if is_retryable(result):
      raise RetryableExportError(result.delay)
    return result

  def export(self, data, timeout_millis = 10_000, **kwargs) -> SpanExportResult:
    return self._exporter.export_with_retry(timeout_millis * 1e-3, data)

  def shutdown(self, timeout_millis = 10_000, **kwargs):
    ...
    self._exporter.shutdown(timeout_millis)
    self._shutdown = True

Fixes #3309

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Tests were added for the RetryableExporter in exporter/opentelemetry-exporter-otlp-proto-common/tests/test_retryable_exporter.py, as well as for the backoff generator in exporter/opentelemetry-exporter-otlp-proto-common/tests/test_backoff.py. Tests were updated throughout the http and grpc otlp exporters, and retry related logic was removed in all cases but for GRPC metrics, which can be split and therefore needed another layer of deadline checking.

Does This PR Require a Contrib Repo Change?

Answer the following question based on these examples of changes that would require a Contrib Repo Change:

• The OTel specification has changed which prompted this PR to update the method interfaces of opentelemetry-api/ or opentelemetry-sdk/

• The method interfaces of test/util have changed

• Scripts in scripts/ that were copied over to the Contrib repo have changed

• Configuration files that were copied over to the Contrib repo have changed (when consistency between repositories is applicable) such as in

  • pyproject.toml
  • isort.cfg
  • .flake8

• When a new .github/CODEOWNER is added

• Major changes to project information, such as in:

  • README.md
  • CONTRIBUTING.md

• Yes. - Link to PR:

• No.

Checklist:

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

[Arnatious]

I based behavior decisions off the described behavior in #2663 (comment) - namely, the shortest timeout always wins.

Processor timeout logic is unaffected - if the processor has a shorter timeout or is tracking a deadline for a batch, it passes that to export() and it is respected, if the timeout is longer, the exporter's timeout attribute (set at creation/from env variables) can be hit and cause the export to fail.

I chose to create a helper object rather than splice this into the inheritance hierarchy to avoid having a mixin with __init__, since the exporter needs to have an object-scoped event and lock, and the GRPC exporters already have several mixins with __init__ it'd have to play along with. A unified rewrite of the inheritence hierarchy shared between http and gprc exporters would probably be better.

[pmcollins]

@Arnatious apologies for the delay and thanks for this PR -- improvements to the area you've addressed are super important.

However, perhaps this is too much of a good thing all at once. Do you have availability to break these changes down into smaller PRs? This would make things much easier on reviewers.