Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add metrics to kafkareceiver #3452

Merged
merged 1 commit into from
Jun 16, 2021

Conversation

Frefreak
Copy link
Member

Description:
Currently kafkareceiver doesn't support metrics type. This PR adds metrics support to it (almost a direct copy/substitute from traces/logs implementation).

Testing:

  • go test ./...: under kafkareceiver folder, all pass
  • make test: some cases failed locally but should be unrelated
  • manual test with this configs (another collector configured with hostmetrics receiver and kafka exporter):
receivers:
  otlp:
    protocols:
      grpc:
      http:
  kafka/a:
    protocol_version: "2.0.0"
    topic: asdf_traces
  kafka/b:
    protocol_version: "2.0.0"
    topic: asdf_metrics

processors:
  batch:

exporters:
  logging:
    logLevel: debug

service:
  pipelines:
    traces:
      receivers: [kafka/a]
      exporters: [logging]
    metrics:
      receivers: [kafka/b]
      exporters: [logging]

Caveats:

If same config is used between traces/metrics/logs, it will errors about:

Error: cannot build pipelines: cannot build receivers: factory for kafka/a is implemented incorrectly: CreateTracesReceiver and CreateMetricsReceiver must return the same receiver pointer when creating receivers of different data types

The error message says traces and metrics, but this would actually also error when used against traces and logs. Because of this currently this receiver needs multiple kafka configs (and should use different topics) to use in traces/metrics/logs.

I thought about merging traces/metrics/logs into one receiver. However in that case we need separated topic config to differentiate between different data types, otherwise the receiver has no way to tell the type of the message its consuming without trial and error, which would be inefficient. This PR can be seen as a starting point of that and just enables using metrics in kafkareceiver.

@Frefreak Frefreak requested review from a team and owais June 16, 2021 11:06
@bogdandrutu bogdandrutu merged commit 46ed756 into open-telemetry:main Jun 16, 2021
@Frefreak Frefreak deleted the kafkareceiver_metrics branch June 17, 2021 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants