Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kafkareceiver] commit unprocessed messages #6222

Closed
arnitolog opened this issue Nov 10, 2021 · 3 comments
Closed

[kafkareceiver] commit unprocessed messages #6222

arnitolog opened this issue Nov 10, 2021 · 3 comments
Assignees
Labels
bug Something isn't working comp:kafka Kafka related issues receiver/kafka Stale

Comments

@arnitolog
Copy link

Describe the bug
it looks like Kafkareceiver commits even un-processed (or processed with error) messages.

Steps to reproduce
configure kafkareciver with the below config:

  receivers:
    kafka:
      protocol_version: 2.8.0
      brokers: kafka-cluster-kafka-bootstrap.kafka.svc.cluster.local:9093
      topic: jaeger-spans
      encoding: jaeger_proto
      group_id: otel-collector
      client_id: otel-collector
      auth:
        tls:
          ca_file: /tmp/certificates/ca.crt
          insecure: true
      autocommit:
        enable: false
      message_marking: 
        after: true
        on_error: false
  exporters:
    otlp:
      endpoint: some-fake-endpoint.svc.cluster.local:9999
      tls:
        insecure: true
  service:
    extensions:
      - health_check
    pipelines:
      traces:
        receivers: [kafka]
        exporters: [otlp]

Push traces into jaeger-spans topic and check LAG for otel-collector consumer group.

What did you expect to see?
LAG should constantly grow as we can't complete the pipeline due to the exporters endpoint is unavailable/not reachable

What did you see instead?
LAG is going up and down, which means that offsets are committed back

What version did you use?
Version:
opentelemetry collector 0.38.0

Additional context
queue and LAG:
image

spans accepted rate:
image

exporter sent rate:
image

@arnitolog arnitolog added the bug Something isn't working label Nov 10, 2021
@alolita alolita added the comp:kafka Kafka related issues label Nov 10, 2021
@github-actions
Copy link
Contributor

github-actions bot commented Nov 7, 2022

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

@github-actions github-actions bot added the Stale label Nov 7, 2022
povilasv referenced this issue in coralogix/opentelemetry-collector-contrib Dec 19, 2022
* Instrument obs_receiver with otel go

* add changelog entry

* remove views configuration

* move otel-go metrics to obsreport

* add UseOtelForMetrics to TelemetrySettings; extract otel sdk initialization changes to another pr; remove public InstrumentWithOtel from obsreport;

* default UseOtelForMetrics to false

* remove duplicate oc registry initialization

* move `UseOtelForMetrics` to the internal package `obsreportconfig`

* address comments

* only register view if featuregate is disabled

* prefix meter name with `receiver/`

* change meter scope name

* upgrade otel metric sdk

* run make gotidy

* update changelog

* revert: otel go version upgrade

* revert: x/sys upgrades
@atoulme
Copy link
Contributor

atoulme commented Mar 12, 2023

Closing as completed.

@atoulme atoulme closed this as completed Mar 12, 2023
@github-actions
Copy link
Contributor

Pinging code owners for receiver/kafka: @pavolloffay @MovieStoreGuy. See Adding Labels via Comments if you do not have permissions to add labels yourself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working comp:kafka Kafka related issues receiver/kafka Stale
Projects
None yet
Development

No branches or pull requests

4 participants