Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

forwardconnector shouldn't set MutatesData: false #7776

Closed
seankhliao opened this issue May 28, 2023 · 0 comments · Fixed by #7779
Closed

forwardconnector shouldn't set MutatesData: false #7776

seankhliao opened this issue May 28, 2023 · 0 comments · Fixed by #7779
Labels
bug Something isn't working

Comments

@seankhliao
Copy link
Contributor

Describe the bug
A filter processor in a second pipeline affects export data in a previous pipeline connected by forwardconnector

Steps to reproduce
Try running the config provided below multiple times, it sometimes works, sometimes breaks.

What did you expect to see?
Original pipeline unaffected by second pipeline,
logging/1: 17 metrics
logging/2: 1 metric

log output:

2023-05-28T17:05:15.454+0100	info	service/telemetry.go:104	Setting up own telemetry...
2023-05-28T17:05:15.454+0100	info	service/telemetry.go:127	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
2023-05-28T17:05:15.454+0100	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "logging/2"}
2023-05-28T17:05:15.456+0100	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "logging/1"}
2023-05-28T17:05:15.459+0100	info	connector/connector.go:636	Development component. May change in the future.	{"kind": "exporter", "name": "forward", "exporter_in_pipeline": "metrics", "receiver_in_pipeline": "metrics"}
2023-05-28T17:05:15.464+0100	info	service/service.go:131	Starting otelcol-custom...	{"Version": "1.0.0", "NumCPU": 16}
2023-05-28T17:05:15.464+0100	info	extensions/extensions.go:30	Starting extensions...
2023-05-28T17:05:15.464+0100	info	[email protected]/metrics_receiver.go:255	Starting discovery manager	{"kind": "receiver", "name": "prometheus_simple", "data_type": "metrics"}
2023-05-28T17:05:15.466+0100	info	[email protected]/metrics_receiver.go:243	Scrape job added	{"kind": "receiver", "name": "prometheus_simple", "data_type": "metrics", "jobName": "prometheus_simple/127.0.0.1:8888"}
2023-05-28T17:05:15.467+0100	info	service/service.go:148	Everything is ready. Begin running and processing data.
2023-05-28T17:05:15.467+0100	info	[email protected]/metrics_receiver.go:289	Starting scrape manager	{"kind": "receiver", "name": "prometheus_simple", "data_type": "metrics"}
2023-05-28T17:05:20.525+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 14, "data points": 17}
2023-05-28T17:05:20.525+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:20.629+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 17, "data points": 21}
2023-05-28T17:05:20.629+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:21.623+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 17, "data points": 21}
2023-05-28T17:05:21.623+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:21.723+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 17, "data points": 21}
2023-05-28T17:05:21.723+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:22.624+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 17, "data points": 21}
2023-05-28T17:05:22.624+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:22.724+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 17, "data points": 21}
2023-05-28T17:05:22.724+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:23.724+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 17, "data points": 21}
2023-05-28T17:05:23.724+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:23.825+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 17, "data points": 21}
2023-05-28T17:05:23.825+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:24.726+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 17, "data points": 21}
2023-05-28T17:05:24.726+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:24.824+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 17, "data points": 21}
2023-05-28T17:05:24.824+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
^C2023-05-28T17:05:25.529+0100	info	otelcol/collector.go:227	Received signal from OS	{"signal": "interrupt"}
2023-05-28T17:05:25.529+0100	info	service/service.go:157	Starting shutdown...
2023-05-28T17:05:25.529+0100	info	extensions/extensions.go:44	Stopping extensions...
2023-05-28T17:05:25.529+0100	info	service/service.go:171	Shutdown complete.

What did you see instead?
Data appears to be dropped in original pipeline
logging/1: 1 metric
logging/2: 1 metric

2023-05-28T17:05:27.499+0100	info	service/telemetry.go:104	Setting up own telemetry...
2023-05-28T17:05:27.499+0100	info	service/telemetry.go:127	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
2023-05-28T17:05:27.499+0100	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "logging/1"}
2023-05-28T17:05:27.501+0100	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "logging/2"}
2023-05-28T17:05:27.505+0100	info	connector/connector.go:636	Development component. May change in the future.	{"kind": "exporter", "name": "forward", "exporter_in_pipeline": "metrics", "receiver_in_pipeline": "metrics"}
2023-05-28T17:05:27.507+0100	info	service/service.go:131	Starting otelcol-custom...	{"Version": "1.0.0", "NumCPU": 16}
2023-05-28T17:05:27.507+0100	info	extensions/extensions.go:30	Starting extensions...
2023-05-28T17:05:27.508+0100	info	[email protected]/metrics_receiver.go:255	Starting discovery manager	{"kind": "receiver", "name": "prometheus_simple", "data_type": "metrics"}
2023-05-28T17:05:27.510+0100	info	[email protected]/metrics_receiver.go:243	Scrape job added	{"kind": "receiver", "name": "prometheus_simple", "data_type": "metrics", "jobName": "prometheus_simple/127.0.0.1:8888"}
2023-05-28T17:05:27.510+0100	info	service/service.go:148	Everything is ready. Begin running and processing data.
2023-05-28T17:05:27.510+0100	info	[email protected]/metrics_receiver.go:289	Starting scrape manager	{"kind": "receiver", "name": "prometheus_simple", "data_type": "metrics"}
2023-05-28T17:05:32.524+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:32.524+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:32.627+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:32.627+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:33.525+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:33.525+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:33.626+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:33.626+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:34.629+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:34.629+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:34.726+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:34.726+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:35.725+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:35.725+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:35.824+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/2", "resource metrics": 1, "metrics": 1, "data points": 1}
2023-05-28T17:05:35.824+0100	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging/1", "resource metrics": 1, "metrics": 1, "data points": 1}
^C2023-05-28T17:05:35.985+0100	info	otelcol/collector.go:227	Received signal from OS	{"signal": "interrupt"}
2023-05-28T17:05:35.985+0100	info	service/service.go:157	Starting shutdown...
2023-05-28T17:05:35.985+0100	info	extensions/extensions.go:44	Stopping extensions...
2023-05-28T17:05:35.986+0100	info	service/service.go:171	Shutdown complete.

What version did you use?
Version: v0.77.0

What config did you use?
Config:

receivers:
  prometheus_simple:
    collection_interval: 0.1s
    endpoint: 127.0.0.1:8888

processors:
  filter:
    metrics:
      metric:
        - 'name != "up"'
exporters:
  logging/1:
  logging/2:
connectors:
  forward:
service:
  pipelines:
    metrics/1:
      receivers:
        - prometheus_simple
      exporters:
        - logging/1
        - forward
    metrics/2:
      receivers:
        - forward
      processors:
        - filter
      exporters:
        logging/2

Additional context

This appears to either consistently work, or consistently not with no change to config.
@seankhliao seankhliao added the bug Something isn't working label May 28, 2023
@seankhliao seankhliao mentioned this issue May 28, 2023
Closed
@djaglowski djaglowski mentioned this issue May 30, 2023
Merged
codeboten pushed a commit that referenced this issue Jun 5, 2023
@djaglowski
Incorporate next consumer's capabilities into connector capabilities (#…
6ababc9 
…7779)

Resolves #7776

The collector copies data whenever handing it to components that may mutate it. The point at which a copy must be made is within a fanout consumer. There are two points at which a fanout consumer may be placed:
1. Immediately after a receiver shared by multiple pipelines
2. Immediately before multiple exporters of a single pipeline

When data is fanned out _to_ a connector (acting as an exporter), we must ensure that the fanout consumer before exporters (point 2 above) is able to take into account whether or not the connector will be handing the data off to a pipeline that will mutate the data.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
1 participant
@seankhliao