syslog_parser with syslog exporter results in logs like 2023-08-09T06:34:49.490942+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:34:48Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)

[captainfalcon23]

Component(s)

No response

What happened?

Description

I am following the advice at:

• https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/syslogexporter
• https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/syslog_parser.md
• https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/syslogexporter/examples/config_with_filelog_receiver.yaml

All I want to do, is send a copy of all my logs, to a syslog endpoint. I am using the following config. I am a bit confused by the documentation how to properly configure the syslog_parser, where to configure it, and how to prevent it affecting my logs when going to sumologic.

Note also I am using the chart from https://github.com/SumoLogic/sumologic-kubernetes-collection for sending logs to sumologic, and using values.yml to add additional config:

  logs:
    enabled: true
    logLevel: info
    config:
      merge:
        extensions:
          file_storage/syslog:
            directory: /var/lib/storage/otc
            timeout: 10s
        exporters:
          #### syslog is in development - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/syslogexporter/README.md
          syslog:
            protocol: tcp
            port: 515
            tls:
              insecure: true
            endpoint: log.internal
            retry_on_failure:
              enabled: true
              initial_interval: 10s
              max_interval: 40s
              max_elapsed_time: 150s
            sending_queue:
              enabled: true
              num_consumers: 20
              storage: file_storage/syslog
              queue_size: 10000
        receivers:
          otlp/extrafiles:
            protocols:
              http:
                endpoint: 0.0.0.0:4319
            #### tried adding this here
            operators:
              - type: syslog_parser
                protocol: rfc5424
        service:
          extensions:
            - health_check
            - file_storage
            - pprof
            - file_storage/syslog
          pipelines:
            logs/otlp/containers:
              exporters:
                - sumologic/containers
                - syslog
            logs/otlp/kubelet:
              exporters:
                - sumologic/systemd
            logs/otlp/systemd:
              exporters:
                - sumologic/systemd
            logs/extrafiles:
              receivers: [otlp/extrafiles]
              processors:
                - memory_limiter
                - batch
              exporters:
                - sumologic/containers
                - syslog

This results in my syslog server receiving logs like this:

023-08-09T06:40:15.764305+00:00 ip-10-82-116-123.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:13Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:15.772269+00:00 ip-10-82-116-123.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:14Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:16.223312+00:00 ip-10-82-116-123.ap-southeast-2.compute.internal %!d(<nil>) 1970-01-01T00:00:00Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:16.231277+00:00 ip-10-82-116-123.ap-southeast-2.compute.internal %!d(<nil>) 1970-01-01T00:00:00Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:16.743542+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:13Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:16.748156+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:15Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:16.753788+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:15Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:16.758331+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:15Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:16.765639+00:00 ip-10-82-116-123.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:14Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:17.224420+00:00 ip-10-82-116-123.ap-southeast-2.compute.internal %!d(<nil>) 1970-01-01T00:00:00Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:17.232145+00:00 ip-10-82-116-123.ap-southeast-2.compute.internal %!d(<nil>) 1970-01-01T00:00:00Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:17.240228+00:00 ip-10-82-116-123.ap-southeast-2.compute.internal %!d(<nil>) 1970-01-01T00:00:00Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:17.484981+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 1970-01-01T00:00:00Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:17.490019+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 1970-01-01T00:00:00Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:17.742741+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:14Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:17.747759+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:16Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:17.753258+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:16Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:17.758243+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:16Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)
2023-08-09T06:40:17.763719+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d(<nil>) 2023-08-09T06:40:16Z %!s(<nil>) %!s(<nil>) %!s(<nil>) %!s(<nil>) - %!s(<nil>)

And I actually see an error in one of the pods like:

error decoding 'receivers': error reading configuration for "otlp/extrafiles": 1 error(s) decoding:

* '' has invalid keys: operators

Expected Result

Logs are received with proper content,

Actual Result

As above.

Collector version

0.79.0

Environment information

Environment

Kubernetes running on AWS

OpenTelemetry Collector configuration

As above

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners for exporter/syslog: @kkujawa-sumo @rnishtala-sumo @astencel-sumo. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[sumo-drosiek]

@captainfalcon23

And I actually see an error in one of the pods like:

operators cannot be used along with otlp receiver. If you want to receive logs with otlp and then parse it, you need to use logstransformprocessor which supports operators.

If you want to receive syslog logs, you can use syslogreceiver or (tcplogreceiver and/or udplogreceiver)

I'm going to work on reproduction and fix for the bug, you mentioned in the title:

syslog_parser with syslog exporter results in logs like 2023-08-09T06:34:49.490942+00:00 ip-10-82-103-213.ap-southeast-2.compute.internal %!d() 2023-08-09T06:34:48Z %!s() %!s() %!s() %!s() - %!s()

[sumo-drosiek]

Also, Sumo Logic distribution doesn't use the upstreamed version yet:

SumoLogic/sumologic-otel-collector#1200
https://github.com/SumoLogic/sumologic-otel-collector/blob/b869634d22bbcc2a5fc6003511a0bb4d8f746151/otelcolbuilder/.otelcol-builder.yaml#L17-L18

so this issue should be created in https://github.com/sumologic/sumologic-kubernetes-collection or https://github.com/SumoLogic/sumologic-otel-collector/

[captainfalcon23]

Interesting...I didn't realise Sumologic has their own forked version of the collector.

I actually ended up just completely changing my solution to using a "otlphttp" exporter, and then on the receiving side, I am using vector.dev as the receiver which offers a significant amount of flexibility and is lightning fast.

Personally, happy for you to either close this issue, or fix it, but it likely won't impact me personally moving forward :)

[sumo-drosiek]

I didn't realise Sumologic has their own forked version of the collector.

We developed syslog exporter and we started moving it to upstream. I'm working on finalizing this and going to fix this issue as well

[crobert-1]

It looks like #27424 fixed this issue, is that correct? If so we can go ahead and close it.

[sumo-drosiek]

@captainfalcon23 I would like to add documentation with examples and explanation, as it is not obvious how to prepare data to replace defaults, especially message content

[sumo-drosiek]

Pr with documentation: #27831

[crobert-1]

@sumo-drosiek Now that #27831 has been merged are we okay to close this, or are there more documentation items left?

[sumo-drosiek]

I'm good, we can close it 🎉

[crobert-1]

Fixed by #27424 and #27831