Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(compression): Make the compression type configurable, default: gzip #3705

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat(compression): Make the compression type configurable, default: gzip #3705

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
661245d
feat(compression): Make the compression type configurable, default: zstd
rnishtala-sumo May 14, 2024
99b58e2
Regenerating golden files
rnishtala-sumo May 14, 2024
63a0454
Move the compression config to the top-level, change default to gzip
rnishtala-sumo May 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .changelog/3705.added.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
feat(compression): Make the compression type configurable, default: gzip
1 change: 1 addition & 0 deletions deploy/helm/sumologic/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,6 +432,7 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `otellogswindows.metrics.enabled` | Enable OpenTelemetry Logs Collector for Windows Nodes metrics | `true` |
| `otellogswindows.serviceLabels` | Add custom labels to OpenTelemetry Logs Collector for Windows Nodes Service | `{}` |
| `otellogswindows.additionalDaemonSets` | OpenTelemetry Logs Collector for Windows Nodes Daemonset per node customization options. See [Best Practices](https://help.sumologic.com/docs/send-data/kubernetes/best-practices/#setting-different-resources-on-different-nodes-for-logs-collector). | `{}` |
| `compression` | Compression can be gzip, snappy, zstd or deflate | `gzip` |
| `metadata.image.repository` | Image repository for otelcol docker container. | `` |
| `metadata.image.tag` | Image tag for otelcol docker container. | `` |
| `metadata.image.pullPolicy` | Image pullPolicy for otelcol docker container. | `IfNotPresent` |
Expand Down
2 changes: 2 additions & 0 deletions deploy/helm/sumologic/conf/events/otelcol/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ exporters:
sumologic:
client: {{ include "sumologic.sumo_client" . }}
endpoint: {{ include "sumologic.events.exporter.endpoint" . }}
compression: {{ .Values.compression }}
log_format: {{ include "sumologic.events.exporter.format" . }}
sending_queue:
enabled: true
Expand All @@ -18,6 +19,7 @@ exporters:
sumologic/sumologic-mock:
client: {{ include "sumologic.sumo_client" . }}
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.compression }}
log_format: {{ include "sumologic.events.exporter.format" . }}
sending_queue:
enabled: true
Expand Down
1 change: 1 addition & 0 deletions deploy/helm/sumologic/conf/logs/collector/otelcol/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
exporters:
otlphttp:
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.{{ .Values.sumologic.clusterDNSDomain }}.:4318
compression: {{ .Values.compression }}
sending_queue:
queue_size: 10
# this improves load balancing at the cost of more network traffic
Expand Down
6 changes: 6 additions & 0 deletions deploy/helm/sumologic/conf/logs/otelcol/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ exporters:
{{ if .Values.sumologic.logs.container.enabled }}
sumologic/containers:
endpoint: ${SUMO_ENDPOINT_DEFAULT_LOGS_SOURCE}
compression: {{ .Values.compression }}
log_format: {{ include "logs.otelcol.container.exporter.format" . }}
sending_queue:
enabled: true
Expand All @@ -20,6 +21,7 @@ exporters:
{{- if eq (include "sumologic-mock.forward-logs-metadata" .) "true" }}
sumologic/sumologic-mock-containers:
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.compression }}
log_format: {{ include "logs.otelcol.container.exporter.format" . }}
sending_queue:
enabled: true
Expand All @@ -35,6 +37,7 @@ exporters:
{{ if .Values.sumologic.logs.systemd.enabled }}
sumologic/systemd:
endpoint: ${SUMO_ENDPOINT_DEFAULT_LOGS_SOURCE}
compression: {{ .Values.compression }}
log_format: json
sending_queue:
enabled: true
Expand All @@ -47,6 +50,7 @@ exporters:
{{- if eq (include "sumologic-mock.forward-logs-metadata" .) "true" }}
sumologic/sumologic-mock-systemd:
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.compression }}
log_format: json
sending_queue:
enabled: true
Expand All @@ -63,6 +67,7 @@ exporters:
sumologic:
client: {{ include "sumologic.sumo_client" . }}
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_LOGS_SOURCE}
compression: {{ .Values.compression }}
log_format: otlp
sending_queue:
enabled: true
Expand All @@ -76,6 +81,7 @@ exporters:
sumologic/sumologic-mock:
client: {{ include "sumologic.sumo_client" . }}
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.compression }}
log_format: otlp
sending_queue:
enabled: true
Expand Down
1 change: 1 addition & 0 deletions deploy/helm/sumologic/conf/metrics/collector/otelcol/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ exporters:
{{- end }}
otlphttp:
endpoint: http://${METADATA_METRICS_SVC}.${NAMESPACE}.svc.{{ .Values.sumologic.clusterDNSDomain }}.:4318
compression: {{ .Values.compression }}
sending_queue:
queue_size: 10000
num_consumers: 10
Expand Down
10 changes: 10 additions & 0 deletions deploy/helm/sumologic/conf/metrics/otelcol/exporters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ sumologic/sumologic-mock-default:
client: {{ include "sumologic.sumo_client" . }}
metric_format: {{ include "metrics.otelcol.exporter.format" . }}
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.compression }}
## Sumo doesn't yet natively support OTLP Histograms
decompose_otlp_histograms: true
## Configuration for sending queue
Expand All @@ -29,6 +30,7 @@ sumologic/sumologic-mock-http:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -47,6 +49,7 @@ sumologic/default:
client: {{ include "sumologic.sumo_client" . }}
metric_format: {{ include "metrics.otelcol.exporter.format" . }}
endpoint: {{ include "metrics.otelcol.exporter.endpoint" . }}
compression: {{ .Values.compression }}
## Sumo doesn't yet natively support OTLP Histograms
decompose_otlp_histograms: true
## Configuration for sending queue
Expand All @@ -68,6 +71,7 @@ sumologic/apiserver:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_APISERVER_METRICS_SOURCE}
compression: {{ .Values.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -82,6 +86,7 @@ sumologic/control_plane:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_CONTROL_PLANE_METRICS_SOURCE}
compression: {{ .Values.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -96,6 +101,7 @@ sumologic/controller:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_CONTROLLER_METRICS_SOURCE}
compression: {{ .Values.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -110,6 +116,7 @@ sumologic/kubelet:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_KUBELET_METRICS_SOURCE}
compression: {{ .Values.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -124,6 +131,7 @@ sumologic/node:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_NODE_METRICS_SOURCE}
compression: {{ .Values.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -138,6 +146,7 @@ sumologic/scheduler:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_SCHEDULER_METRICS_SOURCE}
compression: {{ .Values.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -152,6 +161,7 @@ sumologic/state:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_STATE_METRICS_SOURCE}
compression: {{ .Values.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand Down
5 changes: 5 additions & 0 deletions deploy/helm/sumologic/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ fullnameOverride: ""
## Use the same namespace as namespaceOverride in 'kube-prometheus-stack.namespaceOverride' if Prometheus setup is also enabled
namespaceOverride: ""

## Compression for logs, metrics and events
## Compression types can be gzip, snappy, zstd or deflate
## Default is gzip
compression: gzip

sumologic:
### Setup

Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/events_otc/basic.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_EVENTS_SOURCE}
log_format: otlp
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/events_otc/options.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_EVENTS_SOURCE}
log_format: otlp
sending_queue:
Expand Down
2 changes: 2 additions & 0 deletions tests/helm/testdata/goldenfile/events_otc/sumologic-mock.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,15 @@ data:
verbosity: detailed
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_EVENTS_SOURCE}
log_format: otlp
sending_queue:
enabled: true
storage: file_storage
sumologic/sumologic-mock:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: http://RELEASE-NAME-sumologic-mock.sumologic.svc.cluster.local.:3000/receiver
log_format: otlp
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/logs_otc/basic.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ data:
config.yaml: |
exporters:
otlphttp:
compression: gzip
disable_keep_alives: true
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.cluster.local.:4318
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/logs_otc/debug.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ data:
debug:
verbosity: detailed
otlphttp:
compression: gzip
disable_keep_alives: true
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.cluster.local.:4318
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/logs_otc/options.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ data:
config.yaml: |
exporters:
otlphttp:
compression: gzip
disable_keep_alives: true
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.cluster.local.:4318
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/logs_otc_daemonset/multiple_multiline.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ data:
config.yaml: |
exporters:
otlphttp:
compression: gzip
disable_keep_alives: true
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.cluster.local.:4318
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_logs_otc/debug.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ data:
verbosity: detailed
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_LOGS_SOURCE}
log_format: otlp
sending_queue:
Expand Down
2 changes: 2 additions & 0 deletions tests/helm/testdata/goldenfile/metadata_logs_otc/debug_with_sumologic_mock.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ data:
verbosity: detailed
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_LOGS_SOURCE}
log_format: otlp
sending_queue:
Expand All @@ -26,6 +27,7 @@ data:
storage: file_storage
sumologic/sumologic-mock:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: http://RELEASE-NAME-sumologic-mock.sumologic.svc.cluster.local.:3000/receiver
log_format: otlp
sending_queue:
Expand Down
4 changes: 4 additions & 0 deletions tests/helm/testdata/goldenfile/metadata_logs_otc/debug_with_sumologic_mock_http.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ data:
debug:
verbosity: detailed
sumologic/containers:
compression: gzip
endpoint: ${SUMO_ENDPOINT_DEFAULT_LOGS_SOURCE}
log_format: json
sending_queue:
Expand All @@ -24,6 +25,7 @@ data:
queue_size: 10000
storage: file_storage
sumologic/sumologic-mock-containers:
compression: gzip
endpoint: http://RELEASE-NAME-sumologic-mock.sumologic.svc.cluster.local.:3000/receiver
log_format: json
sending_queue:
Expand All @@ -32,6 +34,7 @@ data:
queue_size: 10000
storage: file_storage
sumologic/sumologic-mock-systemd:
compression: gzip
endpoint: http://RELEASE-NAME-sumologic-mock.sumologic.svc.cluster.local.:3000/receiver
log_format: json
sending_queue:
Expand All @@ -40,6 +43,7 @@ data:
queue_size: 10000
storage: file_storage
sumologic/systemd:
compression: gzip
endpoint: ${SUMO_ENDPOINT_DEFAULT_LOGS_SOURCE}
log_format: json
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_logs_otc/otel.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_LOGS_SOURCE}
log_format: otlp
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_logs_otc/templates.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_LOGS_SOURCE}
log_format: otlp
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_metrics_otc/additional_endpoints.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic/default:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
decompose_otlp_histograms: true
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_METRICS_SOURCE}
max_request_body_size: 16777216
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_metrics_otc/basic.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic/default:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
decompose_otlp_histograms: true
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_METRICS_SOURCE}
max_request_body_size: 16777216
Expand Down
8 changes: 8 additions & 0 deletions tests/helm/testdata/goldenfile/metadata_metrics_otc/custom.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic/apiserver:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_APISERVER_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -26,6 +27,7 @@ data:
timeout: 30s
sumologic/control_plane:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_CONTROL_PLANE_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -37,6 +39,7 @@ data:
timeout: 30s
sumologic/controller:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_CONTROLLER_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -48,6 +51,7 @@ data:
timeout: 30s
sumologic/default:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
decompose_otlp_histograms: true
endpoint: ${SUMO_ENDPOINT_DEFAULT_METRICS_SOURCE}
max_request_body_size: 16777216
Expand All @@ -60,6 +64,7 @@ data:
timeout: 30s
sumologic/kubelet:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_KUBELET_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -71,6 +76,7 @@ data:
timeout: 30s
sumologic/node:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_NODE_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -82,6 +88,7 @@ data:
timeout: 30s
sumologic/scheduler:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_SCHEDULER_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -93,6 +100,7 @@ data:
timeout: 30s
sumologic/state:
client: k8s_%CURRENT_CHART_VERSION%
compression: gzip
endpoint: ${SUMO_ENDPOINT_STATE_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand Down
Loading