Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add doc about configuring TLS with Helm #4328

Merged
merged 8 commits into from
Nov 18, 2024
Merged

Add doc about configuring TLS with Helm #4328

Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1343a8e
Add doc about configuring TLS with Helm
zalegrala Nov 14, 2024
3d0ec43
Add memberlist and readinessProbe to example
zalegrala Nov 14, 2024
fde0b0c
Include server config for listening on TLS
zalegrala Nov 14, 2024
b0bf9ef
Add note about scraping
zalegrala Nov 14, 2024
3737236
Update docs/sources/tempo/configuration/network/tls.md
zalegrala Nov 15, 2024
56cb21e
Update docs/sources/tempo/configuration/network/tls.md
zalegrala Nov 15, 2024
f813be8
Update docs/sources/tempo/configuration/network/tls.md
zalegrala Nov 15, 2024
56b40e6
Add memcached config for TLS
zalegrala Nov 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 126 additions & 2 deletions docs/sources/tempo/configuration/network/tls.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ memberlist:

### Receiver TLS

Additional receiver configuration can be added to support TLS communication for traces being sent to Tempo. The receiver configuration is pulled in from the Open Telemetry collector, and is [documented upstream here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver/config.md#configtls-tlsserversetting).
Additional receiver configuration can be added to support TLS communication for traces being sent to Tempo. The receiver configuration is pulled in from the Open Telemetry collector, and is [documented upstream here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver/config.md#configtls-tlsserversetting). Addition TLS configuration of OTEL components can be found [here](https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/configtls).

An example `tls` block might look like the following:

Expand All @@ -86,7 +86,7 @@ tls:
ca_file: /tls/ca.crt
cert_file: /tls/tls.crt
key_file: /tls/tls.key
min_version: VersionTLS12
min_version: "1.2"
```

The above structure can be set on the following receiver configurations:
Expand All @@ -96,3 +96,127 @@ The above structure can be set on the following receiver configurations:
- `distributor.receivers.zipkin.tls`
- `distributor.receivers.jaeger.protocols.grpc.tls`
- `distributor.receivers.jaeger.protocols.thrift_http.tls`

### Configure TLS with Helm

To configure TLS with the Helm chart, you must have a TLS key-pair and CA certificate stored in a Kubernetes secret. The following example mounts a secret called `tempo-distributed-tls` into the pods at `/tls` and modifies the configuration of Tempo to make use of the files. In this example the Tempo components share a single TLS certificate. Note that the `tls_server_name` configuration must match the certificate.
zalegrala marked this conversation as resolved.
Show resolved Hide resolved

```yaml
compactor:
extraVolumeMounts:
- mountPath: /tls
name: tempo-distributed-tls
extraVolumes:
- name: tempo-distributed-tls
secret:
secretName: tempo-distributed-tls
distributor:
extraVolumeMounts:
- mountPath: /tls
name: tempo-distributed-tls
extraVolumes:
- name: tempo-distributed-tls
secret:
secretName: tempo-distributed-tls
ingester:
extraVolumeMounts:
- mountPath: /tls
name: tempo-distributed-tls
extraVolumes:
- name: tempo-distributed-tls
secret:
secretName: tempo-distributed-tls
metricsGenerator:
extraVolumeMounts:
- mountPath: /tls
name: tempo-distributed-tls
extraVolumes:
- name: tempo-distributed-tls
secret:
secretName: tempo-distributed-tls
querier:
extraVolumeMounts:
- mountPath: /tls
name: tempo-distributed-tls
extraVolumes:
- name: tempo-distributed-tls
secret:
secretName: tempo-distributed-tls
queryFrontend:
extraVolumeMounts:
- mountPath: /tls
name: tempo-distributed-tls
extraVolumes:
- name: tempo-distributed-tls
secret:
secretName: tempo-distributed-tls
tempo:
readinessProbe:
httpGet:
scheme: HTTPS
structuredConfig:
Copy link
Contributor

@markustoivonen markustoivonen Nov 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this section should also have the memcached tls config, or a mention of it in the written part

cache:
  caches:
    - memcached:
      <tls_config>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works and I'll include a bit of config for it in the example. The exporter is a little more work due to the way the annotations are handled, so I'm going to leave that out of this PR.

memberlist:
tls_ca_path: /tls/ca.crt
tls_cert_path: /tls/tls.crt
tls_enabled: true
tls_key_path: /tls/tls.key
tls_server_name: t-dist.tempo-distributed.svc.cluster.znet
zalegrala marked this conversation as resolved.
Show resolved Hide resolved
distributor:
receivers:
otlp:
protocols:
grpc:
tls:
ca_file: /tls/ca.crt
cert_file: /tls/tls.crt
key_file: /tls/tls.key
ingester_client:
grpc_client_config:
tls_ca_path: /tls/ca.crt
tls_cert_path: /tls/tls.crt
tls_enabled: true
tls_key_path: /tls/tls.key
tls_server_name: tempo-distributed.trace.svc.cluster.local
metrics_generator_client:
grpc_client_config:
tls_ca_path: /tls/ca.crt
tls_cert_path: /tls/tls.crt
tls_enabled: true
tls_key_path: /tls/tls.key
tls_server_name: tempo-distributed.trace.svc.cluster.local
querier:
frontend_worker:
grpc_client_config:
tls_ca_path: /tls/ca.crt
tls_cert_path: /tls/tls.crt
tls_enabled: true
tls_key_path: /tls/tls.key
tls_server_name: tempo-distributed.trace.svc.cluster.local
Comment on lines +218 to +225
Copy link
Contributor

@markustoivonen markustoivonen Nov 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This client is the only one that supports two ways of setting TLS. Through this method, but also through here. If you see value in unification, I would remove the latter option since at least for me it just caused confusion when setting up TLS (given, back then I didn't know of the structuredConfig option)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds reasonable to me. Would you like to PR for this? If not I can follow up. We'll want to mark it a breaking change, and validate that the structuredConfig is the right approach in all cases. Wouldn't mind at least other eyes on that portion.

server:
grpc_tls_config:
cert_file: /tls/tls.crt
client_auth_type: VerifyClientCertIfGiven
client_ca_file: /tls/ca.crt
key_file: /tls/tls.key
http_tls_config:
cert_file: /tls/tls.crt
client_auth_type: VerifyClientCertIfGiven
client_ca_file: /tls/ca.crt
key_file: /tls/tls.key
traces:
otlp:
grpc:
enabled: true
```

See the [`prometheus.scrape` docs for Alloy](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.scrape/) to configure TLS on the scrape. A relabel configuration like the following will do this configuration for you dynamically.
zalegrala marked this conversation as resolved.
Show resolved Hide resolved

```json
{
source_labels: ['__meta_kubernetes_pod_annotation_prometheus_io_scheme'],
action: 'replace',
target_label: '__scheme__',
regex: '(https?)',
replacement: '$1',
},
```
Loading