[kube-prometheus-stack] Many warning/error messages "Error on ingesting out-of-order samples" and "Error on ingesting samples with different value but same timestamp" after updating to new chart version.

[levirosal]

Describe the bug a clear and concise description of what the bug is.

I'm installing the new versions of the kube-prometheus-stack chart and I'm getting several warning messages about metrics with the same timestamp and errors about and error messages about Error on ingesting out-of-order samples. I did some tests and this only occurs in versions after the trackTimestampsStaleness flag was enabled. BTW in the PR that was added, these errors and warnings were mentioned.

Do you have an answer if this is a bug or if some configuration needs to be done in the metrics exporters / prometheus to work with this flag?

time=2025-01-21T04:52:14.768Z level=WARN source=scrape.go:1885 msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=serviceMonitor/observability-system/cloudwatch-exporter-prometheus-cloudwatch-exporter/0 target=http://172.30.9.19:9106/metrics num_dropped=13

What's your helm version?

3.1.1

What's your kubectl version?

Server Version: 1.31

Which chart?

kube-prometheus-stack

What's the chart version?

=66.6.0

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

kubelet:
  enabled: true
  namespace: kube-system
  serviceMonitor:
    trackTimestampsStaleness: false
    cAdvisorRelabelings:
      - action: replace
        sourceLabels: [__metrics_path__]
        targetLabel: metrics_path
      - targetLabel: cluster
        replacement: ${CLUSTER_NAME}
      - targetLabel: region
        replacement: ${AWS_REGION}
    probesRelabelings:
      - action: replace
        sourceLabels: [__metrics_path__]
        targetLabel: metrics_path
      - targetLabel: cluster
        replacement: ${CLUSTER_NAME}
      - targetLabel: region
        replacement: ${AWS_REGION}
    relabelings:
      - action: replace
        sourceLabels: [__metrics_path__]
        targetLabel: metrics_path
      - targetLabel: cluster
        replacement: ${CLUSTER_NAME}
      - targetLabel: region
        replacement: ${AWS_REGION}

Enter the command that you execute and failing/misfunctioning.

N/A

Anything else we need to know?

No response