[Bug] Seek on a timestamp causes the subscription to be resetted to the earliest position.

[Gilthoniel]

Search before asking

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version

Pulsar v4.0.2
Pulsar Go client v0.14

Minimal reproduce step

With v4.0.2, after restarting brokers, we observe a wrong position in subscription after a seek by timestamp. The position seems to be the earliest position instead of the next message after the timestamp.

Note that after downgrading to v4.0.1, the seeking position gets back to the proper one so the message after the timestamp.

What did you expect to see?

On a seek by timestamp, we expect the subscription to reset to the next message after the timestamp available in the topic.

What did you see instead?

We observe subscriptions to reset on the earliest position when we use seek by timestamp.

May that be related to #22129 ?

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

@Gilthoniel Thanks for reporting this issue.

The change added in #22792 can be disabled by configuring managedLedgerCursorResetLedgerCloseTimestampMaxClockSkewMillis=-1. This configuration value doesn't exist in broker.conf. For k8s deployments, setting PULSAR_PREFIX_managedLedgerCursorResetLedgerCloseTimestampMaxClockSkewMillis: "-1" in the config map is a way to configure it.

Do you have a chance to test if the problem reproduces with 4.0.2 with this configuration?

[Gilthoniel]

@lhotari thanks, it indeed works properly with v4.0.2 when using -1 so that should confirm the change has introduced a regression.

[lhotari]

@lhotari thanks, it indeed works properly with v4.0.2 when using -1 so that should confirm the change has introduced a regression.

Thanks for confirming. I have updated the release notes with information about this regression and how to apply the workaround: https://github.com/apache/pulsar/releases/tag/v4.0.2 . It will soon be published in https://pulsar.apache.org/release-notes/versioned/pulsar-4.0.2/ .

Known Major Issues in this Release

This release contains a known issue impacting the seek method by timestamp. This issue was reported after the release was published.

• [Bug] Seeking by timestamp causes the subscription to reset to the earliest position (#23910)

Workaround for the issue:

The issue is caused by changes introduced in #22792. To resolve this, disable the functionality by setting the following configuration:

managedLedgerCursorResetLedgerCloseTimestampMaxClockSkewMillis=-1

Workaround Configuration Notes:

• This configuration key needs to be manually added to your broker.conf file as it doesn't exist by default
• For Apache Pulsar Helm Chart Kubernetes deployments, add the following to your values.yaml file under the broker/configData section:

broker:
  configData:
    PULSAR_PREFIX_managedLedgerCursorResetLedgerCloseTimestampMaxClockSkewMillis: "-1"