-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[fix][cpp] Support retry and apply operation timeout for lookup requests #17410
Merged
Demogorgon314
merged 3 commits into
apache:master
from
BewareMyPower:bewaremypower/cpp-lookup-timeout
Sep 15, 2022
Merged
[fix][cpp] Support retry and apply operation timeout for lookup requests #17410
Demogorgon314
merged 3 commits into
apache:master
from
BewareMyPower:bewaremypower/cpp-lookup-timeout
Sep 15, 2022
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BewareMyPower
added
type/bug
The PR fixed a bug or issue reported a bug
component/client-c++
labels
Sep 1, 2022
BewareMyPower
requested review from
ivankelly,
merlimat,
rdhabalia,
jiazhai,
gaoran10,
RobertIndie and
Demogorgon314
September 1, 2022 17:04
@BewareMyPower Please provide a correct documentation label for your PR. |
github-actions
bot
added
doc-not-needed
Your PR changes do not impact docs
and removed
doc-label-missing
labels
Sep 1, 2022
BewareMyPower
force-pushed
the
bewaremypower/cpp-lookup-timeout
branch
from
September 13, 2022 08:52
a4ea7b8
to
765fb3b
Compare
### Motivation Currently the operation timeout only works for requests other than lookup, like SEND and FLOW. However, the lookup requests, which are sent by `LookupService`, should also apply the operation timeout, see https://github.com/apache/pulsar/blob/7075a5ce0d4a70f52625ac8c3d0c48894442b72a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java#L1019-L1025 In addition, no attempts would be retried if lookup failed even due to a retryable reason. For example, if some of all configured brokers were down, the C++ client would fail immediately. Therefore, this PR intends to retry lookup for some certain cases: - The connection cannot be established, except connection timeout, which is controlled by the connection timeout. - A `ServiceNotReady` error is received, except it's caused by `PulsarServerException`, e.g. the listener name is wrong. Then, apply the operation timeout to avoid infinite retries. ### Modifications - Add a `ResultRetryable` error code, which should only be used internally. Complete the futures with this error in the cases said previously. - Add a `RetryableLookupService` implementation to support retries based on the backoff policy. Replace the directly usages of `LookupService` implementations with this class in `ClientImpl`. The following tests are added: - `ClientTest.testMultiBrokerUrl`: verify when multiple brokers are configured, even if one of them is not available, the creation of producer or consumer could still succeed. - `LookupService.testRetry`: verify all lookup methods could be retried. - `LookupService.testTimeout`: verify all lookup methods could be completed with `ResultTimeout` if no brokers are available. ### TODO In future, we should add lookup timeout instead of operation timeout for lookup requests and separate lookup connection pool, see PIP-91.
BewareMyPower
force-pushed
the
bewaremypower/cpp-lookup-timeout
branch
from
September 13, 2022 14:21
765fb3b
to
191cbaa
Compare
shibd
reviewed
Sep 14, 2022
shibd
reviewed
Sep 14, 2022
shibd
reviewed
Sep 14, 2022
shibd
approved these changes
Sep 15, 2022
merlimat
approved these changes
Sep 15, 2022
Demogorgon314
approved these changes
Sep 15, 2022
BewareMyPower
added a commit
that referenced
this pull request
Sep 15, 2022
…sts (#17410) ### Motivation Currently the operation timeout only works for requests other than lookup, like SEND and FLOW. However, the lookup requests, which are sent by `LookupService`, should also apply the operation timeout, see https://github.com/apache/pulsar/blob/7075a5ce0d4a70f52625ac8c3d0c48894442b72a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java#L1019-L1025 In addition, no attempts would be retried if lookup failed even due to a retryable reason. For example, if some of all configured brokers were down, the C++ client would fail immediately. Therefore, this PR intends to retry lookup for some certain cases: - The connection cannot be established, except connection timeout, which is controlled by the connection timeout. - A `ServiceNotReady` error is received, except it's caused by `PulsarServerException`, e.g. the listener name is wrong. Then, apply the operation timeout to avoid infinite retries. ### Modifications - Add a `ResultRetryable` error code, which should only be used internally. Complete the futures with this error in the cases said previously. - Add a `RetryableLookupService` implementation to support retries based on the backoff policy. Replace the directly usages of `LookupService` implementations with this class in `ClientImpl`. The following tests are added: - `ClientTest.testMultiBrokerUrl`: verify when multiple brokers are configured, even if one of them is not available, the creation of producer or consumer could still succeed. - `LookupService.testRetry`: verify all lookup methods could be retried. - `LookupService.testTimeout`: verify all lookup methods could be completed with `ResultTimeout` if no brokers are available. ### TODO In future, we should add lookup timeout instead of operation timeout for lookup requests and separate lookup connection pool, see PIP-91. * Handle ResultRetryable in handleDisconnection (cherry picked from commit 86ea31b)
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
cherry-picked/branch-2.11
doc-not-needed
Your PR changes do not impact docs
release/2.11.0
type/bug
The PR fixed a bug or issue reported a bug
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Currently the operation timeout only works for requests other than
lookup, like SEND and FLOW. However, the lookup requests, which are sent
by
LookupService
, should also apply the operation timeout, seepulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java
Lines 1019 to 1025 in 7075a5c
In addition, no attempts would be retried if lookup failed even due to a
retryable reason. For example, if some of all configured brokers were
down, the C++ client would fail immediately.
Therefore, this PR intends to retry lookup for some certain cases:
is controlled by the connection timeout.
ServiceNotReady
error is received, except it's caused byPulsarServerException
, e.g. the listener name is wrong.Then, apply the operation timeout to avoid infinite retries.
Modifications
ResultRetryable
error code, which should only be usedinternally. Complete the futures with this error in the cases said
previously.
RetryableLookupService
implementation to support retries basedon the backoff policy. Replace the directly usages of
LookupService
implementations with this class in
ClientImpl
.The following tests are added:
ClientTest.testMultiBrokerUrl
: verify when multiple brokers areconfigured, even if one of them is not available, the creation of
producer or consumer could still succeed.
LookupService.testRetry
: verify all lookup methods could be retried.LookupService.testTimeout
: verify all lookup methods could becompleted with
ResultTimeout
if no brokers are available.TODO
In future, we should add lookup timeout instead of operation timeout for
lookup requests and separate lookup connection pool, see PIP-91.
Documentation
Check the box below or label this PR directly.
Need to update docs?
doc-required
(Your PR needs to update docs and you will update later)
doc-not-needed
(Please explain why)
doc
(Your PR contains doc changes)
doc-complete
(Docs have been already added)