[run-tests] support setting bundle number for default namespace when set up cluster #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 tasks
### Motivation in the schema update, will create a `ledgerHandle` and write data to BK, after that `ledgerHandle` is no longer useful and no other object holds references to it. `ledgerHandle` will be recycled with GC, but `ledgerHandle` also hold external connections, which will cause leakage. https://github.com/apache/pulsar/blob/40b9d7ea50cef54becb09f2543193e08375abe0b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java#L452-L456 ### Modifications after the schema is updated, close the `ledgerHandle`, just like schema-read: https://github.com/apache/pulsar/blob/40b9d7ea50cef54becb09f2543193e08375abe0b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java#L519-L525
…hapter (apache#17615) * Sidebar re-org * Separate TLS encryption/authentication using Keystore to parent topics * use language-specific tabs to show code snippets * streamline headings * Update client-libraries-java.md * Add code snippet for python clients to support apache#17482 * Add code snippet for go clients
…ception` (apache#17700) ### Motivation In the production environment, we found two brokers holding the same valid locks. and one has an exceptional revalidate future with `lockBusyException`. after reading the code, there may forget the reset the cache and complete expire exception when getting lockBusyException.
… cookbook (apache#17666) * remove duplicate file `cookbooks-encryption.md` * Revert "remove duplicate file `cookbooks-encryption.md`" This reverts commit a1e5a54. * streamline encryption cookbook and make it single-sourced * Add code snippets for Java/C++/Go clients. * use one svg image to replace two jpg files * client version and title updates * improve security overview * Add note and link for TLS encryption
…spatchThrottlingTest (apache#17702)
…ation (apache#16917) * [improve][txn][PIP-196] Segmented transaction buffer snapshot Master apache#16913 ### Motivation 1. Add configurations. 2. Implement the system topic client for snapshot topic and index topic. 3. Implement AbortedTxnProcessor. 4. Add transaction buffer snapshot metrics. ### Modification Add configuration for segment snaopshot.
…on (apache#16003) ### Motivation Now, the MongoDB source connector only supports the incremental message synchronization. This PR adds support for full message synchronization. Since MongDB 4.0, we can set the starting point for the change stream by the `startAtOperationTime` field. So, we can set it to `0` to make start point the earliest. See https://www.mongodb.com/docs/v4.2/reference/method/db.collection.watch/ for more information.
### Motivation The current `bin/pulsar version` script only prints out the version string. When running custom builds of Pulsar, this is sometimes not enough to know which version you have. Since the `PulsarVersion` class already has more information, I propose we print out all of its information in the same format that the broker logs the information. Here is a sample output for this specific PR: ```shell $ bin/pulsar version Current version of pulsar is: 2.11.0-SNAPSHOT Git Revision 4414e3d Git Branch print-more-version-info Built by Michael Marshall <[email protected]> on mmarshall-rmbp16 at 2020-01-22T09:10:15-0600 ``` ### Modifications * Add more output to the `bin/pulsar version` command. ### Verifying this change This change is a trivial rework / code cleanup without any test coverage. ### Documentation - [x] `doc-not-needed` (Please explain why) ### Matching PR in forked repository PR in forked repository: michaeljmarshall#3
…pache#17743) - when setup is called explicitly, cleanup should be issued before it - remove unnecessary "resetConfig" method calls - resetConfig gets called as part of internalCleanup - when resetConfig is in cleanup, it's possible to override config before calling setup - this allows using setup & cleanup methods instead of using error prone internalSetup and internalCleanup methods which might leave something behind
…mes topics in one broker (apache#17526)
* fix entryFilter stats * fix test * add test comment * review fix
…ed (apache#17704) * [fix][metrics]wrong metrics text generated when label_cluster specified * improve logic branch * mark test group
Signed-off-by: tison <[email protected]> Signed-off-by: tison <[email protected]>
… been published after the topic gets activated on a broker (apache#16618) * Skip creating a replication snapshot if no messages have been published * Adapt test to new behavior where replication snapshots happen only when there are new messages
* Call cleanup method in finally block to ensure it's not skipped * Clear invocations for the mocks that are left around without cleanup * Cleanup PulsarService and PulsarAdmin mocks/spies in MockedPulsarServiceBaseTest * Don't record invocations at all for PulsarService and PulsarAdmin in MockedPulsarServiceBaseTest * Don't record invocations for spies by default * Simplify reseting mocks * Fix PersistentTopicTest * Fix TokenExpirationProducerConsumerTest * Fix SimpleLoadManagerImplTest * Fix FilterEntryTest
…he#17834) - use Bookkeeper defaults by setting BK_METADATA_OPTIONS=none
…7209) Fixes apache#17186 ### Motivation There are some cases in which it is useful to be able to include current position of the message when reset of cursor was made. ### Modifications * Support inclusive seek in c++ consumers. * Add a unit test to verify.
* fix: delete sqlite files after jdbc connection closed This closes apache#17713. Signed-off-by: tison <[email protected]> * uses isolated db file Signed-off-by: tison <[email protected]> * Revert "uses isolated db file" This reverts commit 295db3c. * close in order Signed-off-by: tison <[email protected]> * strong order guarantee Signed-off-by: tison <[email protected]> * factor out defer logic to avoid further bugs Signed-off-by: tison <[email protected]> * Revert "factor out defer logic to avoid further bugs" This reverts commit f7f4634. * Revert "strong order guarantee" This reverts commit 747086f. * use awaitTermination Signed-off-by: tison <[email protected]> Signed-off-by: tison <[email protected]>
…AndCommitForTransaction (apache#17845) * scenario is already covered by PendingAckPersistentTest
…#17691) Co-authored-by: Marvin Cai <[email protected]>
…ManagedLedgerImpl (apache#17293) - a NPE with no description is confusing
…on time (apache#17790) Fixes - apache#17623 - apache#17637 ### Motivation Manually release resources, including `consumer`, `producer`, `pulsar client`, `transaction`, and `topic`. This saves `setup` and `cleanup` time before and after each method. ### Modifications - Manually release resources instead of calling `cleanup` & `setup` each method - remove useless method `markDeletePositionCheck` - `Integer.valueOf(int)` instead of `new Integer(int)`, because `new Integer(int)` is deprecated ### Matching PR in forked repository PR in forked repository: - poorbarcode#10
Fixes apache#17785 ### Motivation The `failureMap` need to be clear after run per unit test. ### Modifications Clear `failureMap` after run per unit test, and only run once `setup()`/`cleanup()` to reduce execution time. ### Matching PR in forked repository PR in forked repository: coderzc#6
…ache#17252) - fixes issue with stats where timestamps might be inconsistent because of visibility issues - fields should be volatile to ensure visibility of updated values in a consistent manner - in replication, the lastDataMessagePublishedTimestamp field in PersistentTopic might be inconsistent unless volatile is used
…he#16891) * add reader config doc * update to the versioned doc * Update site2/docs/io-debezium-source.md Co-authored-by: momo-jun <[email protected]> * Update site2/docs/io-debezium-source.md Co-authored-by: momo-jun <[email protected]> * revert changes to 2.10.1 and 2.9.3 Co-authored-by: momo-jun <[email protected]>
Signed-off-by: Zixuan Liu <[email protected]> Signed-off-by: Zixuan Liu <[email protected]> ### Motivation Improve the compactor tool, using separate TLS config ### Modifications - Add separate TLS config on the compactor, both Keystore and PEM formats are supported - Fix correct use of service URL by `brokerConfig.isBrokerClientTlsEnabled()` value ### Verifying this change Test has been added. ### Documentation Check the box below or label this PR directly. Need to update docs? - [ ] `doc-required` (Your PR needs to update docs and you will update later) - [x] `doc-not-needed` (Please explain why) - [ ] `doc` (Your PR contains doc changes) - [ ] `doc-complete` (Docs have been already added)
…ts (apache#17686) * [feat][broker] Add config to count filtered entries towards rate limits * Make fixes for checkstyle * Remove * import * Fix incorrect conflict resolution in merge commit ### Motivation Currently, when using entry filters, filtered out messages do not count against the rate limit. Therefore, a subscription that is completely filtered will never be throttled due to rate limiting. When the messages are delivered to the consumer for a filtered subscription, those messages will count against the rate limit, and in that case, the message filtering can be throttled because the check to delay `readMoreEntries()` happens before message filtering. Therefore, the rate limit will essentially be increased as a function of the percent of messages let through the filter (some quick math is that the new rate is likely `dispatchRate * (1 / percentDelivered)`, where percent delivered is a percent as a decimal). It's possible that some use cases prefer this behavior, but in my case, I think it'd be valuable to include these filtered messages in the dispatch throttling because these messages still cost the broker network, memory, and cpu. This PR adds a configuration to count filtered out messages towards dispatch rate limits for the broker, the topic, and the subscription. ### Modifications * Add configuration named `dispatchThrottlingForFilteredEntriesEnabled`. Default it to false so we maintain the original behavior. When true, count filtered messages against rate limits. * Refactor the code to `acquirePermitsForDeliveredMessages` so that it is in the `AbstractBaseDispatcher`, which makes it available to the entry filtering logic. ### Verifying this change A new test is added as part of this PR. ### Does this pull request potentially affect one of the following parts: This PR introduces a new config while maintaining the current behavior. ### Documentation - [x] `doc-not-needed` Config docs are auto-generated.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is for running tests for upstream PR apache#17722.