Releases · apache/pulsar

Enhanced Key_Shared subscription implementation
Secure Docker runtime based on Alpine Linux and Java 21
Enhanced Quality of Service controls for multi-tenant deployments
Rate limiting as a foundation for multi-tenant resource management
Upgrade paths from previous versions
Client compatibility and Importance of upgrading Java clients to address CVE-2024-47561

PIPs accepted since 3.0.0 release

PIPs accepted in 4.0.0 release

PIP-327: Support force topic loading for unrecoverable errors (#21752)
PIP-337: SSL Factory Plugin to customize SSL Context and SSL Engine generation (#22016)
PIP-347: add role field in consumer's stat (#22564)
PIP-348: Trigger offload on topic load stage (#22650)
PIP-349: Add additionalSystemCursorNames ignore list for TTL check (#22651)
PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis (#22688)
PIP-351: Additional options for Pulsar-Test client to support KeyStore based TLS (#22694)
PIP-352: Event time based topic compactor (#22710)
PIP-353: Improve transaction message visibility for peek-messages (#22746)
PIP-354: apply topK mechanism to ModularLoadManagerImpl (#22765)
PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22778)
PIP-356: Support Geo-Replication starts at earliest position (#22806)
PIP-357: Correct the conf name in load balance module. (#22823)
PIP-358: let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. (#22889)
PIP-359: Support custom message listener executor for specific subscription (#22902)
PIP-360: Add admin API to display Schema metadata (#22913)
PIP-363: Add callback parameters to the method: org.apache.pulsar.client.impl.SendCallback.sendComplete. (#22940)
PIP-364: Introduce a new load balance algorithm AvgShedder (#22946)
PIP-366: Support to specify different config for Configuration and Local Metadata Store (#23033)
PIP-367: Propose a Contributor Repository for Pulsar (#23061)
PIP-368: Support lookup based on the lookup properties (#23075)
PIP-369: Flag based selective unload on changing ns-isolation-policy (#23116)
PIP-370: configurable remote topic creation in geo-replication (#23124)
PIP-374: Visibility of messages in receiverQueue for the consumers (#23235)
PIP-376: Make topic policies service pluggable (#23248)
PIP-378: Add ServiceUnitStateTableView abstraction (ExtensibleLoadMangerImpl only) (#23300)
PIP-379: Key_Shared Draining Hashes for Improved Message Ordering (#23309)
- Documentation: Preserving order of message delivery by key
PIP-381: Handle large PositionInfo state (#23328)
PIP-383: Support granting/revoking permissions for multiple topics (#23355)
PIP-384: ManagedLedger interface decoupling (#23363)

PIPs accepted in 3.3.0 release

PIP-315: Configurable max delay limit for delayed delivery (#21798)
PIP-321: Introduce allowed-cluster at the namespace level (#21648)
PIP-324: Alpine Docker images (#21716)
PIP-325: Add command to abort transaction (#21731)
PIP-326: Create a BOM to ease dependency management (#21747)
PIP-329: Strategy for maintaining the latest tag to Pulsar docker images (#21872)
PIP-330: getMessagesById gets all messages (#21873)
PIP-342: Support OpenTelemetry metrics in Pulsar client (#22178)
PIP-343: Use picocli instead of jcommander (#22181)
PIP-344: Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) (#22182)
PIP-335: Oxia metadata plugin (#22009)
PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources (#22071)
PIP-282: Change definition of the recently joined consumers position (#20776)

PIPs accepted in 3.2.0 release

PIP-323: Complete Backlog Quota Telemetry (#21709)
PIP-322: Pulsar Rate Limiting Refactoring (#21680)
PIP-320: OpenTelemetry Scaffolding (#21635)
PIP-318: Don't retain null-key messages during topic compaction (#21541)
PIP-315: Configurable max delay limit for delayed del...

Assets 2

04 Oct 18:51

lhotari

v3.3.2

f547100

v3.3.2

2024-10-04

Library updates

[fix][sec] Upgrade Avro to 1.11.4 to address CVE-2024-47561 (#23394)
[fix][sec] Upgrade vertx to 4.5.10 to address CVE-2024-8391 (#23338)
[fix][sec][branch-3.3] Upgrade protobuf-java to 3.25.5 (#23356) (#23358)
[improve][misc] Upgrade Netty to 4.1.113 and netty-tcnative to 2.0.66 (#23255)
[fix] Bump commons-io:commons-io from 2.8.0 to 2.14.0 (#23393)
[fix] Bump io.grpc from 1.56.0 to 1.56.1 (#23276)
[fix][metadata] Upgrade Oxia to 0.3.2 (#23140)
[feat][meta] Bump oxia java version from 0.3.2 to 0.4.5 (#23277)
[improve] Upgrade Pulsar Python client in docker image to 3.5.0 (#23377)
[improve] Install openssl in the docker image to fix compatibility with Apache Pulsar Helm chart (#23362)

Broker

[fix][broker] Cancel possible pending replay read in cancelPendingRead (#23384)
[fix][broker] Execute the pending callbacks in order before ready for incoming requests (#23266)
[fix][broker] Fail fast if the extensible load manager failed to start (#23297)
[fix][broker] Fix 'Disabled replicated subscriptions controller' logic and logging (#23142)
[fix][broker] Fix authenticate order in AuthenticationProviderList (#23111)
[fix][broker] Fix brokers still retry start replication after closed the topic (#23237)
[fix][broker] Fix compatibility issues for PIP-344 (#23136)
[fix][broker] Fix incomplete NAR file extraction which prevents broker from starting (#23274)
[fix][broker] Fix out-of-order issues with ConsistentHashingStickyKeyConsumerSelector (#23327)
[fix][broker] Fix shadow topics cannot be consumed when the entry is not cached (#23147)
[fix][broker] Fix system topic can not be loaded up if it contains data offloaded (#23279)
[fix][broker] Fix the broker registery cannot recover from the metadata node deletion (#23359)
[fix][broker] Fix the bug that elected leader thinks it's a follower (#23138)
[fix][broker] Handle the case when getOwnedServiceUnits fails gracefully (#23119)
[fix][broker] Internal reader of __change_events can not started after metadata store session rebuilt (#23018)
[fix][broker] Let Pending ack handler can retry to init when encounters a metadata store error (#23153)
[fix][broker] Skip reading entries from closed cursor. (#22751)
[fix][broker] Topic can never be loaded up due to broker maintains a failed topic creation future (#23184)
[fix][broker] fix exception may hidden and result in stuck when topic loading (#23102)
[fix][broker] fix pulsar-admin topics stats-internal caused a BK client thread a deadlock (#23258)
[fix][broker] fix replicated namespaces filter in filterAndUnloadMatchedNamespaceAsync (#23100)
[fix][broker] type cast on exceptions in exceptionally can lead to lost calls (#23117)
[fix][broker]A failed consumer/producer future in ServerCnx can never be removed (#23123)
[improve][broker] Add msgInReplay subscription stat and metric to improve Key_Shared observability (#23224)
[improve][broker] Add retry for start service unit state channel (ExtensibleLoadManagerImpl only) (#23230)
[improve][broker] Avoid subscription fenced error with consumer.seek whenever possible (#23163)
[improve][broker] Explicitly close LB internal topics when playing a follower (ExtensibleLoadManagerImpl only) (#23144)
[improve][broker] Improve pulsar_topic_load_failed metric to record correct failed time (#23199)
[improve][broker] Optimize high CPU usage when consuming from topics with ongoing txn (#23189)
[improve][broker] Optimize message payload traffic for ShadowReplicator (#23236)
[improve][broker] Optimize performance for checking max topics when the topic is a system topic (#23185)
[improve][broker] Optimize the performance of individual acknowledgments (#23072)
[improve][broker] Register the broker to metadata store without version id compare (#23298)
[improve][broker] Should notify bundle ownership listener onLoad event when ServiceUnitState start (ExtensibleLoadManagerImpl only) (#23152)
[improve][broker] Support customized shadow managed ledger implementation (#23179)
[improve][broker] Support to specify auth-plugin, auth-parameters and tls-enable arguments when init cluster metadata (#23087)
[improve][broker] Reuse method getAvailableBrokersAsync (#23099)
[improve][admin] PIP-369 Introduce unload flag in ns-isolation-policy set call (#23120)
[fix] StatsOutputStream: add string write function (#23227)

Client

[fix] DLQ to handle bytes key properly (#23172)
[fix][client] Copy orderingKey to retry letter topic and DLQ messages and fix bug in copying (#23182)
[fix][client] Create the retry producer async (#23157)
[fix][client] Fix for early hit beforeConsume for MultiTopicConsumer (#23141)
[fix][client] Fix timeout handling in Pulsar Admin client (#23128)
[fix][client] TransactionCoordinatorClient support retry (#23081)
[fix][client] the nullValue in msgMetadata should be true by default (#22372)
[improve][client] Add maxConnectionsPerHost and connectionMaxIdleSeconds to PulsarAdminBuilder (#22541)
[improve][client] Don't print info logs for each schema loaded by client (#23206)
[improve][client] Add new ServiceUrlProvider implementation: SameAuthParamsAutoClusterFailover (#23129)

Pulsar IO and Pulsar Functions

[fix][io] Upgrade mssql server docker tag in DebeziumMsSqlContainer (#23318)
[improve][fn] Add support for overriding additionalJavaRuntimeArguments with PF_additionalJavaRuntimeArguments env (#23130)

Others

[fix][log] Do not print error log if tenant/namespace does not exist when calling get topic metadata (#23291)
[fix][log] Do not print warn log when concurrently publishing and switching ledgers (#23209)
[feat] Add scripts for updating BK RocksDB ini files (#23178)
[fix][meta] Oxia metadta store: Convert error to MetadataStoreException if operation failed (#23154)
[fix][misc] Log Conscrypt security provider initialization warnings at debug level (#23364)
[improve][misc] Improve AES-GCM cipher performance (#23122)
[improve][misc] Optimize TLS performance by omitting extra buffer copies (#23115)
[improve][pip] PIP-366: Support to specify different config for Configuration and Local Metadata Store (#23041)
[improve][proxy] Reuse authentication instance in pulsar-proxy (#23113)

Tests & CI

[fix][test] Fix flaky SubscriptionSeekTest.testSeekIsByReceive (#23170)
[fix][test] Fix flaky UnloadSubscriptionTest.testMultiConsumer ([#23243](#23243...

Assets 2

04 Oct 18:13

lhotari

v3.0.7

aab48a1

v3.0.7

2024-10-04

Library updates

[fix][sec] Upgrade Avro to 1.11.4 to address CVE-2024-47561 (#23394)
[fix][sec] Upgrade vertx to 4.5.10 to address CVE-2024-8391 (#23338)
[fix][sec][branch-3.0] Upgrade protobuf-java to 3.25.5 (#23356) (#23357)
[fix] Bump commons-io:commons-io from 2.8.0 to 2.14.0 (#23393)
[improve][misc] Upgrade Netty to 4.1.113 and netty-tcnative to 2.0.66 (#23255)
[improve][misc] Upgrade jersey to 2.41 (#22290)
[improve] Upgrade Pulsar Python client in docker image to 3.5.0 (#23377)

Broker

[fix][broker] Cancel possible pending replay read in cancelPendingRead (#23384)
[fix][broker] Execute the pending callbacks in order before ready for incoming requests (#23266)
[fix][broker] Fix 'Disabled replicated subscriptions controller' logic and logging (#23142)
[fix][broker] Fix authenticate order in AuthenticationProviderList (#23111)
[fix][broker] Fix brokers still retry start replication after closed the topic (#23237)
[fix][broker] Fix compatibility issues for PIP-344 (#23136)
[fix][broker] Fix incomplete NAR file extraction which prevents broker from starting (#23274)
[fix][broker] Fix out-of-order issues with ConsistentHashingStickyKeyConsumerSelector (#23327)
[fix][broker] Fix shadow topics cannot be consumed when the entry is not cached (#23147)
[fix][broker] Fix system topic can not be loaded up if it contains data offloaded (#23279)
[fix][broker] Fix the bug that elected leader thinks it's a follower (#23138)
[fix][broker] Handle the case when getOwnedServiceUnits fails gracefully (#23119)
[fix][broker] Internal reader of __change_events can not started after metadata store session rebuilt (#23018)
[fix][broker] Let Pending ack handler can retry to init when encounters a metadata store error (#23153)
[fix][broker] Skip reading entries from closed cursor. (#22751)
[fix][broker] Topic can never be loaded up due to broker maintains a failed topic creation future (#23184)
[fix][broker] fix exception may hidden and result in stuck when topic loading (#23102)
[fix][broker] fix pulsar-admin topics stats-internal caused a BK client thread a deadlock (#23258)
[fix][broker] fix replicated namespaces filter in filterAndUnloadMatchedNamespaceAsync (#23100)
[fix][broker] fix the log format error in the AdminResource (#23278)
[fix][broker] type cast on exceptions in exceptionally can lead to lost calls (#23117)
[fix][broker]A failed consumer/producer future in ServerCnx can never be removed (#23123)
[fix][broker][branch-3.0] Fail fast if the extensible load manager failed to start (#23297) (#23302)
[fix][broker] StatsOutputStream: add string write function (#23227)
[improve][broker] Add msgInReplay subscription stat and metric to improve Key_Shared observability (#23224)
[improve][broker] Add retry for start service unit state channel (ExtensibleLoadManagerImpl only) (#23230)
[improve][broker] Avoid subscription fenced error with consumer.seek whenever possible (#23163)
[improve][broker] Explicitly close LB internal topics when playing a follower (ExtensibleLoadManagerImpl only) (#23144)
[improve][broker] Improve pulsar_topic_load_failed metric to record correct failed time (#23199)
[improve][broker] Optimize high CPU usage when consuming from topics with ongoing txn (#23189)
[improve][broker] Optimize performance for checking max topics when the topic is a system topic (#23185)
[improve][broker] Optimize the performance of individual acknowledgments (#23072)
[improve][broker] Part 2 of PIP-370: add metrics "pulsar_replication_disconnected_count" (#23213)
[improve][broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication (#23169)
[improve][broker] Reduce the CPU pressure from the transaction buffer in rolling restarts (#23062)
[improve][broker] Should notify bundle ownership listener onLoad event when ServiceUnitState start (ExtensibleLoadManagerImpl only) (#23152)
[improve][broker] Support customized shadow managed ledger implementation (#23179)
[improve][broker] Support to specify auth-plugin, auth-parameters and tls-enable arguments when init cluster metadata (#23087) (#23126)
[improve][broker] Reuse method getAvailableBrokersAsync (#23099)
[improve][admin] PIP-369 Introduce unload flag in ns-isolation-policy set call (#23120)

Client

[fix] DLQ to handle bytes key properly (#23172)
[fix][client] Copy orderingKey to retry letter topic and DLQ messages and fix bug in copying (#23182)
[fix][client] Create the retry producer async (#23157)
[fix][client] Fix for early hit beforeConsume for MultiTopicConsumer (#23141)
[fix][client] Fix timeout handling in Pulsar Admin client (#23128)
[fix][client] TransactionCoordinatorClient support retry (#23081)
[fix][client] fix negative message re-delivery twice issue (#20750)
[fix][client] the nullValue in msgMetadata should be true by default (#22372)
[improve][client] Add maxConnectionsPerHost and connectionMaxIdleSeconds to PulsarAdminBuilder (#22541)
[improve][client] Don't print info logs for each schema loaded by client (#23206)
[improve][client] Swallow Conscrypt ClassNotFoundException (#20371)
[improve][client]Add new ServiceUrlProvider implementation: SameAuthParamsAutoClusterFailover (#23129)
[improve][client]PIP-359:Support custom message listener executor for specific subscription (#22861)

Pulsar IO and Pulsar Functions

[fix][fn] Make python install dependencies from requirements.txt (#20174)
[fix][io] Upgrade mssql server docker tag in DebeziumMsSqlContainer (#23318)
[improve][fn] Add support for overriding additionalJavaRuntimeArguments with PF_additionalJavaRuntimeArguments env (#23130)
[fix][sql][branch-3.0] Fix UUID convert failed for JSON schema (#23292)

Others

[feat] Add scripts for updating BK RocksDB ini files (#23178)
[fix][log] Do not print error log if tenant/namespace does not exist when calling get topic metadata (#23291)
[fix][log] Do not print warn log when concurrently publishing and switching ledgers (#23209)
[fix][misc] Log Conscrypt security provider initialization warnings at debug level (#23364)
[improve][misc] Improve AES-GCM cipher performance (#23122)
[improve][misc] Optimize TLS performance by omitting extra buffer copies (#23115)
[improve][proxy] Reuse authentication instance in pulsar-proxy (#23113)

Tests & CI...

Assets 2

01 Aug 17:40

lhotari

v3.3.1

1097245

v3.3.1

2024-08-01

Broker

[feat][broker] PIP-264: Add OpenTelemetry consumer metrics (#22693)
[feat][broker] PIP-321 Introduce allowed-cluster at the namespace level (#22378) (#22961)
[fix] Make operations on individualDeletedMessages in lock scope (#22966)
[fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
[fix][admin] Fix half deletion when attempt to topic with a incorrect API (#23002)
[fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
[fix][broker] Can't connecte to non-persist topic when enable broker client tls (#22991)
[fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
[fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
[fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
[fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
[fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
[fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
[fix][broker] Fix broker OOM when upload a large package. (#22989)
[fix][broker] Fix cursor should use latest ledger config (#22644)
[fix][broker] Fix geo-replication admin client url (#22584)
[fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
[fix][broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog. (#22907)
[fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
[fix][broker] Handle BucketDelayedDeliveryTracker recover failed (#22735)
[fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
[fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
[fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
[fix][broker] Remove blocking calls from Subscription.getStats (#23088)
[fix][broker] Replication stuck when partitions count between two clusters is not the same (#22983)
[fix][broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only) (#22862)
[fix][broker] The topic might reference a closed ledger (#22860)
[fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
[fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
[fix][broker] disable loadBalancerDirectMemoryResourceWeight by default (#22821)
[fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
[fix][broker] fix replicated subscriptions for transactional messages (#22452)
[fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
[fix][broker] maintain last active info in memory only. (#22794)
[fix][broker] rename to changeMaxReadPositionCount (#22656)
[fix][broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata (#22838)
[fix][broker] usedLocallySinceLastReport should always be reset (#22672)
[fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
[fix][broker][branch-3.3] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23085)
[fix][misc] Disable JFR based telemetry collection since it's not used (#22869)
[fix][misc] Topic name from persistence name should decode local name (#22879)
[fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
[fix][ml] Fix race conditions in RangeCache (#22789)
[improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
[improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
[fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
[improve] Refactored BK ClientFactory to return futures (#22853)
[improve] Validate range of argument before long -> int conversion (#22830)
[improve][broker] Add additionalSystemCursorNames ignore list for TTL check (#22614)
[improve][broker] Add subscription prefix for internal reader (#23044)
[improve][broker] Check max producers/consumers limitation first before other ops to save resources (#23074)
[improve][broker] Close protocol handlers before unloading namespace bundles (#22728)
[improve][broker] Do not call cursor.isCursorDataFullyPersistable if disabled dispatcherPauseOnAckStatePersistentEnabled (#22729)
[improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
[improve][broker] Improve CPU resources usege of TopicName Cache (#23052)
[improve][broker] Improve exception for topic does not have schema to check (#22974)
[improve][broker] PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22779)
[improve][broker] PIP-356 Support Geo-Replication starts at earliest position (#22856)
[improve][broker] Reduce number of OpenTelemetry consumer attributes (#22837)
[improve][broker] Reduce the CPU pressure from the transaction buffer in rolling restarts (#23062)
[improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
[improve][broker] Reuse topic OpenTelemetry attributes (#22876)
[improve][broker] Trigger offload on topic load (#22652)
[improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
[improve][broker] avoid creating new objects when intercepting (#22790)
[improve][broker] high CPU usage caused by list topics under namespace (#23049)
[improve][broker] make system topic distribute evenly. (#22953)
[improve][broker]Ensure namespace deletion doesn't fail (#22627)
[improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)

Client

[fix][client] Fix negative acknowledgement by messageId (#23060)
[fix][client] Fix orphan consumer when reconnection and closing are concurrency executing (#22958)
[fix][client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted ([#22854](#22...

Assets 2

01 Aug 17:33

lhotari

v3.2.4

a29f962

v3.2.4

2024-08-01

Broker

[fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
[fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
[fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
[fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
[fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
[fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
[fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
[fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
[fix][broker] Fix broker OOM when upload a large package. (#22989)
[fix][broker] Fix cursor should use latest ledger config (#22644)
[fix][broker] Fix geo-replication admin client url (#22584)
[fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
[fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
[fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
[fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
[fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
[fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
[fix][broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only) (#22862)
[fix][broker] Support lookup options for extensible load manager (#22487)
[fix][broker] The topic might reference a closed ledger (#22860)
[fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
[fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
[fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
[fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
[fix][broker] maintain last active info in memory only. (#22794)
[fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
[fix][broker][branch-3.2] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23084)
[cleanup][ml] ManagedCursor clean up. (#22246)
[fix] Make operations on individualDeletedMessages in lock scope (#22966)
[fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
[fix][misc] Topic name from persistence name should decode local name (#22879)
[fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
[fix][ml] Fix race conditions in RangeCache (#22789)
[improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
[improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
[fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
[improve] Refactored BK ClientFactory to return futures (#22853)
[improve][broker] Clear thread local BrokerEntryMetadata instance before reuse (#22752)
[improve][broker] Close protocol handlers before unloading namespace bundles (#22728)
[improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
[improve][broker] Improve exception for topic does not have schema to check (#22974)
[improve][broker] Include runtime dependencies in server distribution (#22001)
[improve][broker] Optimize PersistentTopic.getLastDispatchablePosition (#22707)
[improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
[improve][broker] Trigger offload on topic load (#22652)
[improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
[improve][broker] avoid creating new objects when intercepting (#22790)
[improve][broker] make system topic distribute evenly. (#22953)
[improve][broker]Ensure namespace deletion doesn't fail (#22627)
[improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)

Client

[fix][client] Fix negative acknowledgement by messageId (#23060)
[fix][client] Fix orphan consumer when reconnection and closing are concurrency executing (#22958)
[fix][client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted (#22854)
[fix][client] Fix resource leak in Pulsar Client since HttpLookupService doesn't get closed (#22858)
[improve][client] improve the class GetTopicsResult (#22766)

Pulsar IO and Pulsar Functions

[fix][fn] Enable optimized Netty direct byte buffer support for Pulsar Function runtimes (#22910)
[fix][fn] Support compression type and crypto config for all producers in Functions and Connectors (#22950)
[improve][fn] Make producer cache bounded and expiring in Functions/Connectors (#22945)

Others

[fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
[fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868)
[fix][cli] Fix expiration of tokens created with "pulsar tokens create" (#22815)
[fix][cli] Fix the pulsar-daemon parameter passthrough syntax (#22905)
[fix][meta] Check if metadata store is closed in RocksdbMetadataStore (#22852)
[fix][offload] Break the fillbuffer loop when met EOF (#22722)

Library updates

[fix][sec] Upgrade Bouncycastle libraries to address CVEs (#22826)
[improve] Upgrade IPAddress to 5.5.0 (#22886)
[improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8 (#22835)
[cleanup][misc] Remove classifier from netty-transport-native-unix-common dependency (#22951)
[fix][misc] Remove RoaringBitmap dependency from pulsar-common (#23008)
[fix][misc] Rename netty native libraries in pulsar-client-admin-shaded (#22954)
[improve][misc] Replace rename-netty-native-libs.sh script with renaming with maven-shade-plugin (#22957)
[improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892)
[improve][misc][branch-3.2] Upgrade to Bookkeeper 4.16.6 (#22963)

Tests...

Assets 2

01 Aug 16:22

lhotari

v3.0.6

b9a4d98

v3.0.6

2024-08-01

Broker

[feat][broker] Implementation of PIP-323: Complete Backlog Quota Telemetry (#21816) (#22740)
[feat][broker][branch-3.0] PIP-321 Introduce allowed-cluster at the namespace level (#22378) (#22960)
[fix] Make operations on individualDeletedMessages in lock scope (#22966)
[fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
[improve] Refactored BK ClientFactory to return futures (#22853)
[fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
[fix][broker] Can't connecte to non-persist topic when enable broker client tls (#22991)
[fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
[fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
[fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
[fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
[fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
[fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
[fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
[fix][broker] Fix broker OOM when upload a large package. (#22989)
[fix][broker] Fix cursor should use latest ledger config (#22644)
[fix][broker] Fix geo-replication admin client url (#22584)
[fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
[fix][broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog. (#22907)
[fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
[fix][broker] Fix wrong logic of method TopicName.getPartition(int index) (#19841)
[fix][broker] Handle BucketDelayedDeliveryTracker recover failed (#22735)
[fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
[fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
[fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
[fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
[fix][broker] Remove blocking calls from Subscription.getStats (#23088)
[fix][broker] Replication stuck when partitions count between two clusters is not the same (#22983)
[fix][broker] Support lookup options for extensible load manager (#22487)
[fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
[fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
[fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
[fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
[fix][broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata (#22838)
[fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
[fix][broker][branch-3.0] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23077)
[fix][broker][branch-3.0] The topic might reference a closed ledger (#22860) (#22900)
[improve][broker] Check max producers/consumers limitation first before other ops to save resources (#23074)
[improve][broker] Clear thread local BrokerEntryMetadata instance before reuse (#22752)
[improve][broker] Follow up #4196 use PulsarByteBufAllocator handle OOM (#20837)
[improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
[improve][broker] Improve CPU resources usege of TopicName Cache (#23052)
[improve][broker] Improve efficiency of checking message deletion (#20490)
[improve][broker] Improve exception for topic does not have schema to check (#22974)
[improve][broker] Include runtime dependencies in server distribution (#22001)
[improve][broker] Optimize PersistentTopic.getLastDispatchablePosition (#22707)
[improve][broker] PIP-356 Support Geo-Replication starts at earliest position (#22856)
[improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
[improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
[improve][broker] [break change] Do not create partitioned DLQ/Retry topic automatically (#22705)
[improve][broker] avoid creating new objects when intercepting (#22790)
[improve][broker] checkTopicExists supports checking partitioned topic without index (#21701)
[improve][broker] do not grant permission for each partition to reduce unnecessary zk metadata (#18222)
[improve][broker] high CPU usage caused by list topics under namespace (#23049)
[improve][broker] make system topic distribute evenly. (#22953)
[improve][broker]Ensure namespace deletion doesn't fail (#22627)
[improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)
[cleanup][ml] ManagedCursor clean up. (#22246)
[fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
[fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
[fix][ml] Fix race conditions in RangeCache (#22789)
[improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
[improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
[fix][admin] Clearly define REST API on Open API (#22783)
[fix][admin] Clearly define REST API on Open API for Namesaces@v2 (#22775)
[fix][admin] Clearly define REST API on Open API for Topics (#22782)
[fix][admin] Fix can't delete tenant for v1 (#22550)
[fix][admin] Fix half deletion when attempt to topic with a incorrect API (#23002)
[fix][admin][part-1]Clearly define REST API on Open API (#22774)
[improve][admin] Check if the topic existed before the permission operations (#22742)

Client

[fix][client] Fix negative acknowledgement by messageId (#23060)
[fix][client] Fix o...

Assets 2

05 Jun 16:26

coderzc

v3.3.0

c0aab49

v3.3.0

PIP

PIP-315: Configurable max delay limit for delayed delivery (#21798)
PIP-321 Introduce allowed-cluster at the namespace level (#21648)
PIP-324: Alpine Docker images (#21716)
PIP-325: Add command to abort transaction (#21731)
PIP-326: Create a BOM to ease dependency management (#21747)
PIP-329: Strategy for maintaining the latest tag to Pulsar docker images (#21872)
PIP-330: getMessagesById gets all messages (#21873)
PIP 342: Support OpenTelemetry metrics in Pulsar client (#22178)
PIP-343: Use picocli instead of jcommander (#22181)
PIP-344 Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) (#22182)
PIP-335: Oxia metadata plugin (#22009)
PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources (#22071)

Broker

[admin][broker] Fix force delete subscription not working (#22423)
[cleanup][admin] Remove unused methods in PersistentTopicsBase (#22424)
[cleanup][broker] Remove unused NamespaceBundleFactory parameter when creating OwnershipCache (#22482)
[cleanup][broker]∂ fix doc for TransactionBuffer#isTxnAborted (#21956)
[cleanup][broker] fix return value of TransactionTimeoutTrackerImpl#addTransaction (#22020)
[cleanup][broker] remove useless code comment (#22459)
[cleanup][meta] Remove com.beust.jcommander.internal import (#22294)
[cleanup][ml] ManagedCursor clean up. (#22246)
[cleanup][test] remove useless TestAuthorizationProvider2 (#22595)
[feat][admin] Enable Gzip Compression by Default in Admin Client (#22464)
[feat][admin] PIP-330: getMessagesById gets all messages (#21918)
[feat][broker] Implementation of PIP-323: Complete Backlog Quota Telemetry (#21816)
[feat][broker] PIP-264: Add Java runtime metrics (#22616)
[feat][broker] PIP-264: Add topic messaging metrics (#22467)
[feat][misc] Add Pulsar BOM (Bill of Materials) (#21871)
[feat][misc] PIP-264: Implement topic lookup metrics using OpenTelemetry (#22058)
[feat][misc] PIP-320: Add OpenTelemetry scaffolding (#22010)
[fix] Fix Reader can be stuck from transaction aborted messages. (#22610)
[fix] Fixed implicit conversions from long -> int (#22055)
[fix] Include swagger annotations in shaded client lib (#22570)
[fix] Restored method as deprecated in AbstractMetadataStore (#21950)
[fix] Test was leaving client instance to null (#22631)
[fix][admin] Clearly define REST API on Open API (#22783)
[fix][admin] Clearly define REST API on Open API for Namesaces@v2 (#22775)
[fix][admin] Clearly define REST API on Open API for Topics (#22782)
[fix][admin] Fix can't delete tenant for v1 (#22550)
[fix][admin] Fix deprecated check (#22653)
[fix][admin] Fix namespace admin api exception response (#22587)
[fix][admin][part-1]Clearly define REST API on Open API (#22774)
[fix][broker] Add the missed opentelemetry-sdk-testing dependency to tests of pulsar-broker-auth-sasl (#22665)
[fix][broker] Avoid being stuck when closing the broker with extensible load manager (#22573)
[fix][broker] Avoid consumers receiving acknowledged messages from compacted topic after reconnection (#21187)
[fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
[fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
[fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
[fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
[fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
[fix][broker] Continue closing even when executor is shut down (#22599)
[fix][broker] Create new ledger after the current ledger is closed (#22034)
[fix][broker] Delete compacted ledger when topic is deleted (#21745)
[fix][broker] Disable system topic message deduplication (#22582)
[fix][broker] Do not migrate internal topics during Blue-Green Migration when ExtensibleLoadBalancer is used (#22478)
[fix][broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency (#22090)
[fix][broker] Expire messages according to ledger close time to avoid client clock skew (#21940)
[fix][broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression (#22576)
[fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533)
[fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
[fix][broker] Fix PersistentSubscription duplicate implementation interface Subscription (#22359)
[fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
[fix][broker] Fix ResourceGroup report local usage (#22340)
[fix][broker] Fix ResourceGroups loading (#21781)
[fix][broker] Fix String wrong format (#21829)
[fix][broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init (#22528)
[fix][broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 (#21885)
[fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
[fix][broker] Fix broker not starting when both transactions and the Extensible Load Manager are enabled (#22139)
[fix][broker] Fix can not subscribe partitioned topic with a suffix-matched regexp (#22025)
[fix][broker] Fix compaction/replication data loss when expire messages (#21865)
[fix][broker] Fix configurationMetadataSyncEventTopic is marked supporting dynamic setting, but not implemented (#22684)
[fix][broker] Fix consumer stops receiving messages when with large backlogs processing (#22454)
[fix][broker] Fix deadlock while skip non-recoverable ledgers. (#21915)
[fix][broker] Fix getMessageById throws 500 (#21919)
[fix][broker] Fix hash collision when using a consumer name that ends with a number (#22053)
[fix][broker] Fix invalid condition in logging exceptions (#22412)
[fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
[fix][broker] Fix issue with GetMessageIdByTimestamp can't find match messageId from compacted ledger (#21600)
[fix][broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured (#21894)
[fix][broker] Fix message drop record in producer stat (#22458)
[fix][broker] Fix metrics pulsar_topic_load_failed_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (#22580)
[fix][broker] Fix nothing changed after removing dynamic configs (#22673)
[fix][broker] Fix potential NPE when disabling the liveness check (#21840)
[fix][broker] Fix reader stuck when read from compacted topic with read compact mode disable (#21969)
[fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set (#21842)
[fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema (#21977)
[fix][broker] Fix thread safety of loadSheddingTask and loadResourceQuotaTask fields (#22660)
[fix][broker] Fix typo (#21843)
[fix][broker] Fix typo in CompactedTopicImpl. (#22235)
[fix][broker] Fix typos in Consumer class (#22532)
[fix][broker] Fix typos in PersistentTopic class (#22364)
[fix][broker] Fix typos in UniformLoadShedder class (#21907)
[fix][broker] Fix write all compacted out entry into compacted topic (#21917)
[fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
[fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
[fix][broker] One topic can be closed multiple times concurrently (#17524)
[fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode (#22494)
[fix][broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21946)
[fix][broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21948)
[fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
[fix][broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState (#22572)
[fix][broker] Replication stopped due to unload topic failed (#21947)
[fix][broker] Restore the broker id to match the format used in existing Pulsar releases (#21937)
[fix][broker] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init (#22112)
[fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
[fix][broker] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs (#22064)
[fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379)
[fix][broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog (#22019)
[fix][broker] Support OIDC providers with JWK without alg field set in keys (#22421)
[fix][broker] Support lookup options for extensible load manager (#22487)
[fix][broker] Support running docker container with gid != 0 (#22081)
[fix][broker] Unify topic-level policies enable judgment conditions (#19501)
[fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321)
[fix][broker] Update topic partition failed when config maxNumPartitionsPerPartitionedTop...