From e000482f66fdac67760761e2937a59e54c0b77c0 Mon Sep 17 00:00:00 2001 From: fengyubiao Date: Mon, 4 Mar 2024 08:23:25 +0800 Subject: [PATCH 01/25] - --- pip/pip-344.md | 138 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 pip/pip-344.md diff --git a/pip/pip-344.md b/pip/pip-344.md new file mode 100644 index 0000000000000..1c29b42a27dda --- /dev/null +++ b/pip/pip-344.md @@ -0,0 +1,138 @@ +# PIP-XXX: PIP-344 Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) + +# Background knowledge + +### Topic auto-creation. +- The partitioned topic auto-creation is dependent on `pulsarClient.getPartitionsForTopic` + - It triggers partitioned metadata creation by `pulsarClient.getPartitionsForTopic` + - And triggers the topic partition creation by producers' registration and consumers' registration. +- When calling `pulsarClient.getPartitionsForTopic(topicName)`, Pulsar will automatically create the partitioned topic metadata if it does not exist, either using `HttpLookupService` or `BinaryProtoLookupService`. + +### Now `pulsarClient.getPartitionsForTopic`'s behavior. +| case | broker allow `auto-create` | param allow
`create if not exists` | non-partitioned topic | partitioned topic | current behavior | +| --- | --- | --- | --- | --- | --- | +| 1 | `true/false` | `true/false` | `exists: true` | | HTTP API: `partitions: 0`
Binary API: `partitions: 0` | +| 2 | `true/false` | `true/false` | | `exists: true`
`partitions: 3` | HTTP API: `partitions: 3`
Binary API: `partitions: 3` | +| 3 | `true` | `true` | | | HTTP API:
  - `create new: true`
  - `partitions: 3`
Binary API:
  - `create new: true`
  - `partitions: 3`
| +| 4 | `true` | `false` | | | HTTP API:
  - `create new: false`
  - `partitions: 0`
Binary API: not support
| +| 5 | `false` | `true` | | | HTTP API:
  - `create new: false`
  - `partitions: 0`
Binary API:
  - `create new: false`
  - `partitions: 0`
| + +# Motivation + +- For the case 4 of `pulsarClient.getPartitionsForTopic`'s behavior, it always tries to create the partitioned metadata, but the API name is `getxxx`. +- For the case 5 of `pulsarClient.getPartitionsForTopic`'s behavior, it returns a `0` partitioned metadata, but the topic not exists. For this behavior, we had discussed [here](https://github.com/apache/pulsar/issues/8813) before. + +# Goals + +- Do not try to create the partitioned metadata when calling + + +# High Level Design + + + +# Detailed Design + +## Design & Implementation Details + + + +## Public-facing Changes + + + +### Public API + + +### Binary protocol + +### Configuration + +### CLI + +### Metrics + + + + +# Monitoring + + + +# Security Considerations + + +# Backward & Forward Compatibility + +## Revert + + + +## Upgrade + + + +# Alternatives + + + +# General Notes + +# Links + + +* Mailing List discussion thread: +* Mailing List voting thread: From e251ef97de0c5c75838e6bff7d3db050706c1c49 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 4 Mar 2024 08:24:57 +0800 Subject: [PATCH 02/25] Update pip-344.md --- pip/pip-344.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 1c29b42a27dda..00684620aeab8 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -1,4 +1,4 @@ -# PIP-XXX: PIP-344 Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) +# PIP-344: Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) # Background knowledge From 98ea4c8f00884c559c1b461de8e23b66c978d881 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 4 Mar 2024 08:25:35 +0800 Subject: [PATCH 03/25] Update pip-344.md --- pip/pip-344.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 00684620aeab8..0673e003ce2c3 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -14,7 +14,7 @@ | 1 | `true/false` | `true/false` | `exists: true` | | HTTP API: `partitions: 0`
Binary API: `partitions: 0` | | 2 | `true/false` | `true/false` | | `exists: true`
`partitions: 3` | HTTP API: `partitions: 3`
Binary API: `partitions: 3` | | 3 | `true` | `true` | | | HTTP API:
  - `create new: true`
  - `partitions: 3`
Binary API:
  - `create new: true`
  - `partitions: 3`
| -| 4 | `true` | `false` | | | HTTP API:
  - `create new: false`
  - `partitions: 0`
Binary API: not support
| +| 4 | `true` | `false` | | | HTTP API:
  - `create new: false`
  - `partitions: 0`
Binary API:
  not support
| | 5 | `false` | `true` | | | HTTP API:
  - `create new: false`
  - `partitions: 0`
Binary API:
  - `create new: false`
  - `partitions: 0`
| # Motivation From 2d17e2a1d00d6f167321352f49d551fd9ce8948e Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 4 Mar 2024 08:27:24 +0800 Subject: [PATCH 04/25] Update pip-344.md --- pip/pip-344.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 0673e003ce2c3..901b5cf169ec0 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -11,11 +11,11 @@ ### Now `pulsarClient.getPartitionsForTopic`'s behavior. | case | broker allow `auto-create` | param allow
`create if not exists` | non-partitioned topic | partitioned topic | current behavior | | --- | --- | --- | --- | --- | --- | -| 1 | `true/false` | `true/false` | `exists: true` | | HTTP API: `partitions: 0`
Binary API: `partitions: 0` | -| 2 | `true/false` | `true/false` | | `exists: true`
`partitions: 3` | HTTP API: `partitions: 3`
Binary API: `partitions: 3` | -| 3 | `true` | `true` | | | HTTP API:
  - `create new: true`
  - `partitions: 3`
Binary API:
  - `create new: true`
  - `partitions: 3`
| -| 4 | `true` | `false` | | | HTTP API:
  - `create new: false`
  - `partitions: 0`
Binary API:
  not support
| -| 5 | `false` | `true` | | | HTTP API:
  - `create new: false`
  - `partitions: 0`
Binary API:
  - `create new: false`
  - `partitions: 0`
| +| 1 | `true/false` | `true/false` | `exists: true` | | REST API: `partitions: 0`
Client API: `partitions: 0` | +| 2 | `true/false` | `true/false` | | `exists: true`
`partitions: 3` | REST API: `partitions: 3`
Client API: `partitions: 3` | +| 3 | `true` | `true` | | | REST API:
  - `create new: true`
  - `partitions: 3`
Client API:
  - `create new: true`
  - `partitions: 3`
| +| 4 | `true` | `false` | | | REST API:
  - `create new: false`
  - `partitions: 0`
Client API:
  not support
| +| 5 | `false` | `true` | | | REST API:
  - `create new: false`
  - `partitions: 0`
Client API:
  - `create new: false`
  - `partitions: 0`
| # Motivation From 4029ec83ffa3c38b2b60d4cee763d6dbda6cd8a7 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 4 Mar 2024 08:40:03 +0800 Subject: [PATCH 05/25] Update pip-344.md --- pip/pip-344.md | 66 ++++++++++++++++++-------------------------------- 1 file changed, 24 insertions(+), 42 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 901b5cf169ec0..227e9e8730d0c 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -19,62 +19,44 @@ # Motivation -- For the case 4 of `pulsarClient.getPartitionsForTopic`'s behavior, it always tries to create the partitioned metadata, but the API name is `getxxx`. -- For the case 5 of `pulsarClient.getPartitionsForTopic`'s behavior, it returns a `0` partitioned metadata, but the topic not exists. For this behavior, we had discussed [here](https://github.com/apache/pulsar/issues/8813) before. - -# Goals - -- Do not try to create the partitioned metadata when calling +The param `create if not exists` of the Client API is always `true.` +- For case 4 of `pulsarClient.getPartitionsForTopic`'s behavior, it always tries to create the partitioned metadata, but the API name is `getxxx`. +- For case 5 of `pulsarClient.getPartitionsForTopic`'s behavior, it returns a `0` partitioned metadata, but the topic does not exist. For the correct behavior of this case, we had discussed [here](https://github.com/apache/pulsar/issues/8813) before. -# High Level Design - - +- Do not create the partitioned metadata when calling `pulsarClient.getPartitionsForTopic`, the partitioned metadata only will be created when consumers/producers are trying to register. +- Instead of returning a `0` partitioned metadata, respond to a not found error when calling `pulsarClient.getPartitionsForTopic` if the topic does not exist. # Detailed Design -## Design & Implementation Details - - - ## Public-facing Changes - +When you call the public API `pulsarClient.getPartitionsForTopic`, pulsar will not create the partitioned metadata anymore. ### Public API - +**LookupService.java** +``` +// This API existed before. Not change it, thus ensuring compatibility. +CompletableFuture getPartitionedTopicMetadata(TopicName topicName); + +// A new API that contains an additional param "createIfAutoCreationEnabled". +CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); +``` ### Binary protocol +**CommandPartitionedTopicMetadata** +``` +message CommandPartitionedTopicMetadata { + + optional bool create_if_auto_creation_enabled = 6 [default = true]; +} +``` + + ### Configuration ### CLI From 0d07254e723274696eae4d20cec9eb415fe23ea9 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 4 Mar 2024 08:53:39 +0800 Subject: [PATCH 06/25] Update pip-344.md --- pip/pip-344.md | 78 +++++++++----------------------------------------- 1 file changed, 14 insertions(+), 64 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 227e9e8730d0c..f438fedaf0334 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -2,13 +2,13 @@ # Background knowledge -### Topic auto-creation. +**Topic auto-creation** - The partitioned topic auto-creation is dependent on `pulsarClient.getPartitionsForTopic` - It triggers partitioned metadata creation by `pulsarClient.getPartitionsForTopic` - And triggers the topic partition creation by producers' registration and consumers' registration. - When calling `pulsarClient.getPartitionsForTopic(topicName)`, Pulsar will automatically create the partitioned topic metadata if it does not exist, either using `HttpLookupService` or `BinaryProtoLookupService`. -### Now `pulsarClient.getPartitionsForTopic`'s behavior. +**Now `pulsarClient.getPartitionsForTopic`'s behavior** | case | broker allow `auto-create` | param allow
`create if not exists` | non-partitioned topic | partitioned topic | current behavior | | --- | --- | --- | --- | --- | --- | | 1 | `true/false` | `true/false` | `exists: true` | | REST API: `partitions: 0`
Client API: `partitions: 0` | @@ -28,7 +28,7 @@ The param `create if not exists` of the Client API is always `true.` Correct the behaviors of case 4 and case 5. -- Do not create the partitioned metadata when calling `pulsarClient.getPartitionsForTopic`, the partitioned metadata only will be created when consumers/producers are trying to register. +- Do not create the partitioned metadata when calling `pulsarClient.getPartitionsForTopic`. The partitioned metadata will only be created when consumers/producers are trying to register. - Instead of returning a `0` partitioned metadata, respond to a not found error when calling `pulsarClient.getPartitionsForTopic` if the topic does not exist. # Detailed Design @@ -43,8 +43,8 @@ When you call the public API `pulsarClient.getPartitionsForTopic`, pulsar will n // This API existed before. Not change it, thus ensuring compatibility. CompletableFuture getPartitionedTopicMetadata(TopicName topicName); -// A new API that contains an additional param "createIfAutoCreationEnabled". -CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); ++ // A new API that contains an additional param "createIfAutoCreationEnabled." ++ CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); ``` ### Binary protocol @@ -52,69 +52,19 @@ CompletableFuture getPartitionedTopicMetadata(TopicNam **CommandPartitionedTopicMetadata** ``` message CommandPartitionedTopicMetadata { - + optional bool create_if_auto_creation_enabled = 6 [default = true]; + + optional bool create_if_auto_creation_enabled = 6 [default = true]; } ``` - -### Configuration - -### CLI - -### Metrics - - - - -# Monitoring - - - -# Security Considerations - - # Backward & Forward Compatibility -## Revert - - - -## Upgrade - - - -# Alternatives - - - -# General Notes +**Old version client and New version Broker** +Pulsar will create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. +- The client does not pass the `create_if_auto_creation_enabled` parameter of `CommandPartitionedTopicMetadata` to the Broker, which defaults to `true` and automatically creates the partitioned topic metadata. -# Links +**New version client and Old version Broker** +Pulsar will create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. +- The client passes the `create_if_auto_creation_enabled` parameter of `CommandPartitionedTopicMetadata` to the Broker; the Broker will discard it and create the partitioned topic metadata automatically. - -* Mailing List discussion thread: -* Mailing List voting thread: +**New version client and New version Broker** +Pulsar will not create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. From d73c0dee037988d12b5ba0cc0b7a2ccb0f80d71c Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 4 Mar 2024 08:54:30 +0800 Subject: [PATCH 07/25] Update pip-344.md --- pip/pip-344.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index f438fedaf0334..747bc1401afd2 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -58,13 +58,13 @@ message CommandPartitionedTopicMetadata { # Backward & Forward Compatibility -**Old version client and New version Broker** -Pulsar will create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. -- The client does not pass the `create_if_auto_creation_enabled` parameter of `CommandPartitionedTopicMetadata` to the Broker, which defaults to `true` and automatically creates the partitioned topic metadata. +- Old version client and New version Broker + - Pulsar will create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. + - The client does not pass the `create_if_auto_creation_enabled` parameter of `CommandPartitionedTopicMetadata` to the Broker, which defaults to `true` and automatically creates the partitioned topic metadata. -**New version client and Old version Broker** -Pulsar will create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. -- The client passes the `create_if_auto_creation_enabled` parameter of `CommandPartitionedTopicMetadata` to the Broker; the Broker will discard it and create the partitioned topic metadata automatically. +- New version client and Old version Broker + - Pulsar will create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. + - The client passes the `create_if_auto_creation_enabled` parameter of `CommandPartitionedTopicMetadata` to the Broker; the Broker will discard it and create the partitioned topic metadata automatically. -**New version client and New version Broker** -Pulsar will not create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. +- New version client and New version Broker + - Pulsar will not create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. From 38de7f2f2031c2c9b194a04790a96024227e199d Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 4 Mar 2024 08:59:36 +0800 Subject: [PATCH 08/25] Update pip-344.md --- pip/pip-344.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/pip/pip-344.md b/pip/pip-344.md index 747bc1401afd2..209cd0853ebb1 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -31,6 +31,14 @@ Correct the behaviors of case 4 and case 5. - Do not create the partitioned metadata when calling `pulsarClient.getPartitionsForTopic`. The partitioned metadata will only be created when consumers/producers are trying to register. - Instead of returning a `0` partitioned metadata, respond to a not found error when calling `pulsarClient.getPartitionsForTopic` if the topic does not exist. +**`pulsarClient.getPartitionsForTopic`'s behavior** +| case | do `auto-create` | non-partitioned topic | partitioned topic | current behavior | +| --- | --- | --- | --- | --- | +| 1 | `true/false` | `exists: true` | | REST/Client API: `partitions: 0` | +| 2 | `true/false` | | `exists: true`
`partitions: 3` | REST/Client API: `partitions: 3` | +| 3 | `true` | | | REST/Client API:
  - `create new: true`
  - `partitions: 3` | +| 4 | `false` | | | REST/Client API:
  - Not found error | + # Detailed Design ## Public-facing Changes From 0e2829b063885ef19f196e8e916ba60cb4f4587a Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 4 Mar 2024 09:01:04 +0800 Subject: [PATCH 09/25] Update pip-344.md --- pip/pip-344.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 209cd0853ebb1..e216f30c38948 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -31,14 +31,6 @@ Correct the behaviors of case 4 and case 5. - Do not create the partitioned metadata when calling `pulsarClient.getPartitionsForTopic`. The partitioned metadata will only be created when consumers/producers are trying to register. - Instead of returning a `0` partitioned metadata, respond to a not found error when calling `pulsarClient.getPartitionsForTopic` if the topic does not exist. -**`pulsarClient.getPartitionsForTopic`'s behavior** -| case | do `auto-create` | non-partitioned topic | partitioned topic | current behavior | -| --- | --- | --- | --- | --- | -| 1 | `true/false` | `exists: true` | | REST/Client API: `partitions: 0` | -| 2 | `true/false` | | `exists: true`
`partitions: 3` | REST/Client API: `partitions: 3` | -| 3 | `true` | | | REST/Client API:
  - `create new: true`
  - `partitions: 3` | -| 4 | `false` | | | REST/Client API:
  - Not found error | - # Detailed Design ## Public-facing Changes @@ -55,6 +47,14 @@ CompletableFuture getPartitionedTopicMetadata(TopicNam + CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); ``` +**`LookupService.getPartitionedTopicMetadata`'s behavior** +| case | do create partitioned metadata | non-partitioned topic | partitioned topic | current behavior | +| --- | --- | --- | --- | --- | +| 1 | `true/false` | `exists: true` | | REST/Client API: `partitions: 0` | +| 2 | `true/false` | | `exists: true`
`partitions: 3` | REST/Client API: `partitions: 3` | +| 3 | `true` | | | REST/Client API:
  - `create new: true`
  - `partitions: 3` | +| 4 | `false` | | | REST/Client API:
  - Not found error | + ### Binary protocol **CommandPartitionedTopicMetadata** From 167abc9470f30e2112c5b85250385f8d0d8f3ca3 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 4 Mar 2024 09:01:32 +0800 Subject: [PATCH 10/25] Update pip-344.md --- pip/pip-344.md | 1 - 1 file changed, 1 deletion(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index e216f30c38948..8b676311145f5 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -47,7 +47,6 @@ CompletableFuture getPartitionedTopicMetadata(TopicNam + CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); ``` -**`LookupService.getPartitionedTopicMetadata`'s behavior** | case | do create partitioned metadata | non-partitioned topic | partitioned topic | current behavior | | --- | --- | --- | --- | --- | | 1 | `true/false` | `exists: true` | | REST/Client API: `partitions: 0` | From 423cf8f9002d44b97e769de0e794aada2c981af5 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 4 Mar 2024 20:45:44 +0800 Subject: [PATCH 11/25] Update pip-344.md --- pip/pip-344.md | 1 + 1 file changed, 1 insertion(+) diff --git a/pip/pip-344.md b/pip/pip-344.md index 8b676311145f5..eed4f4102f2bc 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -36,6 +36,7 @@ Correct the behaviors of case 4 and case 5. ## Public-facing Changes When you call the public API `pulsarClient.getPartitionsForTopic`, pulsar will not create the partitioned metadata anymore. +- [flink-connector-pulsar](https://github.com/apache/flink-connector-pulsar/blob/main/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/ProducerRegister.java#L221-L227) is using this API to create partitioned topic metadata. ### Public API **LookupService.java** From 4f58a2cb550756f876486df15f5c5e6e8572d6e8 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 11 Mar 2024 15:59:33 +0800 Subject: [PATCH 12/25] Update pip-344.md --- pip/pip-344.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index eed4f4102f2bc..ad0c467126804 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -60,7 +60,7 @@ CompletableFuture getPartitionedTopicMetadata(TopicNam **CommandPartitionedTopicMetadata** ``` message CommandPartitionedTopicMetadata { - + optional bool create_if_auto_creation_enabled = 6 [default = true]; + + optional bool metadata_auto_creation_enabled = 6 [default = true]; } ``` From 4a55795c338fb27395a9b2b069500dd3dffe536e Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Wed, 20 Mar 2024 13:03:38 +0800 Subject: [PATCH 13/25] Update pip-344.md --- pip/pip-344.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index ad0c467126804..2cc8d079abbfd 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -64,6 +64,13 @@ message CommandPartitionedTopicMetadata { } ``` +**FeatureFlags** +``` +message FeatureFlags { + + optional bool supports_binary_api_get_partitioned_meta_with_param_created_false = 5 [default = false]; +} +``` + # Backward & Forward Compatibility - Old version client and New version Broker @@ -72,7 +79,9 @@ message CommandPartitionedTopicMetadata { - New version client and Old version Broker - Pulsar will create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. - - The client passes the `create_if_auto_creation_enabled` parameter of `CommandPartitionedTopicMetadata` to the Broker; the Broker will discard it and create the partitioned topic metadata automatically. + - The client passes the `create_if_auto_creation_enabled` parameter of `CommandPartitionedTopicMetadata` to the Broker; + - `create_if_auto_creation_enabled` is `true`: the Broker creates the partitioned topic metadata automatically. + - `create_if_auto_creation_enabled` is `false`: the feature flag `supports_binary_api_get_partitioned_meta_with_param_created_false` will be false, the client-side will prevent this request, and users will get an unsupported error. - New version client and New version Broker - Pulsar will not create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. From 9c378577dc2e010de5a5c90e0d5542780a3d4673 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Wed, 20 Mar 2024 16:09:13 +0800 Subject: [PATCH 14/25] Update pip-344.md --- pip/pip-344.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pip/pip-344.md b/pip/pip-344.md index 2cc8d079abbfd..f2035a9ce7c68 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -42,6 +42,8 @@ When you call the public API `pulsarClient.getPartitionsForTopic`, pulsar will n **LookupService.java** ``` // This API existed before. Not change it, thus ensuring compatibility. +// @Deprecated it is not suggested to use now; please use {@link #getPartitionedTopicMetadata(TopicName, boolean)}. +@Deprecated CompletableFuture getPartitionedTopicMetadata(TopicName topicName); + // A new API that contains an additional param "createIfAutoCreationEnabled." From 40d2b3df284a0f296a15f8af55bcc4b85660fee5 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Wed, 20 Mar 2024 16:22:47 +0800 Subject: [PATCH 15/25] Update pip-344.md --- pip/pip-344.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index f2035a9ce7c68..e9f4b3c7f3429 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -11,15 +11,23 @@ **Now `pulsarClient.getPartitionsForTopic`'s behavior** | case | broker allow `auto-create` | param allow
`create if not exists` | non-partitioned topic | partitioned topic | current behavior | | --- | --- | --- | --- | --- | --- | -| 1 | `true/false` | `true/false` | `exists: true` | | REST API: `partitions: 0`
Client API: `partitions: 0` | -| 2 | `true/false` | `true/false` | | `exists: true`
`partitions: 3` | REST API: `partitions: 3`
Client API: `partitions: 3` | -| 3 | `true` | `true` | | | REST API:
  - `create new: true`
  - `partitions: 3`
Client API:
  - `create new: true`
  - `partitions: 3`
| -| 4 | `true` | `false` | | | REST API:
  - `create new: false`
  - `partitions: 0`
Client API:
  not support
| -| 5 | `false` | `true` | | | REST API:
  - `create new: false`
  - `partitions: 0`
Client API:
  - `create new: false`
  - `partitions: 0`
| +| 1 | `true/false` | `true/false` | `exists: true` | | REST API: `partitions: 0`
Binary API: `partitions: 0` | +| 2 | `true/false` | `true/false` | | `exists: true`
`partitions: 3` | REST API: `partitions: 3`
Binary API: `partitions: 3` | +| 3 | `true` | `true` | | | REST API:
  - `create new: true`
  - `partitions: 3`
Binary API:
  - `create new: true`
  - `partitions: 3`
| +| 4 | `true` | `false` | | | REST API:
  - `create new: false`
  - `partitions: 0`
Binary API:
  not support
| +| 5 | `false` | `true` | | | REST API:
  - `create new: false`
  - `partitions: 0`
Binary API:
  - `create new: false`
  - `partitions: 0`
| + +- Broker allows `auto-create`: see also the config `allowAutoTopicCreation` in `broker.conf`. +- Param allow
`create if not exists` + - Regarding the HTTP API `PersistentTopics.getPartitionedMetadata`, it is an optional param which named `checkAllowAutoCreation,` and the default value is `false`. + - Regarding the `pulsar-admin` API, it depends on the HTTP API `PersistentTopics.getPartitionedMetadata`, and it always sets the param `checkAllowAutoCreation` to `false` and can not be set manually. + - Regarding the client API `HttpLookupService.getPartitionedTopicMetadata`, it depends on the HTTP API `PersistentTopics.getPartitionedMetadata`, and it always sets the param `checkAllowAutoCreation` to `true` and can not be set manually. + - Regarding the client API `BinaryProtoLookupService.getPartitionedTopicMetadata`, it always tries to create partitioned metadata. +- `REST API & HTTP API`: Since there are only two implementations of the 4 ways to get partitioned metadata, we call HTTP API `PersistentTopics.getPartitionedMetadata`, `pulsar-admin`, and `HttpLookupService.getPartitionedTopicMetadata` HTTP API, and call `BinaryProtoLookupService.getPartitionedTopicMetadata` Binary API. # Motivation -The param `create if not exists` of the Client API is always `true.` +The param `create if not exists` of the Binary API is always `true.` - For case 4 of `pulsarClient.getPartitionsForTopic`'s behavior, it always tries to create the partitioned metadata, but the API name is `getxxx`. - For case 5 of `pulsarClient.getPartitionsForTopic`'s behavior, it returns a `0` partitioned metadata, but the topic does not exist. For the correct behavior of this case, we had discussed [here](https://github.com/apache/pulsar/issues/8813) before. @@ -52,10 +60,10 @@ CompletableFuture getPartitionedTopicMetadata(TopicNam | case | do create partitioned metadata | non-partitioned topic | partitioned topic | current behavior | | --- | --- | --- | --- | --- | -| 1 | `true/false` | `exists: true` | | REST/Client API: `partitions: 0` | -| 2 | `true/false` | | `exists: true`
`partitions: 3` | REST/Client API: `partitions: 3` | -| 3 | `true` | | | REST/Client API:
  - `create new: true`
  - `partitions: 3` | -| 4 | `false` | | | REST/Client API:
  - Not found error | +| 1 | `true/false` | `exists: true` | | REST/Binary API: `partitions: 0` | +| 2 | `true/false` | | `exists: true`
`partitions: 3` | REST/Binary API: `partitions: 3` | +| 3 | `true` | | | REST/Binary API:
  - `create new: true`
  - `partitions: 3` | +| 4 | `false` | | | REST/Binary API:
  - Not found error | ### Binary protocol From 0e2220458f6714af2c144ea49f4abab5881250fb Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Wed, 20 Mar 2024 16:44:23 +0800 Subject: [PATCH 16/25] Update pip-344.md --- pip/pip-344.md | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index e9f4b3c7f3429..f9eadc9176524 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -7,6 +7,7 @@ - It triggers partitioned metadata creation by `pulsarClient.getPartitionsForTopic` - And triggers the topic partition creation by producers' registration and consumers' registration. - When calling `pulsarClient.getPartitionsForTopic(topicName)`, Pulsar will automatically create the partitioned topic metadata if it does not exist, either using `HttpLookupService` or `BinaryProtoLookupService`. +- BTW, [flink-connector-pulsar](https://github.com/apache/flink-connector-pulsar/blob/main/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/ProducerRegister.java#L221-L227) is using this API to create partitioned topic metadata. **Now `pulsarClient.getPartitionsForTopic`'s behavior** | case | broker allow `auto-create` | param allow
`create if not exists` | non-partitioned topic | partitioned topic | current behavior | @@ -44,11 +45,10 @@ Correct the behaviors of case 4 and case 5. ## Public-facing Changes When you call the public API `pulsarClient.getPartitionsForTopic`, pulsar will not create the partitioned metadata anymore. -- [flink-connector-pulsar](https://github.com/apache/flink-connector-pulsar/blob/main/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/ProducerRegister.java#L221-L227) is using this API to create partitioned topic metadata. ### Public API **LookupService.java** -``` +```java // This API existed before. Not change it, thus ensuring compatibility. // @Deprecated it is not suggested to use now; please use {@link #getPartitionedTopicMetadata(TopicName, boolean)}. @Deprecated @@ -58,13 +58,29 @@ CompletableFuture getPartitionedTopicMetadata(TopicNam + CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); ``` -| case | do create partitioned metadata | non-partitioned topic | partitioned topic | current behavior | +**PulsarClient.java** +```java +// This API existed before. Not change it, thus ensuring compatibility. +// @Deprecated it is not suggested to use now; please use {@link #getPartitionsForTopic(TopicName, boolean)}. +CompletableFuture> getPartitionsForTopic(String topic); + ++ // A new API that contains an additional param "createIfAutoCreationEnabled." ++ CompletableFuture> getPartitionsForTopic(String topic, boolean createIfAutoCreationEnabled); +``` + +The behavior of the old API `LookupService.getPartitionedTopicMetadata(TopicName)` and `PulsarClient.getPartitionsForTopic(String)`. +- It keeps the same behavior as before; in other words, it also tries to create the partitioned topic metadata automatically. + +The behavior of the new API `LookupService.getPartitionedTopicMetadata(TopicName, boolean)` and `PulsarClient.getPartitionsForTopic(String, boolean)`. +| case | `createIfAutoCreationEnabled` | non-partitioned topic | partitioned topic | current behavior | | --- | --- | --- | --- | --- | | 1 | `true/false` | `exists: true` | | REST/Binary API: `partitions: 0` | | 2 | `true/false` | | `exists: true`
`partitions: 3` | REST/Binary API: `partitions: 3` | | 3 | `true` | | | REST/Binary API:
  - `create new: true`
  - `partitions: 3` | | 4 | `false` | | | REST/Binary API:
  - Not found error | + + ### Binary protocol **CommandPartitionedTopicMetadata** From bcaac2404fcb0f65320152c0208feb225520eea3 Mon Sep 17 00:00:00 2001 From: fengyubiao Date: Thu, 21 Mar 2024 01:21:02 +0800 Subject: [PATCH 17/25] - --- pip/pip-344.md | 55 ++++++++++++++++++++++++++++---------------------- 1 file changed, 31 insertions(+), 24 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index f9eadc9176524..67af8f9116c93 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -35,10 +35,8 @@ The param `create if not exists` of the Binary API is always `true.` # Goals -Correct the behaviors of case 4 and case 5. - -- Do not create the partitioned metadata when calling `pulsarClient.getPartitionsForTopic`. The partitioned metadata will only be created when consumers/producers are trying to register. -- Instead of returning a `0` partitioned metadata, respond to a not found error when calling `pulsarClient.getPartitionsForTopic` if the topic does not exist. +- Regarding the case 4: Add a new API `PulsarClient.getPartitionsForTopic(String, boolean)` to support the feature that just get partitioned topic metadata and do not try to create one. See detail below. +- Regarding the case 5: Instead of returning a `0` partitioned metadata, respond to a not found error when calling `pulsarClient.getPartitionsForTopic(String)` if the topic does not exist. # Detailed Design @@ -58,6 +56,21 @@ CompletableFuture getPartitionedTopicMetadata(TopicNam + CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); ``` +The behavior of the old API `LookupService.getPartitionedTopicMetadata(String)`. +- It keeps the same behavior as before; in other words, it is the same as the case 1-4 below. + +The behavior of the new API `LookupService.getPartitionedTopicMetadata(TopicName, boolean)`. + +| case | client-side param: `createIfAutoCreationEnabled` | non-partitioned topic | partitioned topic | broker-side: topic auto-creation strategy | current behavior | +|------|--------------------------------------------------|-----------------------|-------------------------------------|--------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------| +| 1 | `true/false` | `exists: true` | | | REST/Binary API: `{partitions: 0}` | +| 2 | `true/false` | | `exists: true`
`partitions: 2` | | REST/Binary API: `{partitions: 2}` | +| 3 | `true` | | | `allowAutoTopicCreation`: `true` `allowAutoTopicCreationType`: `non-partitioned` | REST/Binary API:
  - `create new: true`
  - `{partitions: 0}` | +| 4 | `true` | | | `allowAutoTopicCreation`: `true` `allowAutoTopicCreationType`: `partitioned`
`defaultNumPartitions`: `2` | REST/Binary API:
  - `create new: true`
  - `{partitions: 2}` | +| 5 | `false` | | | `allowAutoTopicCreation`: `true` | REST/Binary API:
  - Not found error | +| 6 | `true` | | | `allowAutoTopicCreation`: `false` | REST/Binary API:
  - Not found error | + + **PulsarClient.java** ```java // This API existed before. Not change it, thus ensuring compatibility. @@ -68,16 +81,19 @@ CompletableFuture> getPartitionsForTopic(String topic); + CompletableFuture> getPartitionsForTopic(String topic, boolean createIfAutoCreationEnabled); ``` -The behavior of the old API `LookupService.getPartitionedTopicMetadata(TopicName)` and `PulsarClient.getPartitionsForTopic(String)`. -- It keeps the same behavior as before; in other words, it also tries to create the partitioned topic metadata automatically. +The behavior of the old API `PulsarClient.getPartitionsForTopic(String)`. +- It keeps the same behavior as before; in other words, it is the same as the case 1-4 below. + +The behavior of the new API `PulsarClient.getPartitionsForTopic(String, boolean)`. -The behavior of the new API `LookupService.getPartitionedTopicMetadata(TopicName, boolean)` and `PulsarClient.getPartitionsForTopic(String, boolean)`. -| case | `createIfAutoCreationEnabled` | non-partitioned topic | partitioned topic | current behavior | -| --- | --- | --- | --- | --- | -| 1 | `true/false` | `exists: true` | | REST/Binary API: `partitions: 0` | -| 2 | `true/false` | | `exists: true`
`partitions: 3` | REST/Binary API: `partitions: 3` | -| 3 | `true` | | | REST/Binary API:
  - `create new: true`
  - `partitions: 3` | -| 4 | `false` | | | REST/Binary API:
  - Not found error | +| case | client-side param: `createIfAutoCreationEnabled` | non-partitioned topic | partitioned topic | broker-side: topic autp-creation strategy | current behavior | +|------|--------------------------------------------------|----------------------|-------------------------------------|--------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| 1 | `true/false` | `exists: true` | | | REST/Binary API: `["{tenat}/{ns}/topic"]` | +| 2 | `true/false` | | `exists: true`
`partitions: 2` | | REST/Binary `API`: `["{tenat}/{ns}/topic-partition-0", "{tenat}/{ns}/topic-partition-1"]` | +| 3 | `true` | | | `allowAutoTopicCreation`: `true` `allowAutoTopicCreationType`: `non-partitioned` | REST/Binary API:
  - `create new: true`
  - `["{tenat}/{ns}/topic"]` | +| 4 | `true` | | | `allowAutoTopicCreation`: `true` `allowAutoTopicCreationType`: `partitioned`
`defaultNumPartitions`: `2` | REST/Binary API:
  - `create new: true`
  - `["{tenat}/{ns}/topic-partition-0", "{tenat}/{ns}/topic-partition-1"]` | +| 5 | `false` | | | `allowAutoTopicCreation`: `true` | REST/Binary API:
  - Not found error | +| 5 | `true` | | | `allowAutoTopicCreation`: `false` | REST/Binary API:
  - Not found error | @@ -99,15 +115,6 @@ message FeatureFlags { # Backward & Forward Compatibility -- Old version client and New version Broker - - Pulsar will create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. - - The client does not pass the `create_if_auto_creation_enabled` parameter of `CommandPartitionedTopicMetadata` to the Broker, which defaults to `true` and automatically creates the partitioned topic metadata. - -- New version client and Old version Broker - - Pulsar will create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. - - The client passes the `create_if_auto_creation_enabled` parameter of `CommandPartitionedTopicMetadata` to the Broker; - - `create_if_auto_creation_enabled` is `true`: the Broker creates the partitioned topic metadata automatically. - - `create_if_auto_creation_enabled` is `false`: the feature flag `supports_binary_api_get_partitioned_meta_with_param_created_false` will be false, the client-side will prevent this request, and users will get an unsupported error. +- Old version client and New version Broker: The client will call the old API. -- New version client and New version Broker - - Pulsar will not create the partitioned topic metadata when calling `pulsarClient.getPartitionsForTopic`. +- New version client and Old version Broker: the feature flag `supports_binary_api_get_partitioned_meta_with_param_created_false` will be `false`, call the deprecated old API. From 4ee0526da4135b327728c59c8b086e279829fbe7 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 21 Mar 2024 01:28:07 +0800 Subject: [PATCH 18/25] Update pip-344.md --- pip/pip-344.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 67af8f9116c93..2a21a0844e649 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -49,8 +49,11 @@ When you call the public API `pulsarClient.getPartitionsForTopic`, pulsar will n ```java // This API existed before. Not change it, thus ensuring compatibility. // @Deprecated it is not suggested to use now; please use {@link #getPartitionedTopicMetadata(TopicName, boolean)}. -@Deprecated -CompletableFuture getPartitionedTopicMetadata(TopicName topicName); ++ @Deprecated +- CompletableFuture getPartitionedTopicMetadata(TopicName topicName); ++ default CompletableFuture> getPartitionedTopicMetadata(String topic) { ++ getPartitionsForTopic(topic, true); ++ } + // A new API that contains an additional param "createIfAutoCreationEnabled." + CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); @@ -74,8 +77,11 @@ The behavior of the new API `LookupService.getPartitionedTopicMetadata(TopicName **PulsarClient.java** ```java // This API existed before. Not change it, thus ensuring compatibility. -// @Deprecated it is not suggested to use now; please use {@link #getPartitionsForTopic(TopicName, boolean)}. -CompletableFuture> getPartitionsForTopic(String topic); ++ @Deprecated it is not suggested to use now; please use {@link #getPartitionsForTopic(TopicName, boolean)}. +- CompletableFuture> getPartitionsForTopic(String topic); ++ default CompletableFuture> getPartitionsForTopic(String topic) { ++ getPartitionsForTopic(topic, true); ++ } + // A new API that contains an additional param "createIfAutoCreationEnabled." + CompletableFuture> getPartitionsForTopic(String topic, boolean createIfAutoCreationEnabled); From 268c5ec2f0ad7b5612eaa07e6428138beaf6613c Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 21 Mar 2024 01:37:53 +0800 Subject: [PATCH 19/25] Update pip-344.md --- pip/pip-344.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 2a21a0844e649..cf7e764d7303b 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -59,9 +59,6 @@ When you call the public API `pulsarClient.getPartitionsForTopic`, pulsar will n + CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); ``` -The behavior of the old API `LookupService.getPartitionedTopicMetadata(String)`. -- It keeps the same behavior as before; in other words, it is the same as the case 1-4 below. - The behavior of the new API `LookupService.getPartitionedTopicMetadata(TopicName, boolean)`. | case | client-side param: `createIfAutoCreationEnabled` | non-partitioned topic | partitioned topic | broker-side: topic auto-creation strategy | current behavior | @@ -87,9 +84,6 @@ The behavior of the new API `LookupService.getPartitionedTopicMetadata(TopicName + CompletableFuture> getPartitionsForTopic(String topic, boolean createIfAutoCreationEnabled); ``` -The behavior of the old API `PulsarClient.getPartitionsForTopic(String)`. -- It keeps the same behavior as before; in other words, it is the same as the case 1-4 below. - The behavior of the new API `PulsarClient.getPartitionsForTopic(String, boolean)`. | case | client-side param: `createIfAutoCreationEnabled` | non-partitioned topic | partitioned topic | broker-side: topic autp-creation strategy | current behavior | From 69d864500437775cc9a6ac283db758238a39855c Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 21 Mar 2024 01:38:36 +0800 Subject: [PATCH 20/25] Update pip-344.md --- pip/pip-344.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index cf7e764d7303b..97a213ba7ec59 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -7,7 +7,6 @@ - It triggers partitioned metadata creation by `pulsarClient.getPartitionsForTopic` - And triggers the topic partition creation by producers' registration and consumers' registration. - When calling `pulsarClient.getPartitionsForTopic(topicName)`, Pulsar will automatically create the partitioned topic metadata if it does not exist, either using `HttpLookupService` or `BinaryProtoLookupService`. -- BTW, [flink-connector-pulsar](https://github.com/apache/flink-connector-pulsar/blob/main/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/ProducerRegister.java#L221-L227) is using this API to create partitioned topic metadata. **Now `pulsarClient.getPartitionsForTopic`'s behavior** | case | broker allow `auto-create` | param allow
`create if not exists` | non-partitioned topic | partitioned topic | current behavior | @@ -32,6 +31,7 @@ The param `create if not exists` of the Binary API is always `true.` - For case 4 of `pulsarClient.getPartitionsForTopic`'s behavior, it always tries to create the partitioned metadata, but the API name is `getxxx`. - For case 5 of `pulsarClient.getPartitionsForTopic`'s behavior, it returns a `0` partitioned metadata, but the topic does not exist. For the correct behavior of this case, we had discussed [here](https://github.com/apache/pulsar/issues/8813) before. +- BTW, [flink-connector-pulsar](https://github.com/apache/flink-connector-pulsar/blob/main/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/ProducerRegister.java#L221-L227) is using this API to create partitioned topic metadata. # Goals From c7d076bcad6ae350224a1769655ae2fd1f048fd3 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 21 Mar 2024 01:40:09 +0800 Subject: [PATCH 21/25] Update pip-344.md --- pip/pip-344.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 97a213ba7ec59..2b1af559a1c09 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -117,4 +117,4 @@ message FeatureFlags { - Old version client and New version Broker: The client will call the old API. -- New version client and Old version Broker: the feature flag `supports_binary_api_get_partitioned_meta_with_param_created_false` will be `false`, call the deprecated old API. +- New version client and Old version Broker: the feature flag `supports_binary_api_get_partitioned_meta_with_param_created_false` will be `false`, call the old API. From 71bbf3dceaff8e39a5d94f5ee7d186d7743d43ca Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 21 Mar 2024 11:12:16 +0800 Subject: [PATCH 22/25] Update pip-344.md --- pip/pip-344.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 2b1af559a1c09..dcaec728b86f3 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -47,13 +47,8 @@ When you call the public API `pulsarClient.getPartitionsForTopic`, pulsar will n ### Public API **LookupService.java** ```java -// This API existed before. Not change it, thus ensuring compatibility. -// @Deprecated it is not suggested to use now; please use {@link #getPartitionedTopicMetadata(TopicName, boolean)}. -+ @Deprecated + - CompletableFuture getPartitionedTopicMetadata(TopicName topicName); -+ default CompletableFuture> getPartitionedTopicMetadata(String topic) { -+ getPartitionsForTopic(topic, true); -+ } + // A new API that contains an additional param "createIfAutoCreationEnabled." + CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); From e895378759264fb19057bc07044cc979a4f998e9 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 21 Mar 2024 11:18:43 +0800 Subject: [PATCH 23/25] Update pip-344.md --- pip/pip-344.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index dcaec728b86f3..7b88f53fcd314 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -112,4 +112,4 @@ message FeatureFlags { - Old version client and New version Broker: The client will call the old API. -- New version client and Old version Broker: the feature flag `supports_binary_api_get_partitioned_meta_with_param_created_false` will be `false`, call the old API. +- New version client and Old version Broker: The feature flag `supports_binary_api_get_partitioned_meta_with_param_created_false` will be `false`. The client will get a not-support error if the param `createIfAutoCreationEnabled` is false. From 17dbd79e60ac8bf82c896682141731e810a8962f Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 21 Mar 2024 15:59:50 +0800 Subject: [PATCH 24/25] Update pip-344.md --- pip/pip-344.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index 7b88f53fcd314..f4f033c803b3e 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -50,7 +50,13 @@ When you call the public API `pulsarClient.getPartitionsForTopic`, pulsar will n - CompletableFuture getPartitionedTopicMetadata(TopicName topicName); -+ // A new API that contains an additional param "createIfAutoCreationEnabled." +/** + * 1. Get the partitions if the topic exists. Return "{partition: n}" if a partitioned topic exists; return "{partition: 0}" if a non-partitioned topic exists. + * 2. When {@param createIfAutoCreationEnabled} is "false," neither partitioned topic nor non-partitioned topic does not exist. You will get an {@link PulsarClientException.NotFoundException}. + * 2-1. You will get a {@link PulsarClientException.NotSupportedException} if the broker's version is an older one that does not support this feature. + * 3. When {@param createIfAutoCreationEnabled} is "true," it will trigger an auto-creation for this topic(using the default topic auto-creation strategy you set for the broker), and the corresponding result is returned. For the result, see case 1. + * @version 3.3.0 + */ + CompletableFuture getPartitionedTopicMetadata(TopicName topicName, boolean createIfAutoCreationEnabled); ``` @@ -75,8 +81,14 @@ The behavior of the new API `LookupService.getPartitionedTopicMetadata(TopicName + getPartitionsForTopic(topic, true); + } -+ // A new API that contains an additional param "createIfAutoCreationEnabled." -+ CompletableFuture> getPartitionsForTopic(String topic, boolean createIfAutoCreationEnabled); +/** + * 1. Get the partitions if the topic exists. Return "[{partition-0}, {partition-1}....{partition-n}}]" if a partitioned topic exists; return "[{topic}]" if a non-partitioned topic exists. + * 2. When {@param createIfAutoCreationEnabled} is "false", neither the partitioned topic nor non-partitioned topic does not exist. You will get an {@link PulsarClientException.NotFoundException}. + * 2-1. You will get a {@link PulsarClientException.NotSupportedException} if the broker's version is an older one that does not support this feature. + * 3. When {@param createIfAutoCreationEnabled} is "true," it will trigger an auto-creation for this topic(using the default topic auto-creation strategy you set for the broker), and the corresponding result is returned. For the result, see case 1. + * @version 3.3.0 + */ +CompletableFuture> getPartitionsForTopic(String topic, boolean createIfAutoCreationEnabled); ``` The behavior of the new API `PulsarClient.getPartitionsForTopic(String, boolean)`. From 1a14760e4b25d0f6b88685badc4f0d5b8042b0f4 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 21 Mar 2024 16:03:34 +0800 Subject: [PATCH 25/25] Update pip-344.md --- pip/pip-344.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pip/pip-344.md b/pip/pip-344.md index f4f033c803b3e..5eafc6fd5c279 100644 --- a/pip/pip-344.md +++ b/pip/pip-344.md @@ -53,7 +53,7 @@ When you call the public API `pulsarClient.getPartitionsForTopic`, pulsar will n /** * 1. Get the partitions if the topic exists. Return "{partition: n}" if a partitioned topic exists; return "{partition: 0}" if a non-partitioned topic exists. * 2. When {@param createIfAutoCreationEnabled} is "false," neither partitioned topic nor non-partitioned topic does not exist. You will get an {@link PulsarClientException.NotFoundException}. - * 2-1. You will get a {@link PulsarClientException.NotSupportedException} if the broker's version is an older one that does not support this feature. + * 2-1. You will get a {@link PulsarClientException.NotSupportedException} if the broker's version is an older one that does not support this feature and the Pulsar client is using a binary protocol "serviceUrl". * 3. When {@param createIfAutoCreationEnabled} is "true," it will trigger an auto-creation for this topic(using the default topic auto-creation strategy you set for the broker), and the corresponding result is returned. For the result, see case 1. * @version 3.3.0 */ @@ -84,7 +84,7 @@ The behavior of the new API `LookupService.getPartitionedTopicMetadata(TopicName /** * 1. Get the partitions if the topic exists. Return "[{partition-0}, {partition-1}....{partition-n}}]" if a partitioned topic exists; return "[{topic}]" if a non-partitioned topic exists. * 2. When {@param createIfAutoCreationEnabled} is "false", neither the partitioned topic nor non-partitioned topic does not exist. You will get an {@link PulsarClientException.NotFoundException}. - * 2-1. You will get a {@link PulsarClientException.NotSupportedException} if the broker's version is an older one that does not support this feature. + * 2-1. You will get a {@link PulsarClientException.NotSupportedException} if the broker's version is an older one that does not support this feature and the Pulsar client is using a binary protocol "serviceUrl". * 3. When {@param createIfAutoCreationEnabled} is "true," it will trigger an auto-creation for this topic(using the default topic auto-creation strategy you set for the broker), and the corresponding result is returned. For the result, see case 1. * @version 3.3.0 */