Make standalone use setMetadataStoreUrl #14384

gaozhangmin · 2022-02-19T18:15:08Z

Motivation

Remove deprecated zookeeperServers in standalone.
Fix incorrect ZK connection String after metadataStoreUrl added ZK prefix.

Modifications

setZookeeperServers to setMetadataStoreUrl

Documentation

Check the box below or label this PR directly (if you have committer privilege).

Need to update docs?

doc-required

(If you need help on updating docs, create a doc issue)
no-need-doc

(Please explain why)
doc

(If this PR contains doc changes)

github-actions · 2022-02-19T18:15:32Z

@gaozhangmin:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

github-actions · 2022-02-19T18:20:03Z

@gaozhangmin:Thanks for providing doc info!

gaozhangmin · 2022-02-21T10:08:27Z

/pulsarbot run-failure-checks

codelipenghui · 2022-02-23T04:45:07Z

/pulsarbot run-failure-checks

gaozhangmin · 2022-02-25T06:58:42Z

/pulsarbot run-failure-checks

gaozhangmin · 2022-02-25T09:12:51Z

/pulsarbot run-failure-checks

...r-functions/worker/src/main/java/org/apache/pulsar/functions/worker/PulsarWorkerService.java

pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerUtils.java

...java/org/apache/pulsar/packages/management/storage/bookkeeper/BookKeeperPackagesStorage.java

...he/pulsar/packages/management/storage/bookkeeper/BookKeeperPackagesStorageConfiguration.java

gaozhangmin · 2022-02-25T16:25:29Z

/pulsarbot run-failure-checks

gaozhangmin · 2022-02-28T02:04:54Z

/pulsarbot run-failure-checks

…kString

gaozhangmin · 2022-02-28T11:26:17Z

/pulsarbot run-failure-checks

gaozhangmin · 2022-02-28T12:44:56Z

@nicoloboschi there were backward compatibility problem before, I fixed it. PTAL again. Thx

if (StringUtils.isNotBlank(configuration.getMetadataStoreUrl())) {
                ledgersStoreServers = configuration.getMetadataStoreUrl();
                if (ledgersStoreServers.startsWith(ZK_SCHEME_IDENTIFIER)) {
                    ledgersStoreServers = ledgersStoreServers.substring(ZK_SCHEME_IDENTIFIER.length());
                }
            } else {
                ledgersStoreServers = configuration.getZookeeperServers();
            }

gaozhangmin · 2022-03-01T00:20:24Z

/pulsarbot run-failure-checks

gaozhangmin · 2022-03-01T02:15:14Z

/pulsarbot run-failure-checks

…tible across different versions (#17690) * [fix][functions] Ensure InternalConfigurationData data model is compatible across different versions * style * fix the other way ### Motivation After #14384, the broker and the client expects that the `InternalConfigurationData` contains `metadataStoreUrl` and `configurationMetadataStoreUrl` fields. However the broker is no more compatible with old clients. #14384 is landed to branch-2.11 and [2.10.2](#17291) Example scenario: - broker on 2.10.1 - function worker on 2.10.1 1. upgrade fn worker to 2.11.0 or 2.10.2 2. the fn worker starts and download the internal config from the broker 3. broker serves a json with old fields (`zookeeperServers` and `configurationStoreServers`) 4. fn worker reads the json and convert it to a `InternalConfigurationData` instance. It expects to see the fields filled `metadataStoreUrl` and `configurationMetadataStoreUrl` but they aren't 5. NPE on fn worker ``` 2022-09-15T17:42:16,072+0000 [main] INFO org.apache.pulsar.functions.worker.PulsarWorkerService - Initializing Pulsar Functions namespace... 2022-09-15T17:42:16,192+0000 [main] ERROR org.apache.pulsar.functions.worker.FunctionWorkerStarter - Encountered error in function worker. java.lang.NullPointerException: null at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.removeIdentifierFromMetadataURL(MetadataStoreFactoryImpl.java:73) at org.apache.pulsar.functions.worker.WorkerUtils.initializeDlogNamespace(WorkerUtils.java:188) at org.apache.pulsar.functions.worker.PulsarWorkerService.initializeStandaloneWorkerService(PulsarWorkerService.java:281) at org.apache.pulsar.functions.worker.PulsarWorkerService.initAsStandalone(PulsarWorkerService.java:208) at org.apache.pulsar.functions.worker.Worker.start(Worker.java:54) at org.apache.pulsar.functions.worker.FunctionWorkerStarter.main(FunctionWorkerStarter.java:76) ``` Additionaly there's the same issue if we upgrade the broker before the fn worker: 1. the broker gets the upgrade. won't serve `zookeeperServers` field 2. fn worker restarts for some reasons. 3. fn worker gets the internal config and look for `zookeeperServers` field which is empty in the json 4. NPE ### Modifications * Restore old fields in `InternalConfigurationData` and add fallback the old values in the new fields getters * Added unit test - [x] `doc-not-needed`

…tible across different versions (#17690) * [fix][functions] Ensure InternalConfigurationData data model is compatible across different versions * style * fix the other way After #14384, the broker and the client expects that the `InternalConfigurationData` contains `metadataStoreUrl` and `configurationMetadataStoreUrl` fields. However the broker is no more compatible with old clients. #14384 is landed to branch-2.11 and [2.10.2](#17291) Example scenario: - broker on 2.10.1 - function worker on 2.10.1 1. upgrade fn worker to 2.11.0 or 2.10.2 2. the fn worker starts and download the internal config from the broker 3. broker serves a json with old fields (`zookeeperServers` and `configurationStoreServers`) 4. fn worker reads the json and convert it to a `InternalConfigurationData` instance. It expects to see the fields filled `metadataStoreUrl` and `configurationMetadataStoreUrl` but they aren't 5. NPE on fn worker ``` 2022-09-15T17:42:16,072+0000 [main] INFO org.apache.pulsar.functions.worker.PulsarWorkerService - Initializing Pulsar Functions namespace... 2022-09-15T17:42:16,192+0000 [main] ERROR org.apache.pulsar.functions.worker.FunctionWorkerStarter - Encountered error in function worker. java.lang.NullPointerException: null at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.removeIdentifierFromMetadataURL(MetadataStoreFactoryImpl.java:73) at org.apache.pulsar.functions.worker.WorkerUtils.initializeDlogNamespace(WorkerUtils.java:188) at org.apache.pulsar.functions.worker.PulsarWorkerService.initializeStandaloneWorkerService(PulsarWorkerService.java:281) at org.apache.pulsar.functions.worker.PulsarWorkerService.initAsStandalone(PulsarWorkerService.java:208) at org.apache.pulsar.functions.worker.Worker.start(Worker.java:54) at org.apache.pulsar.functions.worker.FunctionWorkerStarter.main(FunctionWorkerStarter.java:76) ``` Additionaly there's the same issue if we upgrade the broker before the fn worker: 1. the broker gets the upgrade. won't serve `zookeeperServers` field 2. fn worker restarts for some reasons. 3. fn worker gets the internal config and look for `zookeeperServers` field which is empty in the json 4. NPE * Restore old fields in `InternalConfigurationData` and add fallback the old values in the new fields getters * Added unit test - [x] `doc-not-needed` (cherry picked from commit 1a34b87)

…tible across different versions (#17690) * [fix][functions] Ensure InternalConfigurationData data model is compatible across different versions * style * fix the other way ### Motivation After #14384, the broker and the client expects that the `InternalConfigurationData` contains `metadataStoreUrl` and `configurationMetadataStoreUrl` fields. However the broker is no more compatible with old clients. #14384 is landed to branch-2.11 and [2.10.2](#17291) Example scenario: - broker on 2.10.1 - function worker on 2.10.1 1. upgrade fn worker to 2.11.0 or 2.10.2 2. the fn worker starts and download the internal config from the broker 3. broker serves a json with old fields (`zookeeperServers` and `configurationStoreServers`) 4. fn worker reads the json and convert it to a `InternalConfigurationData` instance. It expects to see the fields filled `metadataStoreUrl` and `configurationMetadataStoreUrl` but they aren't 5. NPE on fn worker ``` 2022-09-15T17:42:16,072+0000 [main] INFO org.apache.pulsar.functions.worker.PulsarWorkerService - Initializing Pulsar Functions namespace... 2022-09-15T17:42:16,192+0000 [main] ERROR org.apache.pulsar.functions.worker.FunctionWorkerStarter - Encountered error in function worker. java.lang.NullPointerException: null at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.removeIdentifierFromMetadataURL(MetadataStoreFactoryImpl.java:73) at org.apache.pulsar.functions.worker.WorkerUtils.initializeDlogNamespace(WorkerUtils.java:188) at org.apache.pulsar.functions.worker.PulsarWorkerService.initializeStandaloneWorkerService(PulsarWorkerService.java:281) at org.apache.pulsar.functions.worker.PulsarWorkerService.initAsStandalone(PulsarWorkerService.java:208) at org.apache.pulsar.functions.worker.Worker.start(Worker.java:54) at org.apache.pulsar.functions.worker.FunctionWorkerStarter.main(FunctionWorkerStarter.java:76) ``` Additionaly there's the same issue if we upgrade the broker before the fn worker: 1. the broker gets the upgrade. won't serve `zookeeperServers` field 2. fn worker restarts for some reasons. 3. fn worker gets the internal config and look for `zookeeperServers` field which is empty in the json 4. NPE ### Modifications * Restore old fields in `InternalConfigurationData` and add fallback the old values in the new fields getters * Added unit test - [x] `doc-not-needed` (cherry picked from commit 1a34b87)

…tible across different versions (apache#17690) * [fix][functions] Ensure InternalConfigurationData data model is compatible across different versions * style * fix the other way After apache#14384, the broker and the client expects that the `InternalConfigurationData` contains `metadataStoreUrl` and `configurationMetadataStoreUrl` fields. However the broker is no more compatible with old clients. apache#14384 is landed to branch-2.11 and [2.10.2](apache#17291) Example scenario: - broker on 2.10.1 - function worker on 2.10.1 1. upgrade fn worker to 2.11.0 or 2.10.2 2. the fn worker starts and download the internal config from the broker 3. broker serves a json with old fields (`zookeeperServers` and `configurationStoreServers`) 4. fn worker reads the json and convert it to a `InternalConfigurationData` instance. It expects to see the fields filled `metadataStoreUrl` and `configurationMetadataStoreUrl` but they aren't 5. NPE on fn worker ``` 2022-09-15T17:42:16,072+0000 [main] INFO org.apache.pulsar.functions.worker.PulsarWorkerService - Initializing Pulsar Functions namespace... 2022-09-15T17:42:16,192+0000 [main] ERROR org.apache.pulsar.functions.worker.FunctionWorkerStarter - Encountered error in function worker. java.lang.NullPointerException: null at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.removeIdentifierFromMetadataURL(MetadataStoreFactoryImpl.java:73) at org.apache.pulsar.functions.worker.WorkerUtils.initializeDlogNamespace(WorkerUtils.java:188) at org.apache.pulsar.functions.worker.PulsarWorkerService.initializeStandaloneWorkerService(PulsarWorkerService.java:281) at org.apache.pulsar.functions.worker.PulsarWorkerService.initAsStandalone(PulsarWorkerService.java:208) at org.apache.pulsar.functions.worker.Worker.start(Worker.java:54) at org.apache.pulsar.functions.worker.FunctionWorkerStarter.main(FunctionWorkerStarter.java:76) ``` Additionaly there's the same issue if we upgrade the broker before the fn worker: 1. the broker gets the upgrade. won't serve `zookeeperServers` field 2. fn worker restarts for some reasons. 3. fn worker gets the internal config and look for `zookeeperServers` field which is empty in the json 4. NPE * Restore old fields in `InternalConfigurationData` and add fallback the old values in the new fields getters * Added unit test - [x] `doc-not-needed` (cherry picked from commit 1a34b87) (cherry picked from commit 0993a34)

github-actions bot assigned gaozhangmin Feb 19, 2022

github-actions bot added the doc-label-missing label Feb 19, 2022

github-actions bot added doc-not-needed Your PR changes do not impact docs and removed doc-label-missing labels Feb 19, 2022

gaozhangmin force-pushed the standalone-setMetadataStoreUrl branch from b467605 to 3c4dcc2 Compare February 21, 2022 06:55

RobertIndie approved these changes Feb 21, 2022

View reviewed changes

codelipenghui approved these changes Feb 23, 2022

View reviewed changes

codelipenghui added this to the 2.11.0 milestone Feb 23, 2022

gaozhangmin force-pushed the standalone-setMetadataStoreUrl branch from 7617a36 to ae4c657 Compare February 25, 2022 06:11

gaozhangmin force-pushed the standalone-setMetadataStoreUrl branch from ae4c657 to 628515b Compare February 25, 2022 07:25

nicoloboschi requested changes Feb 25, 2022

View reviewed changes

gaozhangmin force-pushed the standalone-setMetadataStoreUrl branch from 1d1b3d5 to 098be8c Compare February 25, 2022 11:01

gavingaozhangmin added 4 commits February 28, 2022 12:33

Make stadnalone use setMetadataStoreUrl and fix functions incorrect z…

540a01d

…kString

fix unit test error

e03fb7e

fix code style

7074f4c

use removeIdentifierFromMetadataURL

67a5f6f

gaozhangmin force-pushed the standalone-setMetadataStoreUrl branch 2 times, most recently from f51129b to 67a5f6f Compare February 28, 2022 06:29

fix npe

a26f647

gaozhangmin force-pushed the standalone-setMetadataStoreUrl branch from f18c847 to a26f647 Compare February 28, 2022 08:16

fix unit test error

f09866d

RobertIndie mentioned this pull request Feb 28, 2022

Possible code changes required for #13077 #13760

Closed

fix npe

41c7d07

nicoloboschi approved these changes Feb 28, 2022

View reviewed changes

codelipenghui merged commit e20f34b into apache:master Mar 1, 2022

gaoran10 changed the title ~~Make stadnalone use setMetadataStoreUrl~~ Make standalone use setMetadataStoreUrl Mar 7, 2022

gaozhangmin pushed a commit to gaozhangmin/pulsar that referenced this pull request Mar 8, 2022

Fix potential npe bug of apache#14384

aa8b761

codelipenghui pushed a commit that referenced this pull request Mar 9, 2022

Fix potential npe bug of #14384 (#14595)

7c1f17a

Nicklee007 pushed a commit to Nicklee007/pulsar that referenced this pull request Apr 20, 2022

Make stadnalone use setMetadataStoreUrl (apache#14384)

bea5fd7

Nicklee007 pushed a commit to Nicklee007/pulsar that referenced this pull request Apr 20, 2022

Fix potential npe bug of apache#14384 (apache#14595)

f525775

horizonzy mentioned this pull request Aug 26, 2022

[Branch-2.10] [fix] [broker] Fix bookeeper packages npe #17291

Merged

4 tasks

nicoloboschi mentioned this pull request Sep 16, 2022

[fix][functions] Ensure InternalConfigurationData data model is compatible across different versions #17690

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make standalone use setMetadataStoreUrl #14384

Make standalone use setMetadataStoreUrl #14384

gaozhangmin commented Feb 19, 2022 •

edited

Loading

github-actions bot commented Feb 19, 2022

github-actions bot commented Feb 19, 2022

gaozhangmin commented Feb 21, 2022

codelipenghui commented Feb 23, 2022

gaozhangmin commented Feb 25, 2022

gaozhangmin commented Feb 25, 2022

gaozhangmin commented Feb 25, 2022

gaozhangmin commented Feb 28, 2022

gaozhangmin commented Feb 28, 2022

gaozhangmin commented Feb 28, 2022 •

edited

Loading

gaozhangmin commented Mar 1, 2022

gaozhangmin commented Mar 1, 2022

Make standalone use setMetadataStoreUrl #14384

Make standalone use setMetadataStoreUrl #14384

Conversation

gaozhangmin commented Feb 19, 2022 • edited Loading

Motivation

Modifications

Documentation

github-actions bot commented Feb 19, 2022

github-actions bot commented Feb 19, 2022

gaozhangmin commented Feb 21, 2022

codelipenghui commented Feb 23, 2022

gaozhangmin commented Feb 25, 2022

gaozhangmin commented Feb 25, 2022

gaozhangmin commented Feb 25, 2022

gaozhangmin commented Feb 28, 2022

gaozhangmin commented Feb 28, 2022

gaozhangmin commented Feb 28, 2022 • edited Loading

gaozhangmin commented Mar 1, 2022

gaozhangmin commented Mar 1, 2022

gaozhangmin commented Feb 19, 2022 •

edited

Loading

gaozhangmin commented Feb 28, 2022 •

edited

Loading