Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][common] Make Bookkeeper metadata options configurable #17834

Merged

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Sep 26, 2022

Motivation

PIP-45 related PRs #12770 and #13296 introduced changes in Pulsar and Bookkeeper configuration for Pulsar so that Pulsar Metadata driver is used for metadata operations in the bookkeeper client (in broker) and in the bookkeeper server (the bookies).
It would be good to have a way to opt-out of this change and configure Pulsar and Bookies without Pulsar Metadata store and use Zookeeper directly.

This PR will allow to use Bookkeeper defaults by setting export BK_METADATA_OPTIONS=none.

If one would want to configure Bookies without Pulsar Metadata, it is necessary to configure
metadataServiceUri in bookkeeper.conf and bookkeeperMetadataServiceUri in broker.conf.

For example bookkeeper.conf

metadataServiceUri=zk://<zk_node_1>:2181/ledgers,zk://<zk_node_2>:2181/ledgers,zk://<zk_node_3>:2181/ledgers

The main purpose of this PR is to make it easier to compare the behavior of Pulsar when using Pulsar Metadata driver for Bookkeeper compared to using Zookeeper directly. It is not necessary make BK_METADATA_OPTIONS configurable to achieve this. However it's better to have an option to control it since that could prevent misconfiguration in cases where it is desired to not use Pulsar metadata driver.

The additional benefit of being able to override BK_METADATA_OPTIONS is that the Pulsar Metadata driver classes wouldn't get loaded at all when they aren't used.

Modifications

Make BK_METADATA_OPTIONS configurable.

Documentation

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-complete
    (Docs have been already added)

Matching PR in forked repository

PR in forked repository: lhotari#91

- use Bookkeeper defaults by setting BK_METADATA_OPTIONS=none
Copy link
Contributor

@Jason918 Jason918 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Technoboy- Technoboy- closed this Sep 27, 2022
@Technoboy- Technoboy- reopened this Sep 27, 2022
@lhotari lhotari merged commit 6dd38a4 into apache:master Sep 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc-not-needed Your PR changes do not impact docs ready-to-test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants