Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat][broker]PIP-180 Shadow Topic - Part V - Support shadow topic creation. #17711

Merged
merged 2 commits into from
Oct 3, 2022
Merged

[feat][broker]PIP-180 Shadow Topic - Part V - Support shadow topic creation. #17711

merged 2 commits into from
Oct 3, 2022

Conversation

Jason918
Copy link
Contributor

@Jason918 Jason918 commented Sep 19, 2022 •
edited
Loading

Master Issue: #16153

Motivation

The main purpose of this part is to support shadow topic creation.

Note:

  1. A topic must defined as shadow topic at creation, or else it will create its own ledgers. So this PR uses topic property(key="PULSAR.SHADOW_SOURCE", value="$SourceTopic") to identify the topic is shadow topic.
  2. shadow topic can't change to non-shadow or change its shadow source in its whole life time. You must delete it first and recreate.
  3. Deleting an shadow topic is just like deleting normal topic.

Modifications

Key changes:

  1. Read topic properties from storage before loading a topic, see checkOwnershipAndCreatePersistentTopic
  2. Add admin interface and cli for createShadowTopic and getShadowSource
  3. Add ShadowManagedLedgerImpl, but it's not implemented yet. It's quite complicated and it will be implemented in the following PRs.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • ShadowTopicTest

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • Anything that affects deployment

Documentation

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    Java doc added in pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/Topics.java

  • doc-complete
    (Docs have been already added)

Matching PR in forked repository

PR in forked repository:Jason918#7

@Jason918 Jason918 requested review from eolivelli, codelipenghui and mattisonchao and removed request for eolivelli September 19, 2022 04:25
@Jason918 Jason918 self-assigned this Sep 19, 2022
@github-actions github-actions bot added the doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. label Sep 19, 2022
@Jason918 Jason918 added type/feature The PR added a new feature or issue requested a new feature area/broker and removed doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. labels Sep 19, 2022
@Jason918 Jason918 added this to the 2.12.0 milestone Sep 19, 2022
@github-actions github-actions bot added the doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. label Sep 19, 2022
@Jason918
Copy link
Contributor Author

/pulsarbot run-failure-checks

@Jason918 Jason918 requested a review from AnonHxy September 22, 2022 08:53
@Jason918 Jason918 mentioned this pull request Sep 23, 2022
Closed
mattisonchao
mattisonchao reviewed Sep 26, 2022
View reviewed changes
Copy link
Member

@mattisonchao mattisonchao left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, thanks for your great work!
I left some comments. and I also have a question, hope you can answer.

Question:
Sorry for missing previous PRs. Can we shadow the shadowed topic?

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java Outdated Show resolved Hide resolved
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java Outdated Show resolved Hide resolved
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java Outdated Show resolved Hide resolved
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java Outdated Show resolved Hide resolved
mattisonchao
mattisonchao reviewed Sep 26, 2022
View reviewed changes
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java
return MapUtils.getString(properties, PROPERTY_SOURCE_TOPIC_KEY);
}

public static final String PROPERTY_SOURCE_TOPIC_KEY = "PULSAR.SHADOW_SOURCE";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uppercase or lowercase? I remember the replicator being lowercase. But I'm not sure which is better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would appear in topic property. I think uppercase should cause less conflict.

@Jason918
Copy link
Contributor Author

Can we shadow the shadowed topic?

@mattisonchao
In theory we can, BUT not now, because ShadowManagedLedger will watch the managed ledger metadata info of source topic which contains all the ledger ids.
If we want to support shadow the shadow topic, we need to find the "real" source topic to watch.
As for shadow replicator, it should be working properly in a shadow topic.

@Jason918
Copy link
Contributor Author

@AnonHxy @mattisonchao PTAL

@AnonHxy
Copy link
Contributor

AnonHxy commented Sep 26, 2022

LGTM

mattisonchao
mattisonchao approved these changes Sep 27, 2022
View reviewed changes
Copy link
Member

@mattisonchao mattisonchao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Jason918
Copy link
Contributor Author

Jason918 commented Oct 2, 2022

/pulsarbot run-failure-checks

@Jason918 Jason918 merged commit 1148204 into apache:master Oct 3, 2022
@rdhabalia rdhabalia mentioned this pull request Oct 8, 2022
Merged
4 tasks
rdhabalia
rdhabalia reviewed Oct 8, 2022
View reviewed changes
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ShadowManagedLedgerImpl.java

@Override
synchronized void initialize(ManagedLedgerInitializeLedgerCallback callback, Object ctx) {
// TODO: ShadowManagedLedger has different initialize process from normal ManagedLedger,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should avoid TODOs and try to implement full PR or make it extendable PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should avoid TODOs and try to implement full PR or make it extendable PR.

@rdhabalia This would make PIP-180 a very huge PR and very hard to review. The next PR is coming up and have implemented this.

@rdhabalia
Copy link
Contributor

this PR introduced test failure:
FIX: #17965

@liangyepianzhou liangyepianzhou mentioned this pull request May 11, 2023
Merged
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rdhabalia rdhabalia rdhabalia left review comments

@mattisonchao mattisonchao mattisonchao approved these changes

@AnonHxy AnonHxy AnonHxy approved these changes

@codelipenghui codelipenghui Awaiting requested review from codelipenghui

@eolivelli eolivelli Awaiting requested review from eolivelli

Assignees

@Jason918 Jason918

Labels
area/broker doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. ready-to-test type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Milestone
3.0.0
Development

Successfully merging this pull request may close these issues.
4 participants
@Jason918 @AnonHxy @rdhabalia @mattisonchao