Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PIP 110: Topic metadata #12629

Closed
Technoboy- opened this issue Nov 4, 2021 · 4 comments · Fixed by #12818
Closed

PIP 110: Topic metadata #12629

Technoboy- opened this issue Nov 4, 2021 · 4 comments · Fixed by #12818
Assignees
Labels
Milestone

Comments

@Technoboy-
Copy link
Contributor

Motivation

The original discussion mail :
https://lists.apache.org/thread/m9dkhq1fs6stsdwh78h84fsl5hs5v67f

Introduce the ability to store metadata about topics.

This would be very useful as with metadata you could add labels and other
pieces of information that would allow defining the purpose of a topic,
custom application-level properties.
This feature will allow application-level diagnostic tools and maintenance
tools to not need external databases to store such metadata.

Imagine that we could add a simple key value map (String keys and String
values) to the topic.
These metadata could be set during topic creation and also updated.

API Changes

We need to add a map type field called metadata to the below methods signature:

  • PersistentTopics#createPartitionedTopic
    PersistentTopics#createPartitionedTopic(AsyncResponse asyncResponse, String tenant, String namespace,String encodedTopic,int numPartitions,boolean createLocalTopicOnly, Map<String, String> metadata);  
    
  • PersistentTopics#createNonPartitionedTopic
    PersistentTopics#createNonPartitionedTopic(String tenant, String namespace, String encodedTopic, boolean authoritative, Map<String, String> metadata); 
    

Implementation

  • For PartitionedTopic
    We will put these metadata to the PartitionedTopicMetadata, serialized to store in configurationMetadataStore, like partitioned-topic partitions.

    public class PartitionedTopicMetadata {
    
     /* Number of partitions for the topic */
     public int partitions;
     
     /* Topic metadata */
     public Map<String, String> metadata;
    
  • For NonPartitionedTopic
    We will store the metadata to ML(ManagedLedger) which already supports storing custom properties.

    ManagedLedger#setProperties(Map<String, String> properties) 
    

Compatibility

The proposal will not introduce any compatibility issues.

Tests Plan

Unit tests & integration tests

@Technoboy- Technoboy- self-assigned this Nov 4, 2021
@Technoboy- Technoboy- changed the title PIP : Topic metadata PIP110: Topic metadata Nov 4, 2021
@Technoboy- Technoboy- changed the title PIP110: Topic metadata PIP 110: Topic metadata Nov 4, 2021
@codelipenghui
Copy link
Contributor

LGTM +1

@eolivelli
Copy link
Contributor

We should prevent the user to set properties on individual partitions in a partitioned topic (because partitions are like topic for many aspects)

Please also explain the permissions needed to update such properties (and add authz coverage in the test cases )

@eolivelli
Copy link
Contributor

@Technoboy- did you see my comment?

@Technoboy-
Copy link
Contributor Author

@Technoboy- did you see my comment?

Ok, I will update this in the pr.

codelipenghui pushed a commit that referenced this issue Jan 25, 2022
…#12818)

Fixes #12629

## Motivation
The original discussion mail :
https://lists.apache.org/thread/m9dkhq1fs6stsdwh78h84fsl5hs5v67f

Introduce the ability to store metadata about topics.

This would be very useful as with metadata you could add labels and other
pieces of information that would allow defining the purpose of a topic,
custom application-level properties.
This feature will allow application-level diagnostic tools and maintenance
tools to not need external databases to store such metadata.

Imagine that we could add a simple key value map (String keys and String
values) to the topic.
These metadata could be set during topic creation and also updated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants