Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow BalanceStrategy to provide custom assignment data #1592

Merged
merged 3 commits into from
Feb 4, 2020
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions balance_strategy.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,10 @@ type BalanceStrategy interface {
// Plan accepts a map of `memberID -> metadata` and a map of `topic -> partitions`
// and returns a distribution plan.
Plan(members map[string]ConsumerGroupMemberMetadata, topics map[string][]int32) (BalanceStrategyPlan, error)

// AssignmentData returns the serialized assignment data for the specified
// memberID
AssignmentData(plan BalanceStrategyPlan, memberID string, generationID int32) ([]byte, error)
}

// --------------------------------------------------------------------
Expand Down Expand Up @@ -132,6 +136,11 @@ func (s *balanceStrategy) Plan(members map[string]ConsumerGroupMemberMetadata, t
return plan, nil
}

// AssignmentData simple strategies do not require any shared assignment data
func (s *balanceStrategy) AssignmentData(plan BalanceStrategyPlan, memberID string, generationID int32) ([]byte, error) {
return nil, nil
}

type balanceStrategySortable struct {
topic string
memberIDs []string
Expand Down Expand Up @@ -268,6 +277,19 @@ func (s *stickyBalanceStrategy) Plan(members map[string]ConsumerGroupMemberMetad
return plan, nil
}

// AssignmentData serializes the set of topics currently assigned to the
// specified member as part of the supplied balance plan
func (s *stickyBalanceStrategy) AssignmentData(plan BalanceStrategyPlan, memberID string, generationID int32) ([]byte, error) {
topics, ok := plan[memberID]
if !ok {
return nil, nil
}
return encode(&StickyAssignorUserDataV1{
Topics: topics,
Generation: generationID,
}, nil)
}

func strsContains(s []string, value string) bool {
for _, entry := range s {
if entry == value {
Expand Down
16 changes: 5 additions & 11 deletions consumer_group.go
Original file line number Diff line number Diff line change
Expand Up @@ -331,20 +331,14 @@ func (c *consumerGroup) syncGroupRequest(coordinator *Broker, plan BalanceStrate
MemberId: c.memberID,
GenerationId: generationID,
}
strategy := c.config.Consumer.Group.Rebalance.Strategy
for memberID, topics := range plan {
assignment := &ConsumerGroupMemberAssignment{Topics: topics}

// Include topic assignments in group-assignment userdata for each consumer-group member
if c.config.Consumer.Group.Rebalance.Strategy.Name() == StickyBalanceStrategyName {
userDataBytes, err := encode(&StickyAssignorUserDataV1{
Topics: topics,
Generation: generationID,
}, nil)
if err != nil {
return nil, err
}
assignment.UserData = userDataBytes
userDataBytes, err := strategy.AssignmentData(plan, memberID, generationID)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering why you'd need to send the plan and member to AssignmentData to get the topics:

	topics, ok := plan[memberID]
	if !ok {
		return nil, nil
	}

if that was already obtained in the range above?
Line 335 for memberID, topics := range plan

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking here was in the case the state being shipped needed to understand data about the larger plan, but I am happy limiting the scope to just the given member and its assigned topics.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the fast response!
I'll 👀 later today

if err != nil {
return nil, err
}
assignment.UserData = userDataBytes
if err := req.AddGroupAssignmentMember(memberID, assignment); err != nil {
return nil, err
}
Expand Down