Kafka Resource for Simplified Routine Load Broker Management

[jselvareef]

Feature request

Is your feature request related to a problem? Please describe.
Currently, when creating a routine load in StarRocks to consume data from a Kafka topic, we must specify the broker_list property directly in the routine load. This creates significant overhead when managing multiple routine loads.

For instance, with 20 or more routine loads, the same broker_list must be repeated in all of them. If maintenance occurs in Kafka, such as adding or removing brokers, we must update the broker_list in each of these routine loads individually. This is inefficient and prone to error.

Describe the solution you'd like
Introduce a new type of "RESOURCE" where type=kafka. This resource would include properties like broker_list. When creating a routine load, instead of specifying the broker_list, users would reference this Kafka resource.

This approach centralizes the management of the broker_list in the Kafka resource. If updates are needed, they can be made in a single place without requiring changes to every routine load that relies on it.

Describe alternatives you've considered

• Manually updating the broker_list in every routine load after Kafka maintenance. However, this is not scalable or practical when dealing with a large number of routine loads.

• Writing automation scripts to update routine loads programmatically. This adds complexity and requires external tooling, which could introduce additional failure points.

Additional context
This feature would significantly improve the user experience for managing Kafka-based routine loads in environments with high routine load counts. It would also reduce the risk of misconfiguration during Kafka maintenance activities.