Proposal: plugin mechanism for Dragonfly

[lowzj]

Why you need it?

More and more customized requirements that are not covered by Dragonfly come out.

• support different protocols(now only HTTP/HTTPS): feat: scheduler change piece cost from nanosecond to millisecond dragonfly#1119 (comment)
• pluggable strategy for file transfering and scheduling: Pluggable strategy for file transfering and scheduling #191
• separation of scheduler and cdnmgr: [feature request] The separation of scheduler and cdnmgr #437
• make supernode's CDN optional: [feature request] make supernode's CDN optional #356
• an area aware schedule policy to dragonfly: [feature request]an area aware schedule policy to dragonfly. #428
• different scheduler: [feature request]load the scheduleMgr by compile a file of plug-ins #1002
• different ways to get supernode list: [feature request] make the dfget to get the list of supernodes more flexible #1293
• add a new distribute pattern based on p2p: Proposal: Dragonfly supports seed pattern to solve massive proxy requests which may cause huge pressure on rss or supernode. #1284
• security requirements: [feature request] support configurable hashing and crypto algorithms for Dragonfly #1204

and so on.

Here is a simple plugin implementation in supernode: supernode/plugins.
But I think that it's not enough for the whole Dragonfly project including componentssupernode/dfget/dfdaemon. So there should be a plugin mechanism for Dragonfly, and put its implementation into pkg package.

How it could be?

Here is a document comparing the 3 ways to implement plugin:

• vendor the core code
• golang-plugins
• hashicorp/go-plugin: https://github.com/hashicorp/go-plugin

Dragonfly can select a suitable solution to implement as its foundation for users to implements their own plugins conveniently.

Based on this, Dragonfly abstracts some pluggable components in the next step.
Furthermore, a new individual project dragonflyoss/plugins could be created for users to submit their plugin implementation which cannot be merged into the main project for the time being but hope to be developed in the community.

Other related information

[inoc603]

I think before we choose a plugin mechanism, we should refactor our code to identify the parts where customization is necessary, and make them configurable in code first.

For example, if we need adding customizable protocols in dfdaemon, we should first define the interface for a custom protocol, and probably implement http/https as one.

type Protocol interface {
  // ...
}

func WithProtocol(p Protocol) Option {
  // ...
}

Then we can define the plugin interface for users to customize. Take go plugin as an example, we may ask the user to provide a specific function in their plugin

func ProtocolFactory(config map[string]interface{}) Protocol

Then for plugins that would be widely used, we can implement them in dragonfly. For other more specific requirements, the user may choose to implement them as plugin or fork the project, and both methods will use the same interface in code.

[lowzj]

I think before we choose a plugin mechanism, we should refactor our code to identify the parts where customization is necessary, and make them configurable in code first.

@inoc603 agreed. It's necessary to define an interface for a component before we make it pluggable and configurable.

But I think that the definition interface of pluggable components and the common implementation of plugin mechanism can be executed in parallel.

The plugin mechanism should provide these features:

• provide a way for Dragonfly to select a specific implementation of a plugin
• provide a way for user to register their own implementation of a plugin
• provide a way to construct a plugin with configuration

And Dragonfly components can use this mechanism to make themself pluggable in a unified way.

[lowzj]

I will create a project here to list and track tasks about interface definition.

[inoc603]

I think before we choose a plugin mechanism, we should refactor our code to identify the parts where customization is necessary, and make them configurable in code first.

@inoc603 agreed. It's necessary to define an interface for a component before we make it pluggable and configurable.

But I think that the definition interface of pluggable components and the common implementation of plugin mechanism can be executed in parallel.

The plugin mechanism should provide these features:

• provide a way for Dragonfly to select a specific implementation of a plugin
• provide a way for user to register their own implementation of a plugin
• provide a way to construct a plugin with configuration

And Dragonfly components can use this mechanism to make themself pluggable in a unified way.

For dynamically loaded plugins, I think we can use a very simple configuration:

# a list of plugins that will be dynamically loaded
plugins:
-
  # type of the protocol, should be one of the plugin types provided by dragonfly
  type: protocol
  # name of the plugin
  name: nfs
  # how to load the plugin
  # take go plugin as an example, we need the path of the plugin
  path: /path/to/plugin
  # configuration for the plugin
  args:
    # any valid yaml

[lowzj]

For dynamically loaded plugins, I think we can use a very simple configuration:

# a list of plugins that will be dynamically loaded
plugins:
-
  # type of the protocol, should be one of the plugin types provided by dragonfly
  type: protocol
  # name of the plugin
  name: nfs
  # how to load the plugin
  # take go plugin as an example, we need the path of the plugin
  path: /path/to/plugin
  # configuration for the plugin
  args:
    # any valid yaml

Where should the configurations be placed? An individual file or integrated with current supernod.yml/dfget.yml/dfdaemon.yml?

[zcc35357949]

Is there any further updates for this issue?