[feature request] provide a more flexible way to pass parameter from dfdaemon to dfget

[yeya24]

Why you need it?

Currently we just directly pass the parameters from dfdaemon to dfget. However these flags are global and sometimes different proxies may need different parameters such as timeout or urlfilters.

We should provide a more flexible way to construct dfget parameters in dfdaemon.

How it could be?

Other related information

Related:
dragonflyoss/dragonfly#670
dragonflyoss/dragonfly#686

[starnop]

Maybe we can add a string field for dfdaemon such as dfgetParam which will be passed to dfget just as @lowzj mentioned here.

In addition, we can set different params for different proxy regex by config file /etc/dragonfly/dfdaemon.yml. Just like as follows:

globalDfgetParams:
  - key1: value1
  - key2: value2

# Requests that match the regular expressions will be proxied with dfget,
# otherwise they'll be proxied directly. Requests will be handled by the first
# matching rule.
proxies:
  # proxy all http image layer download requests with dfget
- regx: blobs/sha256.*
  dfgetParams: 
    - key1: value1
    - key2: value2

......

@lowzj @yeya24 WDYT?

[lowzj]

The dfgetParam of a proxy inherits and overrides the flags of dfdaemon passed to dfget.

[yeya24]

Do we need that key-value map? What about just using k-v pair like minrate: 20k?

IMO if we want to use config files to pass variables, is it confusing to still pass variables from dfdaemon flags? Maybe we can add a field of globalParams in configuration and remove most of flags from dfdaemon.

[starnop]

Do we need that key-value map? What about just using k-v pair like minrate: 20k?

I agree with that.

[starnop]

IMO if we want to use config files to pass variables, is it confusing to still pass variables from dfdaemon flags? Maybe we can add a field of globalParams in configuration and remove most of flags from dfdaemon.

Good suggestion. A globalParams is necessary and I think it's better to temporarily reserve the way to pass parameters via flag. WDYT?

So let's do these things next:

1. use config file to pass the values to dfget firstly and merge the configs from config file and flags.
2. provide a config template and with a script that automatically generates it

@lowzj @yeya24 Any other supplements？

[lowzj]

agree

[starnop]

@yeya24 Would you like to do that? 😄

[yeya24]

Sure. I'd like to work on it.

[yeya24]

I have a question about how to deal with the config files between dfdaemon and dfget. If we set some dfgetParams, how we deal with dfget.yml?

[lowzj]

The properties of dfget from CLI flags have a higher priority than config files.
The priority of properties that dfdfaemon passed to dfget:

lower  | 1. `globalParams` in `dfdaemon` config file
       | 2. `dfdaemon` flags, they will override the `globalParams`
higher | 3. `dfgetParams` of every `proxy` property in `dfdaemon` config file

[lowzj]

I thinks this feature has already been implemented. I will close it, if not please reopen again.