Adding ability to expand environment variables used in config file

[andrewkroh]

Adds feature requested in #114. If we choose to accept this enhancement then the documentation will need updated.

Overview

This feature replaces ${var} or $var in config files according to the values of the current environment variables.

• The replacement is case-sensitive and occurs before the YAML is parsed.
• References to undefined variables are replaced by an empty string.
• A default value can be given by using the form ${var:default value}.

Examples

Input	Environment	Output
name: $NAME	export NAME=elastic	name: elastic
name: ${NAME}	export NAME=elastic	name: elastic
name: ${NAME:beats}		name: beats
name: ${NAME:beats}	export NAME=elastic	name: elastic
hosts: [$HOSTS]	export HOSTS="'localhost:9200', 'localhost:9202'"	hosts: ['localhost:9200', 'localhost:9202']

Troubleshooting

When logging is enabled at the info level or higher, a message is logged for each environment variable replacement. For example:

2016/01/13 23:00:27.925523 cfgfile.go:87: INFO Replacing config environment variable '${NAME}' with 'elastic'
2016/01/13 23:00:27.925786 cfgfile.go:87: INFO Replacing config environment variable '${HOSTS}' with ''localhost:9200', 'localhost:9202''

[ruflin]

Is there a reason we can't use the function provided by golang? https://golang.org/pkg/os/#ExpandEnv

[andrewkroh]

os.ExpandEnv(key) is os.Expand(key, os.Getenv) and that's what this does, but with two differences. 1) It logs each replacement which is very important for end-user troubleshooting. 2) It supports setting a default value if you use the form ${var:default value}.

[ruflin]

That is great, especially the logging. Makes later debugging much easier.

[tsg]

I'm generally +1 on this one. I also like the other Elasticsearch config option, which is to overwrite variables from CLI with something like -Dshipper.name=test. Perhaps we only want one of these two features, since I think they solve the same problem, but the -D one is probably more difficult to implement.

[ruflin]

@tsg +1 on the -D option. I think there are 2 different use cases: Environment variables for larger deployments, client flags more for testing. Of course, one can compensate for the other.

[yoanisgil]

Just curious, what's the ETA for this to be publicly available?

[ruflin]

@yoanisgil The best option to already use and test this feature is using our nightly builds: https://beats-nightlies.s3.amazonaws.com/index.html

[yoanisgil]

@ruflin thanks for the update. I guess I could give it a try but I was just wondering whether this is ready for production usage or not (hence my question about an ETA for public availability).

[ruflin]

@yoanisgil It will be part of the next major release, but it is hard to give an exact date. The nightly builds are quite stable, but of course I would not recommend to run them in production, as we still heavily change stuff on a daily base (and sometimes break stuff).

[tony2944]

Hi
Whether the variable subsitution can be used for
-path /opt/logs/aa.log.${date_param} like this.