Move "beat" (ex-"shipper") config settings to be top level (#1570)

* Move "beat" (ex-shipper) config settings to be top level

Part of #1417. Done in a non-BWC way, because I think it's not worth
complicating the code. I'd rather have a script that migrates the
configuration.

* Make winlogbeat accept the new top-level settings

The list has grown quite long, switching to ucfg validation for this would be nice.
Also added explicit key names to config.

CHANGELOG.asciidoc

@@ -14,6 +14,9 @@ https://github.com/elastic/beats/compare/v5.0.0-alpha2...master[Check the HEAD d
 
 *Affecting all Beats*
 
+- All configuration settings under `shipper:` are moved to be top level configuration settings. I.e.
+  `shipper.name:` becomes `name:` in the configuration file. #1570
+
 *Packetbeat*
 
 *Topbeat*

filebeat/filebeat.yml

@@ -373,57 +373,56 @@ output:
     #pretty: false
 
 
-############################# Shipper #########################################
-
-shipper:
-  # The name of the shipper that publishes the network data. It can be used to group
-  # all the transactions sent by a single shipper in the web interface.
-  # If this options is not defined, the hostname is used.
-  #name:
-
-  # The tags of the shipper are included in their own field with each
-  # transaction published. Tags make it easy to group servers by different
-  # logical properties.
-  #tags: ["service-X", "web-tier"]
-
-  # Optional fields that you can specify to add additional information to the
-  # output. Fields can be scalar values, arrays, dictionaries, or any nested
-  # combination of these.
-  #fields:
-  #  env: staging
-
-  # If this option is set to true, the custom fields are stored as top-level
-  # fields in the output document instead of being grouped under a fields
-  # sub-dictionary. Default is false.
-  #fields_under_root: false
-
-  # Uncomment the following if you want to ignore transactions created
-  # by the server on which the shipper is installed. This option is useful
-  # to remove duplicates if shippers are installed on multiple servers.
-  #ignore_outgoing: true
-
-  # How often (in seconds) shippers are publishing their IPs to the topology map.
-  # The default is 10 seconds.
-  #refresh_topology_freq: 10
-
-  # Expiration time (in seconds) of the IPs published by a shipper to the topology map.
-  # All the IPs will be deleted afterwards. Note, that the value must be higher than
-  # refresh_topology_freq. The default is 15 seconds.
-  #topology_expire: 15
-
-  # Internal queue size for single events in processing pipeline
-  #queue_size: 1000
-
-  # Sets the maximum number of CPUs that can be executing simultaneously. The
-  # default is the number of logical CPUs available in the system.
-  #max_procs:
-
-  # Configure local GeoIP database support.
-  # If no paths are not configured geoip is disabled.
-  #geoip:
-    #paths:
-    #  - "/usr/share/GeoIP/GeoLiteCity.dat"
-    #  - "/usr/local/var/GeoIP/GeoLiteCity.dat"
+############################# General #########################################
+
+# The name of the shipper that publishes the network data. It can be used to group
+# all the transactions sent by a single shipper in the web interface.
+# If this options is not defined, the hostname is used.
+#name:
+
+# The tags of the shipper are included in their own field with each
+# transaction published. Tags make it easy to group servers by different
+# logical properties.
+#tags: ["service-X", "web-tier"]
+
+# Optional fields that you can specify to add additional information to the
+# output. Fields can be scalar values, arrays, dictionaries, or any nested
+# combination of these.
+#fields:
+#  env: staging
+
+# If this option is set to true, the custom fields are stored as top-level
+# fields in the output document instead of being grouped under a fields
+# sub-dictionary. Default is false.
+#fields_under_root: false
+
+# Uncomment the following if you want to ignore transactions created
+# by the server on which the shipper is installed. This option is useful
+# to remove duplicates if shippers are installed on multiple servers.
+#ignore_outgoing: true
+
+# How often (in seconds) shippers are publishing their IPs to the topology map.
+# The default is 10 seconds.
+#refresh_topology_freq: 10
+
+# Expiration time (in seconds) of the IPs published by a shipper to the topology map.
+# All the IPs will be deleted afterwards. Note, that the value must be higher than
+# refresh_topology_freq. The default is 15 seconds.
+#topology_expire: 15
+
+# Internal queue size for single events in processing pipeline
+#queue_size: 1000
+
+# Sets the maximum number of CPUs that can be executing simultaneously. The
+# default is the number of logical CPUs available in the system.
+#max_procs:
+
+# Configure local GeoIP database support.
+# If no paths are not configured geoip is disabled.
+#geoip:
+  #paths:
+  #  - "/usr/share/GeoIP/GeoLiteCity.dat"
+  #  - "/usr/local/var/GeoIP/GeoLiteCity.dat"
 
 
 ############################# Logging #########################################

filebeat/tests/system/config/filebeat.yml.j2

@@ -69,30 +69,28 @@ filebeat:
 
 
 ############################# Shipper ############################################
-shipper:
-
- # The name of the shipper that publishes the network data. It can be used to group
- # all the transactions sent by a single shipper in the web interface.
- # If this options is not defined, the hostname is used.
- name: {{shipperName}}
-
- # The tags of the shipper are included in their own field with each
- # transaction published. Tags make it easy to group servers by different
- # logical properties.
- tags: [
-    {%- if agent_tags -%}
-        {%- for tag in agent_tags -%}
-            "{{ tag }}"
-            {%- if not loop.last %}, {% endif -%}
-        {%- endfor -%}
-    {%- endif -%}]
-
-
- # Uncomment the following if you want to ignore transactions created
- # by the server on which the shipper is installed. This option is useful
- # to remove duplicates if shippers are installed on multiple servers.
- # ignore_outgoing: true
 
+# The name of the shipper that publishes the network data. It can be used to group
+# all the transactions sent by a single shipper in the web interface.
+# If this options is not defined, the hostname is used.
+name: {{shipperName}}
+
+# The tags of the shipper are included in their own field with each
+# transaction published. Tags make it easy to group servers by different
+# logical properties.
+tags: [
+{%- if agent_tags -%}
+	{%- for tag in agent_tags -%}
+		"{{ tag }}"
+		{%- if not loop.last %}, {% endif -%}
+	{%- endfor -%}
+{%- endif -%}]
+
+
+# Uncomment the following if you want to ignore transactions created
+# by the server on which the shipper is installed. This option is useful
+# to remove duplicates if shippers are installed on multiple servers.
+# ignore_outgoing: true
 
 
 ############################# Output ############################################

libbeat/beat/beat.go

@@ -111,11 +111,11 @@ type Beat struct {
 
 // BeatConfig struct contains the basic configuration of every beat
 type BeatConfig struct {
-	Output  map[string]*common.Config
-	Logging logp.Logging
-	Shipper publisher.ShipperConfig
-	Filters []filter.FilterConfig
-	Path    paths.Path
+	Shipper publisher.ShipperConfig   `config:",inline"`
+	Output  map[string]*common.Config `config:"output"`
+	Logging logp.Logging              `config:"logging"`
+	Filters []filter.FilterConfig     `config:"filters"`
+	Path    paths.Path                `config:"path"`
 }
 
 // Run initializes and runs a Beater implementation. name is the name of the

libbeat/cfgfile/cfgfile_test.go

@@ -83,7 +83,7 @@ func TestExpandEnv(t *testing.T) {
 		{`password: "abc$${!"`, `password: "abc${!"`, ""},
 
 		// The full error contains the line number.
-		{"shipper:\n  name: ${var", "", "failure while expanding environment " +
+		{"test:\n  name: ${var", "", "failure while expanding environment " +
 			"variables in config.yml at line=2, unterminated brace"},
 	}
 

libbeat/docs/shipperconfig.asciidoc

@@ -11,57 +11,56 @@
 //////////////////////////////////////////////////////////////////////////
 
 [[configuration-shipper]]
-=== Shipper
+=== Top level configuration options
 
-The `shipper` section contains configuration options for the Beat and some
+The top level section contains configuration options for the Beat and some
 general settings that control its behaviour.
 
 Here is an example configuration:
 
 [source,yaml]
 ------------------------------------------------------------------------------
-shipper:
-  # The name of the shipper that publishes the network data. It can be used to group
-  # all the transactions sent by a single shipper in the web interface.
-  # If this options is not defined, the hostname is used.
-  #name:
-
-  # The tags of the shipper are included in their own field with each
-  # transaction published. Tags make it easy to group servers by different
-  # logical properties.
-  tags: ["service-X", "web-tier"]
-
-  # Uncomment the following if you want to ignore transactions created
-  # by the server on which the shipper is installed. This option is useful
-  # to remove duplicates if shippers are installed on multiple servers.
-  ignore_outgoing: true
-
-  # How often (in seconds) shippers are publishing their IPs to the topology map.
-  # The default is 10 seconds.
-  refresh_topology_freq: 10
-
-  # Expiration time (in seconds) of the IPs published by a shipper to the topology map.
-  # All the IPs will be deleted afterwards. Note, that the value must be higher than
-  # refresh_topology_freq. The default is 15 seconds.
-  topology_expire: 15
-
-  # Configure local GeoIP database support.
-  # If no paths are not configured geoip is disabled.
-  #geoip:
-    #paths:
-    #  - "/usr/share/GeoIP/GeoLiteCity.dat"
-    #  - "/usr/local/var/GeoIP/GeoLiteCity.dat"
+# The name of the shipper that publishes the network data. It can be used to group
+# all the transactions sent by a single shipper in the web interface.
+# If this options is not defined, the hostname is used.
+#name:
+
+# The tags of the shipper are included in their own field with each
+# transaction published. Tags make it easy to group servers by different
+# logical properties.
+tags: ["service-X", "web-tier"]
+
+# Uncomment the following if you want to ignore transactions created
+# by the server on which the shipper is installed. This option is useful
+# to remove duplicates if shippers are installed on multiple servers.
+ignore_outgoing: true
+
+# How often (in seconds) shippers are publishing their IPs to the topology map.
+# The default is 10 seconds.
+refresh_topology_freq: 10
+
+# Expiration time (in seconds) of the IPs published by a shipper to the topology map.
+# All the IPs will be deleted afterwards. Note, that the value must be higher than
+# refresh_topology_freq. The default is 15 seconds.
+topology_expire: 15
+
+# Configure local GeoIP database support.
+# If no paths are not configured geoip is disabled.
+#geoip:
+#paths:
+#  - "/usr/share/GeoIP/GeoLiteCity.dat"
+#  - "/usr/local/var/GeoIP/GeoLiteCity.dat"
 
 ------------------------------------------------------------------------------
 
-==== Shipper Options
+==== Top Level Options
 
-You can specify the following options under the `shipper` section:
+You can specify the following options:
 
 ===== name
 
 The name of the Beat. If this option is empty, the `hostname` of the server is
-used. The name is included as the `shipper` field in each published transaction. You can
+used. The name is included as the `beat.name` field in each published transaction. You can
 use the name to group all transactions sent by a single Beat.
 
 At startup, each Beat can publish its IP, port, and name to Elasticsearch. This information
@@ -81,8 +80,7 @@ Example:
 
 [source,yaml]
 ------------------------------------------------------------------------------
-shipper:
-  name: "my-shipper"
+name: "my-shipper"
 ------------------------------------------------------------------------------
 
 ===== tags
@@ -97,8 +95,7 @@ Example:
 
 [source,yaml]
 --------------------------------------------------------------------------------
-shipper:
-  tags: ["my-service", "hardware", "test"]
+tags: ["my-service", "hardware", "test"]
 --------------------------------------------------------------------------------
 
 [[libbeat-configuration-fields]]
@@ -114,8 +111,7 @@ Example:
 
 [source,yaml]
 ------------------------------------------------------------------------------
-shipper:
-  fields: {project: "myproject", instance-id: "574734885120952459"}
+fields: {project: "myproject", instance-id: "574734885120952459"}
 ------------------------------------------------------------------------------
 
 ===== fields_under_root
@@ -129,11 +125,10 @@ Example:
 
 [source,yaml]
 ------------------------------------------------------------------------------
-shipper:
-  fields_under_root: true
-  fields:
-    instance_id: i-10a64379
-    region: us-east-1
+fields_under_root: true
+fields:
+  instance_id: i-10a64379
+  region: us-east-1
 ------------------------------------------------------------------------------
 
 ===== ignore_outgoing