diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 2957c0850f4..d7a19ffe10a 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -69,6 +69,9 @@ https://github.com/elastic/beats/compare/v5.0.0-rc1...5.0[Check the HEAD diff] //////////////////////////////////////////////////////////// + +include::libbeat/docs/release-notes/5.0.0.asciidoc[] + [[release-notes-5.0.0-ga]] === Beats version 5.0.0-GA https://github.com/elastic/beats/compare/v5.0.0-rc1...v5.0.0[View commits] diff --git a/libbeat/docs/release-notes/5.0.0.asciidoc b/libbeat/docs/release-notes/5.0.0.asciidoc new file mode 100644 index 00000000000..68169824bc8 --- /dev/null +++ b/libbeat/docs/release-notes/5.0.0.asciidoc @@ -0,0 +1,240 @@ +[[release-notes-5.0.0]] +=== Beats version 5.0.0 + +The list below covers changes between 1.x to 5.0.0 releases. + + +==== Breaking changes + +*Affecting all Beats* + +- Rename the `filters` section to `processors`. {pull}1944[1944] +- Require braces for environment variable expansion in config files {pull}1304[1304] +- On DEB/RPM installations, the binary files are now found under `/usr/share/{{beat_name}}/bin`, not in `/usr/bin`. {pull}1385[1385] +- The logs are written by default to self rotating files, instead of syslog. {pull}1371[1371] +- Remove deprecated `host` option from elasticsearch, logstash and redis outputs. {pull}1474[1474] +- All configuration settings under `shipper:` are moved to be top level configuration settings. I.e. + `shipper.name:` becomes `name:` in the configuration file. {pull}1570[1570] +- The `topology_expire` option of the Elasticserach output was removed. {pull}1907[1907] +- The Elasticsearch template is now loaded by default. {pull}1993[1993] +- The Redis output `index` setting is renamed to `key`. `index` still works but it's deprecated. {pull}2077[2077] +- The undocumented file output `index` setting was removed. Use `filename` instead. {pull}2077[2077] +- Change Elasticsearch output index configuration to be based on format strings. If index has been configured, no date will be appended anymore to the index name. {pull}2119[2119] +- If the path specified by the `-c` flag is not absolute and `-path.config` is not specified, it + is considered relative to the current working directory. {pull}2245[2245] +- Rename `tls` configurations section to `ssl`. {pull}2330[2330] +- Rename `certificate_key` configuration to `key`. {pull}2330[2330] +- Replace `tls.insecure` with `ssl.verification_mode` setting. {pull}2330[2330] +- Replace `tls.min/max_version` with `ssl.supported_protocols` setting requiring full protocol name. {pull}2330[2330] +- A dynamic mapping rule is added to the default Elasticsearch template to treat strings as keywords by default. {pull}2688[2688] + + +*Filebeat* + +- Scalar values in used in the `fields` configuration setting are no longer automatically converted to strings. {pull}1092[1092] +- Count field was removed from event as not used in filebeat {issue}778[778] +- Default location for the registry file was changed to be `data/registry` from the binary directory, + rather than `.filebeat` in the current working directory. This affects installations for zip/tar.gz/source, + the location for DEB and RPM packages stays the same. {pull}1373[1373] +- The state for files which fall under ignore_older is not stored anymore. This has the consequence, that if a file which fell under ignore_older is updated, the whole file will be crawled. +- Ignore symlinks by default, but they can be enabled with `symlinks` config {pull}1686[1686] {pull}2478[2478] +- Set `close_inactive` default to 5 minutes (was 1 hour before) +- Set `clean_removed` and `close_removed` to true by default + + +*Packetbeat* + +- Rename output fields in the dns package. Former flag `recursion_allowed` becomes `recursion_available`. {pull}803[803] + Former SOA field `ttl` becomes `minimum`. {pull}803[803] +- The fully qualified domain names which are part of output fields values of the dns package now terminate with a dot. {pull}803[803] +- Remove the `count` field from the exported event {pull}1210[1210] +- Configuration of redis topology support changed. {pull}1353[1353] +- Move all Packetbeat configuration options under the packetbeat namespace {issue}1417[1417] +- Set `enabled` ` in `packetbeat.protocols.icmp` configuration to `true` by default. {pull}1988[1988] +- Group HTTP fields under `http.request` and `http.response` {pull}2167[2167] +- Export `http.request.body` and `http.response.body` when configured under `include_body_for` {pull}2167[2167] +- Move `ignore_outgoing` config to `packetbeat.ignore_outgoing` {pull}2393[2393] + +*Winlogbeat* + +- The `message_inserts` field was replaced with the `event_data` field {issue}1053[1053] +- The `category` field was renamed to `task` to better align with the Windows Event Log API naming {issue}1053[1053] +- Remove the `count` field from the exported event {pull}1218[1218] + + +==== Bugfixes + +*Affecting all Beats* + +- Logstash output will not retry events that are not JSON-encodable {pull}927[927] +- Drain response buffers when pipelining is used by Redis output. {pull}1353[1353] +- Unterminated environment variable expressions in config files will now cause an error {pull}1389[1389] +- Fix issue with the automatic template loading when Elasticsearch is not available on Beat start. {issue}1321[1321] +- Fix bug affecting `-cpuprofile`, `-memprofile`, and `-httpprof` CLI flags {pull}1415[1415] +- Fix race when multiple outputs access the same event with logstash output manipulating event {issue}1410[1410] {pull}1428[1428] +- Seed random number generator using crypto.rand package. {pull}1503{1503] +- Fix beats hanging in `-configtest` {issue}1213[1213] +- Reset backoff factor on partial ACK. {issue}1803[1803] +- Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. {issue}1829[1829] +- Fix logstash output with pipelining mode enabled not reconnecting. {issue}1876[1876] +- Empty configuration sections become merge-able with variables containing full path. {pull}1900[1900] +- Fix error message about required fields missing not printing the missing field name. {pull}1900[1900] +- Fix sync publisher `PublishEvents` return value if client is closed concurrently. {pull}2046[2046] +- Fix Logstash output handles error twice when asynchronous sending fails. {pull}2441[2441] +- Fix Elasticsearch structured error response parsing error. {issue}2229[2229] +- Fixed the run script to allow the overriding of the configuration file. {issue}2171[2171] +- Fix logstash output crash if no hosts are configured. {issue}2325[2325] +- Fix beats failing to start due to invalid duplicate key error in configuration file. {pull}2521[2521] +- Fix panic on non writable logging directory. {pull}2571[2571] +- Fix ignoring all fields from drop_fields in case the first field is unknown. {pull}2685[2685] +- Fix dynamic configuration int/uint` to float type conversion. {pull}2698[2698] +- Fix primitive types conversion if values are read from environment variables. {pull}2698[2698] + + +*Filebeat* + +- Stop filebeat if started without any prospectors defined or empty prospectors {pull}644[644] {pull}647[647] +- Improve shutdown of crawler and prospector to wait for clean completion {pull}720[720] +- Omit `fields` from Filebeat events when null {issue}899[899] +- Improvements in registrar dealing with file rotation. {pull}1281[1281] +- Multiline reader normalizing newline to use `\n`. {pull}1552[1552] +- Fix potential data loss between Filebeat restarts, reporting unpublished lines as published. {issue}2041[2041] +- Fix open file handler issue. {issue}2028[2028] {pull}2020[2020] +- Fix async publisher sending empty events {pull}2455[2455] +- Fix potential issue with multiple harvester per file on large file numbers or slow output {pull}2541[2541] +- Fix input buffer on encoding problem. {pull}2416[2416] +- Fix issue when `clean_removed` and `clean_inactive` were used together that states were not directly removed from the registry. +- Fix issue where upgrading a 1.x registry file resulted in duplicate state entries. {pull}2792[2792] + + +*Packetbeat* + +- Create a proper BPF filter when ICMP is the only enabled protocol {issue}757[757] +- Check column length in pgsql parser. {issue}565[565] +- Harden pgsql parser. {issue}565[565] +- Add missing nil-check to memcached GapInStream handler. {issue}1162[1162] +- Fix NFSv4 Operation returning the first found first-class operation available in compound requests. {pull}1821[1821] +- Fix TCP overlapping segments not being handled correctly. {pull}1898[1898] +- Fix mapping for some Packetbeat flow metrics that were not marked as being longs. {issue}2177[2177] +- Fix handling of messages larger than the maximum message size (10MB). {pull}2470[2470] +- Fix the `bpf_filter` setting. {issue}2660[2660] +- Fix compile issues for OpenBSD. {pull}1347[1347] + + +*Winlogbeat* + +- Fix panic when reading messages larger than 32K characters on Windows XP and 2003. {pull}1498[1498] +- Fix panic that occurs when reading a large events on Windows Vista and newer. {pull}1499[1499] +- Adding missing argument to the "Stop processing" log message. {pull}1590[1590] +- Fix issue with rendering forwarded event log records. {pull}1891[1891] +- Fix potential data loss between Winlogbeat restarts, reporting unpublished lines as published. {issue}2041[2041] +- Fix corrupt registry file that occurs on power loss by disabling file write caching. {issue}2313[2313] + + + +==== Added + +*Affecting all Beats* + +- Update to Go 1.7. {pull}2306[2306] +- Add option to Elasticsearch output to pass http parameters in index operations {issue}805[805] +- Improve Logstash and Elasticsearch backoff behavior. {pull}927[927] +- Add Kafka output. {pull}942[942] {pull}2188[2188] {pull}2190[2190] {pull}2284[2284] +- Add config file option to configure GOMAXPROCS. {pull}969[969] +- Improve shutdown handling in libbeat. {pull}1075[1075] +- Log total non-zero internal metrics on shutdown. {pull}2349[2349] +- Add `fields` and `fields_under_root` options under the `shipper` configuration {pull}1092[1092] +- Add the ability to use a SOCKS5 proxy with the Logstash output {issue}823[823] +- The `-configtest` flag will now print "Config OK" to stdout on success {pull}1249[1249] +- Add support for TLS to Redis output. {pull}1353[1353] +- Add SOCKS5 proxy support to Redis output. {pull}1353[1353] +- Failover and load balancing support in Redis output. {pull}1353[1353] +- Multiple-worker per host support for Redis output. {pull}1353[1353] +- Added ability to escape `${x}` in config files to avoid environment variable expansion {pull}1389[1389] +- Configuration options and CLI flags for setting the home, data and config paths. {pull}1373[1373] +- Configuration options and CLI flags for setting the default logs path. {pull}1437[1437] +- Add Elasticsearch template files compatible with Elasticsearch 2.x. {pull}1501[1501] +- Load the mapping template depending on the Elasticsearch version. {pull}1993[1993] +- Add conditions to processors. {pull}1623[1623] +- Enhance `contains` condition to work on fields that are arrays of strings. {issue}2237[2237] +- Add OR/AND/NOT to the condition associated with the processors. {pull}1983[1983] +- Improve error message if compiling regular expression from config files fails. {pull}1900[1900] +- Compression support in the Elasticsearch output. {pull}1835[1835] +- Periodically log internal metrics. {pull}1955[1955] +- Add `enabled` setting to all output modules. {pull}1987[1987] +- Command line flag `-c` can be used multiple times. {pull}1985[1985] +- Add `-E` CLI flag for overwriting single config options via command line. {pull}1986[1986] +- Check stdout being available when console output is configured. {issue}2035[2035] +- Add script to generate the Kibana index-pattern from fields.yml. {pull}2122[2122] +- Enhance Redis output key selection based on format string. {pull}2169[2169] +- Configurable Redis `keys` using filters and format strings. {pull}2169[2169] +- Lookup the configuration file relative to the `-path.config` CLI flag. {pull}2245[2245] +- Re-write `import_dashboards` script in Golang. {pull}2155[2155] +- Add support for encrypted private key files by introducing `ssl.key_passphrase` setting. {pull}2330[2330] +- Add `beat.version` fields to all events. +- Make sure Beats sent always float values when they are defined as float by sending 5.00000 instead of 5. {pull}2627[2627] + + +*Filebeat* + +- Add the ability to set a list of tags for each prospector {pull}1092[1092] +- Add JSON decoding support {pull}1143[1143] +- The registry format was changed to an array instead of dict. The migration to the new format will happen automatically at the first startup. {pull}1703[1703] +- Introduce `close_removed` and `close_renamed` harvester options. {issue}1600[1600] +- Introduce `close_eof` harvester option. {issue}1600[1600] +- Add `clean_removed` and `clean_inactive` config option. {issue}1600[1600] +- Introduce `close_timeout` harvester options {issue}1926[1926] +- Strip BOM from first message in case of BOM files {issue}2351[2351] +- Add `harvester_limit` option {pull}2417[2417] + + +*Packetbeat* + +- Change the DNS library used throughout the dns package to github.com/miekg/dns. {pull}803[803] +- Add support for NFS v3 and v4. {pull}1231[1231] +- Add support for EDNS and DNSSEC. {pull}1292[1292] +- Add `enabled` setting to Packetbeat protocols. {pull}1988[1988] +- Add `enabled` setting to Packetbeat network flows configuration. {pull}1988[1988] +- Add Cassandra protocol analyzer to Packetbeat. {pull}1959[1959] +- Match connections with IPv6 addresses to processes {pull}2254[2254] +- Add IP address to `-devices` command output {pull}2327[2327] +- Add configuration option for the maximum message size. Used to be hard-coded to 10 MB. {pull}2470[2470] + +*Metricbeat* + +- First public release, containing the following modules: system, Apache, MySQL, PostgreSQL, Nginx, Redis, MongoDB, and Zookeeper. + Metricbeat is replacing Topbeat in 5.0, and its default behaviour is to export system statistics like CPU usage, + memory usage, Disk IO similar with what Topbeat 1.x is doing. +- Add `pgid` field to process information. {pull} 2021[2021] +- Use the new `scaled_float` Elasticsearch type for the percentage values. {pull}2156[2156] +- Improve mapping by converting `half_float` to `scaled_float` and integers to long. {pull}2430[2430] +- Add experimental `cgroup` metrics to the system/process MetricSet. {pull}2184[2184] +- Add experimental `haproxy` module. {pull}2384[2384] + + +*Winlogbeat* + +- Add caching of event metadata handles and the system render context for the wineventlog API {pull}888[888] +- Improve config validation by checking for unknown top-level YAML keys. {pull}1100[1100] +- Add the ability to set `tags`, `fields`, and `fields_under_root` as options for each event log {pull}1092[1092] +- Add additional data to the events published by Winlogbeat. The new fields are `activity_id`, +`event_data`, `keywords`, `opcode`, `process_id`, `provider_guid`, `related_activity_id`, +`task`, `thread_id`, `user_data`, and `version`. {issue}1053[1053] +- Add `event_id`, `level`, and `provider` configuration options for filtering events {pull}1218[1218] +- Add `include_xml` configuration option for including the raw XML with the event {pull}1218[1218] + + +==== Deprecated + +*Affecting all Beats* + +- The support for doing GeoIP lookups is deprecated and will be removed in version 6.0. {pull}1601[1601] +- Topology map is deprecated. This applies to the settings: `refresh_topology_freq`, `topology_expire`, `save_topology`, + `host_topology`, `password_topology`, `db_topology`. +- Setting `port` has been deprecated in Redis and Logstash outputs. {pull}2620[2620] + +*Filebeat* + +- Deprecate `close_older` option and replace it with `close_inactive`. {issue}2051[2051] +- Deprecate `force_close_files` option and replace it with `close_removed` and `close_renamed`. {issue}1600[1600] + diff --git a/libbeat/docs/release.asciidoc b/libbeat/docs/release.asciidoc index 161e21779e6..e22045ed3e2 100644 --- a/libbeat/docs/release.asciidoc +++ b/libbeat/docs/release.asciidoc @@ -6,6 +6,7 @@ -- This section summarizes the changes in each release. +* <> * <> * <> * <> @@ -26,7 +27,6 @@ This section summarizes the changes in each release. * <> * <> * <> - -- include::../../CHANGELOG.asciidoc[]