Release on: 2021-01-26
- Fix "fatal error: concurrent map read and map write" due to reads of a concurrently mutated map in inventories.payload.MarshalJSON()
- Fix an issue on arm64 where non-gauge metrics from Python checks were treated as gauges.
- On Windows, fixes uninstall/upgrade problem if core agent is not running but other services are.
- Fix NPM UDP destination address decoding when source address ends with .8 during offset guessing.
- On Windows, changes the password generating algorithm to have a minimum length of 16 and a maximum length of 20 (from 12-18). Improves compatibility with environments that have longer password requirements.
Release on: 2021-01-14
- Please refer to the 7.25.0 tag on integrations-core for the list of changes on the Core Checks
- Add com.datadoghq.ad.tags container auto-discovery label in AWS Fargate environment.
- Package the gstatus command line tool binary for GlusterFS integration metric collection.
- Queried domain can be tracked as part of DNS stats
- APM: The agent is now able to skip top-level span computation in cases when the client has marked them by means of the Datadog-Client-Computed-Top-Level header.
- APM: The maximum allowed key length for tags has been increased from 100 to 200.
- APM: Improve Oracle SQL obfuscation support.
- APM: Added support for Windows pipes. To enable it, set the pipe path using DD_APM_WINDOWS_PIPE_NAME. For more details check PR #6615
- Pause containers are now detected and auto excluded based on the io.kubernetes container labels.
- APM: new datadog_agent.obfuscate_sql_exec_plan function exposed to python checks to enable obfuscation of json-encoded SQL Query Execution Plans.
- APM: new obfuscate_sql_values option in apm_config.obfuscation enabling optional obfuscation of SQL queries contained in JSON data collected from some APM services (ES & Mongo)
- Support the ddog-gov.com site option in the Windows GUI installer.
- Adds config setting for ECS metadata endpoint client timeout (ecs_metadata_timeout), value in milliseconds.
- Add loader config to allow selecting specific loader at runtime. This config is available at init_config and instances level.
- Added additional container information to the status page when collect all container logs is enabled in agent status.
- On Windows, it will no longer be required to supply the ddagentuser name on upgrade. Previously, if a non-default or domain user was used, the same user had to be provided on subsequent upgrades.
- Added --flare flag to agent check to save check results to the agent logs directory. This enables flare to pick up check results.
- Added new config option for JMXFetch collect_default_jvm_metrics that enables/disables default JVM metric collection.
- Allow empty message for DogStatsD events (e.g. "_e{10,0}:test title|")
- Expires the cache key for availability of ECS metadata endpoint used to fetch EC2 resource tags every 5 minutes.
- Data coming from kubernetes pods now have new kube_ownerref_kind and kube_ownerref_name tags for each of the pod's OwnerRef property, indicating its Kind and Name, respectively.
- We improved the way Agents get the Kubernetes cluster ID from the Cluster Agent. It used to be that the cluster agent would create a configmap which had to be mounted as an env variable in the agent daemonset, blocking the process-agent from starting if not found. Now the process-agent will start, only the Kubernetes Resources collection will be blocked.
- Events sent by the runtime security agent to the backend use a new taxonomy.
- Scrub container args as well for orchestrator explorer.
- Support custom autodiscovery identifiers on Kubernetes using the ad.datadoghq.com/<container_name>.check.id pod annotation.
- The CPU check now collects system-wide context switches on Linux.
- Add
--tableoption toagent checkcommand to output results in condensed tabular format instead of JSON. - APM: improve performance by changing the msgpack serialization implementation.
- APM: improve the performance of the msgpack deserialization for the v0.5 payload format.
- APM: improve performance of trace processing by removing some heap allocations.
- APM: improve sublayer computation performance by reducing the number of heap allocations.
- APM: improved stats computation performance by removing some string concatenations.
- APM: improved trace signature computation by avoiding heap allocations.
- APM: improve stats computation performance.
- Update from alpine:3.10 to alpine:3.12 the base image in Dogstatsd's Dockerfiles.
- APM: remove the already deprecated apm_config.extra_aggregators config option.
- Fix macos dlopen failures by ensuring cmake preserves the required runtime search path.
- Fix memory leak on check unscheduling, which could be noticeable for checks submitting large amounts of metrics/tags.
- Exclude pause containers using the cdk/pause.* image.
- Fixed missing some Agent environment variables in the flare
- Fix a bug that prevented the logs Agent from discovering the correct init containers source and service on Kubernetes.
- The logs agent now uses the container image name as logs source instead of kubernetes when a standard service value was defined for the container.
- Fixes panic on concurrent map access in Kubernetes metadata tag collector.
- Fixed a bug that could potentially cause missing container tags for check metrics.
- Fix a potential panic on ECS when the ECS API is returning empty docker ID
- Fix systemd check id to handle multiple instances. The fix will make check id unique for each different instances.
- Fix missing tags on pods that were not seen with a running container yet.
- Fix snmp listener subnet loop to use correct subnet pointer when creating snmpJob object.
- Upgrade the embedded pip version to 20.3.3 to get a newer vendored version of urllib3.
- The Agent, Logs Agent and the system-probe are now compiled with Go
1.14.12 - Upgrade embedded
libkrb5Kerberos library to v1.18.3. This version drops support for the encryption types marked as "weak" in the docs of the library
Release on: 2020-12-17
- Fix a bug when parsing the current version of an integration that prevented upgrading from an alpha or beta prerelease version.
- During a domain installation in a child domain, the Windows installer can now use a user from a parent domain.
- The Datadog Agent had a memory leak where some tags would be collected but never cleaned up after their entities were removed from a Kubernetes cluster due to their IDs not being recognized. This has now been fixed, and all tags are garbage collected when their entities are removed.
- Updated the shipped CA certs to latest (2020-12-08)
Release on: 2020-12-03
- Please refer to the 7.24.0 tag on integrations-core for the list of changes on the Core Checks
- tcp_queue_length check: the previous metrics reported by this check (
tcp_queue.rqueue.size,tcp_queue.rqueue.min,tcp_queue.rqueue.max,tcp_queue.wqueue.size,tcp_queue.wqueue.min,tcp_queue.wqueue.max) were generating too much data because there was one time series generated per TCP connection. Those metrics have been replaced bytcp_queue.read_buffer_max_usage_pct,tcp_queue.write_buffer_max_usage_pctwhich are aggregating all the connections of a container. These metrics are reporting the maximum usage in percent (amount of data divided by the queue capacity) of the busiest buffer. Additionally, only_count_nb_context option from the tcp_queue_length check configuration has been removed and will be ignored from now on.
- Added new configuration flag, system_probe_config.enable_conntrack_all_namespaces, false by default. When set to true, this will allow system probe to monitor conntrack entries (for NAT info) in all namespaces that are peers of the root namespace.
- Added JMX version and java runtime version to agent status page
kubernetes_pod_annotations_as_tags(DD_KUBERNETES_POD_ANNOTATIONS_AS_TAGS) now support regex wildcards:'{"*":"<PREFIX>_%%annotation%%"}'can be used as value to collect all pod annotations as tags.kubernetes_node_labels_as_tags(DD_KUBERNETES_NODE_LABELS_AS_TAGS) now support regex wildcards:'{"*":"<PREFIX>_%%label%%"}'can be used as value to collect all node labels as tags. Note:kubernetes_pod_labels_as_tags(DD_KUBERNETES_POD_LABELS_AS_TAGS) supports this already.- Listening for conntrack updates from all network namespaces (system_probe_config.enable_conntrack_all_namespaces flag) is now turned on by default
- Expand pause container image filter
- Adds misconfig check for hidepid=2 option on proc mount.
- It's possible to ignore
auto_conf.yamlconfiguration files usingignore_autoconforDD_IGNORE_AUTOCONF. Example: DD_IGNORE_AUTOCONF="redisdb kubernetes_state" - APM: The trace-agent now automatically sets the GOMAXPROCS value in Linux containers to match allocated CPU quota, as opposed to the matching the entire node's quota.
- APM: Lowered CPU usage when using analytics.
- APM: Move UTF-8 validation from the span normalizer to the trace decoder, which reduces the number of times each distinct string will be validated to once, which is beneficial when the v0.5 trace format is used.
- Add the config forwarder_retry_queue_payloads_max_size which defines the maximum size in bytes of all the payloads in the forwarder's retry queue.
- When enabled, JMXFetch now logs to its own log file. Defaults to
jmxfetch.login the default agent log directory, and can be configured withjmx_log_file. - Added UDS support for JMXFetch JMXFetch upgraded to 0.40.3
- dogstatsd_mapper_profiles may now be defined as an environment variable DD_DOGSTATSD_MAPPER_PROFILES formatted as JSON
- Add orchestrator explorer related section into DCA Status
- Added byte count per log source and display it on the status page.
- APM: refactored the SQL obfuscator to be significantly more efficient.
- IO check: device_blacklist_re has been deprecated in favor of device_exclude_re.
- The config options tracemalloc_whitelist and tracemalloc_blacklist have been deprecated in favor of tracemalloc_include and tracemalloc_exclude.
APM: Fix a bug where non-float64 numeric values in apm_config.analyzed_spans would disable this functionality.
Disable stack protector on system-probe to make it buildable on the environments which stack protector is enabled by default.
Some linux distributions like Alpine Linux enable stack protector by default which is not available on eBPF.
Fix a panic in containerd if retrieved ociSpec is nil
Fix random panic in Kubelet searchPodForContainerID due to concurrent modification of pod.Status.AllContainers
Add retries to Kubernetes host tags retrievals, minimize the chance of missing/changing host tags every 30mins
Fix rtloader build on strict posix environment, e.g. musl libc on Alpine Linux.
Allows system_probe to be enabled without enabling network performance monitoring.
Set
network_config.enabled=falsein yoursystem-probe.yamlwhen running the system-probe without networks enabled.Fixes truncated output for status of compliance checks in Security Agent.
Under some circumstances, the Agent would delete all tags for a workload if they were collected from different sources, such as the kubelet and docker, but deleted from only one of them. Now, the agent keeps track of tags per collector correctly.
- The utilities provided by the sysstat package have been removed: the
iostat,mpstat,pidstat,sar,sadf,cifsiostatandnfsiostat-sysstatbinaries have been removed from the packaged Agent. This has no effect on the behavior of the Agent and official integrations, but your custom checks may be affected if they rely on these embedded binaries. - Activate security-agent service by default in the Linux packages of the Agent (RPM/DEB). The security-agent won't be started if the file /etc/datadog-agent/security-agent.yaml does not exist.
Release on: 2020-10-21
- The
ec2_prefer_imdsv2parameter was ignored when fetching EC2 tags from the metadata endpoint. This fixes a misleading warning log that was logged even whenec2_prefer_imdsv2was left disabled in the Agent configuration. - Support of secrets in JSON environment variables, added in 7.23.0, is reverted due to a side effect (e.g. a string value of "-" would be loaded as a list). This feature will be fixed and added again in a future release.
- The Windows installer can now install on domains where the domain name is different from the Netbios name.
Release on: 2020-10-06
- Please refer to the 7.23.0 tag on integrations-core for the list of changes on the Core Checks
- Network monitoring: enable DNS stats collection by default.
- APM: Decoding errors reported by the datadog.trace-agent.receiver.error and datadog.trace_agent.normalizer.traces_dropped contain more detailed reason tags in case of EOFs and timeouts.
- Running the agent flare with the -p flag now includes profiles for the trace-agent.
- APM: An SQL query obfuscation cache was added under the feature flag DD_APM_FEATURES=sql_cache. In most cases where SQL queries are repeated or prepared, this can significantly reduce CPU work.
- Secrets handles are not supported inside JSON value set through environment variables. For example setting a secret in a list DD_FLARE_STRIPPED_KEYS='["ENC[auth_token_name]"]' datadog-agent run
- Add basic support for UTF16 (BE and LE) encoding.
It should be manually enabled in a log configuration using
encoding: utf-16-beorencoding: utf-16-leother values are unsupported and ignored by the agent.
- Add new configuration parameter to allow 'GroupExec' permission on the secret-backend command. Set to 'true' the new parameter 'secret_backend_command_allow_group_exec_perm' to activate it.
- Add a map from DNS rcode to count of replies received with that rcode
- Enforces a size limit of 64MB to uncompressed sketch payloads (distribution metrics). Payloads above this size will be split into smaller payloads before being sent.
- APM: Span normalization speed has been increased by 15%.
- Improve the
kubeletcheck error reporting in the output ofagent statusin the case where the agent cannot properly connect to the kubelet. - Add space_id, space_name, org_id and org_name as tags to both autodiscovered containers as well as checks found through autodiscovery on Cloud Foundry/Tanzu.
- Improves compliance check status view in the security-agent status command.
- Include compliance benchmarks from github.com/DataDog/security-agent-policies in the Agent packages and the Cluster Agent image.
- Windows Docker image is now based on Windows Server Nano instead of Windows Server Core.
- Allow sending the GCP project ID under the
project_id:host tag key, in addition to theproject:host tag key, with thegce_send_project_id_tagconfig setting. - Add kubeconfig to GCE excluded host tags (used on GKE)
- The cluster name can now be longer than 40 characters, however the combined length of the host name and cluster name must not exceed 254 characters.
- When requesting EC2 metadata, you can use IMDSv2 by turning
on a new configuration option (
ec2_prefer_imdsv2). - When tailing logs from container in a kubernetes environment long lines (>16kB usually) that got split by the container runtime (docker & containerd at least) are now reassembled pending they do not exceed the upper message length limit (256kB).
- Move the cluster-id ConfigMap creation, and Orchestrator Explorer controller instantiation behind the orchestrator_explorer config flag to avoid it failing and generating error logs.
- Add caching for sending kubernetes resources for live containers
- Agent log format improvement: logs can have kv-pairs as context to make it easier to get all logs for a given context Sample: 2020-09-17 12:17:17 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | check:io | Done running check
- The CRI check now supports container exclusion based on container name, image and kubernetes namespace.
- Added a network_config config to the system-probe that allows the network module to be selectively enabled/disabled. Also added a corresponding DD_SYSTEM_PROBE_NETWORK_ENABLED env var. The network module will only be disabled if the network_config exists and has enabled set to false, or if the env var is set to false. To maintain compatibility with previous configs, the network module will be enabled in all other cases.
- Log a warning when a log file is rotated but has not finished tailing the file.
- The NTP check now uses the cloud provider's recommended NTP servers by default, if the Agent detects that it's running on said cloud provider.
- process_config.orchestrator_additional_endpoints and process_config.orchestrator_dd_url are deprecated in favor of: orchestrator_explorer.orchestrator_additional_endpoints and orchestrator_explorer.orchestrator_dd_url.
- Allow agent integration install to work even if the datadog agent configuration file doesn't exist. This is typically the case when this command is run from a Dockerfile in order to build a custom image from the datadog official one.
- Implement variable interpolation in the tagger when inferring the standard tags
from the
DD_ENV,DD_SERVICEandDD_VERSIONenvironment variables - Fix a bug that was causing not picking checks and logs for containers targeted by container-image-based autodiscovery. Or picking checks and logs for containers that were not targeted by container-image-based autodiscovery. This happened when several image names were pointing to the same image digest.
- APM: Allow digits in SQL literal identifiers (e.g. 1sad123jk)
- Fixes an issue with not always reporting ECS Fargate task_arn tag due to a race condition in the tag collector.
- The SUSE SysVInit service now correctly starts the Agent as the dd-agent user instead of root.
- APM: Allow double-colon operator in SQL obfuscator.
- UDP packets can be sent in two ways. In the "connected" way, a connect call is made first to assign the remote/destination address, and then packets get sent with the send function or sendto function with destination address set to NULL. In the "unconnected" way, packets get sent using sendto function with a non NULL destination address. This fix addresss a bug where network stats were not being generated for UDP packets sent using the "unconnected" way.
- Fix the Windows systray not appearing sometimes (bug introduced with 6.20.0).
- The Chocolatey package now uses a fixed URL to the MSI installer.
- Fix logs tagging inconsistency for restarted containers.
- On macOS, in Agent v6, the unversioned python binaries in
/opt/datadog-agent/embedded/bin(example:python,pip) now correctly point to the Python 2 binaries. - Fix truncated cgroup name on copy with bpf_probe_read_str in OOM kill and TCP queue length checks.
- Use double-precision floats for metric values submitted from Python checks.
- On Windows, the ddtray executable now has a digital signature
- Updates the logs package to get the short image name from Kubernetes ContainerSpec, rather than ContainerStatus. This works around a known issue where the image name in the ContainerStatus may be incorrect.
- On Windows, the Agent now responds to control signals from the OS and shuts down gracefully. Coincidentally, a Windows Agent Container will now gracefully stop when receiving the stop command.
- All Agents binaries are now compiled with Go
1.14.7 - JMXFetch upgraded from 0.38.2 to 0.39.1
- Move the orchestrator related settings process_config.orchestrator_additional_endpoints and process_config.orchestrator_dd_url into the orchestrator_explorer section.
Release on: 2020-09-17
- Please refer to the 7.22.1 tag on integrations-core for the list of changes on the Core Checks
- Define a default logs file (security-agent.log) for the security-agent.
- Fix segfault when listing Garden containers that are in error state.
- Do not activate security-agent service by default in the Linux packages of the Agent (RPM/DEB). The security-agent was already properly starting and exiting if not activated in configuration.
Release on: 2020-08-25
- Please refer to the 7.22.0 tag on integrations-core for the list of changes on the Core Checks
- Implements agent-side compliance rule evaluation in security agent using expressions.
- Add IO operations monitoring for Docker check (docker.io.read/write_operations)
- Track TCP connection churn on system-probe
- The new Runtime Security Agent collects file integrity monitoring events. It is disabled by default and only available for Linux for now.
- Make security-agent part of automatically started agents in RPM/DEB/etc. packages (will do nothing and exit 0 by default)
- Add support for receiving and processing SNMP traps, and forwarding them as logs to Datadog.
- APM: A new trace ingestion endpoint was introduced at /v0.5/traces which supports a more compact payload format, greatly improving resource usage. The spec for the new wire format can be viewed here. Tracers supporting this change will automatically use the new endpoint.
Adds a gauge for system.mem.slab_reclaimable. This is part of slab memory that might be reclaimed (i.e. caches). Datadog 7.x adds SReclaimable memory, if available on the system, to the system.mem.cached gauge by default. This may lead to inconsistent metrics for clients migrating from Datadog 5.x, where system.mem.cached didn't include SReclaimable memory. Adding a gauge for system.mem.slab_reclaimable allows inverse calculation to remove this value from the system.mem.cached gauge.
Expand GCR pause container image filter
Kubernetes events for pods, replicasets and deployments now have tags that match the metrics metadata. Namely, pod_name, kube_deployment, kube_replicas_set.
Enabled the collection of the kubernetes resource requirements (requests and limits) by bumping the agent-payload dep. and collecting the resource requirements.
Implements resource fallbacks for complex compliance check assertions.
Add system.cpu.num_cores metric with the number of CPU cores (windows/linux)
compliance: Add support for Go custom compliance checks and implement two for CIS Kubernetes
Make DSD Mapper also map metrics that already contain tags.
If the retrieval of the AWS EC2 instance ID or hostname fails, previously-retrieved values are now sent, which should mitigate host aliases flapping issues in-app.
Increase default timeout on AWS EC2 metadata endpoints, and make it configurable with
ec2_metadata_timeoutAdd container incl./excl. lists support for ECS Fargate (process-agent)
Adds support for a heap profile and cpu profile (of configurable length) to be created and included in the flare.
Upgrade embedded Python 3 to 3.8.5. Link to Python 3.8 changelog: https://docs.python.org/3/whatsnew/3.8.html
Note that the Python 2 version shipped in Agent v6 continues to be version 2.7.18 (unchanged).
Upgrade pip to v20.1.1. Link to pip 20.1.1 changelog: https://pip.pypa.io/en/stable/news/#id54
Upgrade pip-tools to v5.3.1. Link to pip-tools 5.3.1 changelog: https://github.com/jazzband/pip-tools/blob/master/CHANGELOG.md
Introduces support for resolving pathFrom from in File and Audit checks.
On Windows, always add the user to the required groups during installation.
APM: A series of changes to internal algorithms were made which reduced CPU usage between 20-40% based on throughput.
- Allow integration commands to work for pre-release versions.
- [Windows] Ensure
PYTHONPATHvariable is ignored correctly when initializing the Python runtime. - Enable listening for conntrack info from all namespaces in system probe
- Fix cases where the resolution of secrets in integration configs would not be performed for autodiscovered containers.
- Fixes submission of containers blkio metrics that may modify array after being already used by aggregator. Can cause missing tags on containerd.* metrics
- Restore support of JSON-formatted lists for configuration options passed as environment variables.
- Don't allow pressing the disable button on checks twice.
- Fix container_include_metrics support for all container checks
- Fix a bug where the Agent disables collecting tags when the cluster checks advanced dispatching is enabled in the Daemonset Agent.
- Fixes a bug where the ECS metadata endpoint V2 would get queried even though it was not configured with the configuration option cloud_provider_metadata.
- Fix a bug when a kubernetes job has exited after some time the tagger does not update it even if it did change its state.
- Fixes the Agent failing to start on sysvinit on systems with dpkg >= 1.19.3
- The agent was collecting docker container logs (metrics) even if they are matching DD_CONTAINER_EXCLUDE_LOGS (resp. DD_CONTAINER_EXCLUDE_METRICS) if they were started before the agent. This is now fixed.
- Fix a bug where the Agent would not remove tags for pods that no longer exist, potentially causing unbounded memory growth.
- Fix pidfile support on security-agent
- Fixed system-probe not working on CentOS/RHEL 8 due to our custom SELinux policy. We now install the custom policy only on CentOS/RHEL 7, where the system-probe is known not to work with the default. On other platform the default will be used.
- Stop sending payload for Cloud Foundry applications containers that have no container_name tag attached to avoid them showing up in the UI with empty name.
- APM: datadog.trace_agent.receiver.* metrics are now also tagged by endpoint_version
Release on: 2020-07-22
- JMXFetch upgraded to 0.38.2 to fix Java 7 support.
- Fix init of security-agent - exit properly when no feature requiring it is activated and avoid conflicting with core agent port bindings.
Release on: 2020-07-16
- Please refer to the 7.21.0 tag on integrations-core for the list of changes on the Core Checks
- APM: The maximum allowed payload size by the agent was increased from 10MB to 50MB. This could result in traffic increases for users which were affected by this issue.
- APM: The maximum connection limit over a 30s period was removed. This can result in an increase of tracing data for users that were affected by this limitation.
- Add support of new DatadogMetric CRD in DCA. Allows to autoscale based on any valid Datadog query.
- Add packages scripts for dogstatsd that have the same features as the agent: create symlink for binary, create dd-agent user and group, setup the service and cleanup those when uninstalling.
- Adds OOM Kill probe to ebpf package and corresponding corecheck to the agent.
- The Datadog IoT Agent is now available for 32 bit ARM architecture (armv7l/armhf).
- Add Compliance agent in Cluster Agent to monitor Kubernetes objects
- Add docker.cpu.limit and containerd.cpu.limit metrics, reporting maximum cpu time (hz or ns) available for each container based on their limits. (Only supported on Linux)
- Addition of a gRPC server and a hostname resolution endpoint, including a grpc-gateway that exposes said endpoint as a REST service.
- Adding a 'log_format_rfc3339' option to use the RFC3339 format for the log time.
- Compliance Agent implementing scheduling of compliance checks for Docker and Kubernetes benchmarks.
- Expose agent's sql obfuscation to python checks via new datadog_agent.obfuscate_sql method
- Support installing non-core integrations with the
integrationcommand, such as those located in theintegrations-extrasrepository.
- The Agent
statuscommand now includes the flavor of the Agent that is running. - The Agent GUI now includes the flavor of the Agent that is running.
- Adds Tagger information to Datadog Agent flare for support investigations.
- Add a static collector in the tagger package for tags that do not change after pod start (such as those from an environment variable).
- Add
autodiscovery_subnetto available SNMP template extra configs - When enabling collect_ec2_tags or collect_gce_tags option, EC2/GCE tags are now cached to avoid missing tags when user exceed his AWS/GCE quotas.
- Chocolatey package can be installed on Domain Controller
- The Agent now collects the Availability Zone a Fargate Task (using platform version 1.4 or later) is running in as an "availability_zone" tag.
- Enabled the collection of the init-containers by bumping the agent-payload dep. and collecting the init-containers.
- The Agent now collects recommended "app.kubernetes.io" Kubernetes labels as tags by default, and exposes them under a "kube_app" prefix.
- Docker and Containerd checks now support filtering containers by kube_namespace.
- Add support for sampling to distribution metrics
- Flare now includes the permission information for parents of config and log file directories.
- Collect processes namespaced PID.
- You can now enable or disable the dogstatsd-stats troubleshooting feature at
runtime using the
config set dogstatsd_statscommand of the Agent. - API Keys are now sanitized for logs_config and additional_endpoints.
- Upgrade gosnmp to support more authentication and privacy protocols for v3 connections.
- Use the standard tag 'service' as a log collection attribute for container's logs collected from both kubernetes and docker log sources.
- agent check returns non zero exit code when trace malloc is enabled (tracemalloc_debug: true) when using python 2
- Added the checksum type to the checksum key itself, as it is deprecated to have a separate checksum_type key.
- Add
lowercase_device_tagoption to the systemiocore check on Windows. When enabled, sends metrics with a lowercaseddevicetag, which is consistent with thesystem.io.*metrics of Agent v5 and thesystem.disk.*metrics of all Agent versions.
Fix missing values from cluster-agent status command.
Add missing
device_nametag in iostats_pdhFixes an issue where DD_TAGS were not applied to EKS Fargate pods and containers.
Add
freetdslinux dep needed for SQL Server to run in Docker Agent.APM : Fix parsing of non-ASCII numerals in the SQL obfuscator. Previously unicode characters for which unicode.IsDigit returns true could cause a hang in the SQL obfuscator
APM: correctly obfuscate AUTH command.
Dogstatsd standalone: when running on a systemd-based system, do not stop Dogstatsd when journald is stopped or restarted.
Fix missing logs and metrics for docker-labels based autodiscovery configs after container restart.
Fix bugs introduced in 7.20.0/6.20.0 in the Agent 5 configuration import command: the command would not import some Agent config settings, including
api_key, and would write some Docker & Kubernetes config settings to wrongly-located files.Fixes tag extraction from Kubernetes pod labels when using patterns on certain non-alphanumeric label names (e.g. app.kubernetes.io/managed-by).
Fixes the /ready health endpoint on the cluster-agent.
The /ready health endpoint was reporting 200 at the cluster-agent startup and was then, permanently reporting 500 even though the cluster-agent was experiencing no problem. In the body of the response, we could see that a healthcheck component was failing. This change fixes this issue.
This fix aims to cover the case when the agent is running inside GKE with workload identity enabled. If workload identity is enabled, access to /instance/name is forbidden, resulting into an empty host alias.
Fix hostname resolution issue preventing the Process and APM agents from picking up a valid hostname on some containerized environments
Fix a bug which causes certain configuration options to be ignored by the
process-agentin the presence of asystem-probe.yaml.Process agent and system probe now correctly accept multiple API keys per endpoint.
The
device_nametag is not used anymore to populate theDevicefield of a series. Only thedevicetag is considered.Fixes problem on Windows where ddagentuser home directory is left behind.
Revert upgrade of GoSNMP and addition of extra authentication protocols.
Add support for examining processes inside Docker containers running under systemd cgroups. This also reduces agent logging volume as it's able to capture those statistics going forward.
APM: The agent now exits with code 0 when the API key is not specified. This is so to prevent the Windows SCM from restarting the process.
- All Agents binaries are now compiled with Go
1.13.11. - In Debug mode, DogStatsD log a warning message when a burst of metrics is detected.
- JMXFetch upgraded to 0.38.0
- JQuery, used in the web-based agent GUI, has been upgraded to 3.5.1
Release on: 2020-06-17
- Please refer to the 7.20.2 tag on integrations-core for the list of changes on the Core Checks
Release on: 2020-06-11
- Please refer to the 7.20.1 tag on integrations-core for the list of changes on the Core Checks
Release on: 2020-06-11
- Please refer to the 7.20.0 tag on integrations-core for the list of changes on the Core Checks
- Pod and container tags autodiscovered via pod annotations now support multiple values for the same key.
- Install script creates
install_inforeport - Agent detects
install_inforeport and sends it through Host metadata - Adding logic to get standard
servicetag from Pod Metadata Labels. - APM: A new endpoint was added which helps augment and forward profiles to Datadog's intake.
- APM: Information about APM is now included in the agent's status output (both in the GUI and in the 'agent status' command).
- Introducing the 'cloud_provider_metadata' option in the Agent configuration to restrict which cloud provider metadata endpoints will be queried.
- Add collector for Garden containers running applications in CloudFoundry environment to view them in the live container list and container map.
- JMXFetch (helper for JMX checks) is now restarted if it crashes on Windows.
- Add scaffold for security/compliance agent CLI.
container_exclude_metricsandcontainer_include_metricscan be used to filter metrics collection for autodiscovered containers.container_exclude_logsandcontainer_include_logscan be used to filter logs collection for autodiscovered containers.- Support SNMP autodiscovery via a new configuration listener, with new template variables.
- Support Tencent Cloud provider.
- When installing the Agent using Chocolatey, information about the installation is saved for diagnostic and telemetry purposes.
- The Agent's flare now includes information about the method used to install the Agent.
- Ignore AKS pause containers hosted in the aksrepos.azurecr.io container registry.
- On Linux and MacOS, add a new
device_nametag on IOstats and disk checks. - Windows installer can use the command line key
HOSTNAME_FQDN_ENABLEDto set the config value ofhostname_fqdn. - Add missing
device_nametags on docker, containerd and network checks. Make series managedevice_nametag ifdeviceis missing. - Support custom tagging of docker container data via an autodiscovery "tags" label key.
- Improved performances in metric aggregation logic. Use 64 bits context keys instead of 128 bits in order to benefit from better performances using them as map keys (fast path methods) + better performances while computing the hash thanks to inlining.
- Count of DNS responses with error codes are tracked for each connection.
- Latency of successful and failed DNS queries are tracked for each connection.Queries that time out are also tracked separately.
- Enrich dogstatsd metrics with task_arn tag if DD_DOGSTATSD_TAG_CARDINALITY=orchestrator.
- More pause containers from
ecr,gcrandmcrare excluded automatically by the Agent. - Improve cluster name auto-detection on Azure AKS.
- APM: Improve connection reuse with HTTP keep-alive in trace agent.
- Increase default timeout to collect metadata from GCE endpoint.
- Use insertion sort in the aggregator context keys generator as it provides better performances than the selection sort. In cases where the insertion sort was already used, improved its threshold selecting between it and Go stdlib sort.
- Expose distinct endpoints for liveness and readiness probes.
- The liveness probe (
/live) fails in case of unrecoverable error that deserve an agent restart. (Ex.: goroutine deadlock or premature exit) - The readiness probe (
/ready) fails in case of recoverable errors or errors for which an agent restart would be more nasty than useful (Ex.: the forwarder fails to connect to DataDog)
- The liveness probe (
- Exclude automatically pause containers for OpenShift, EKS and AKS Windows
- Introduce
kube_cluster_nameandecs_cluster_nametags in addition tocluster_name. Add the possibility to stop sending thecluster_nametag using the parameterdisable_cluster_name_tag_keyin Agent config. The Agent keeps sendingkube_cluster_nameand ecs_cluster_name tags regardless of disable_cluster_name_tag_key. - Configure additional process and orchestrator endpoints by environment variable.
- The process agent can be canfigured to collect containers from multiple sources (e.g kubelet and docker simultaneously).
- Upgrading the embedded Python 2 to the latest, and final, 2.7.18 release.
- Improve performance of system-probe conntracker.
- Throttle netlink socket on workloads with high connection churn.
container_excludereplacesac_exclude.container_includereplacesac_include.ac_excludeandac_includewill keep being supported but the Agent ignores them in favor ofcontainer_excludeandcontainer_includeif they're defined.
- APM: Fix a small programming error causing the "superfluous response.WriteHeader call" warning.
- Fixes missing container stats in ECS Fargate 1.4.0.
- Ensure Python checks are always garbage-collected after they're unscheduled by AutoDiscovery.
- Fix for autodiscovered checks not being rescheduled after container restart.
- On Windows, fix calculation of the
system.swap.pct_freemetric. - Fix a bug in the file tailer on Windows where the log-agent would keep a lock on the file preventing users from renaming the it.
- Upgrade embedded ntplib to
0.3.4 - JMXFetch upgraded to 0.36.2
- Rebranded puppy agent as iot-agent.
Release on: 2020-05-12
- Please refer to the 7.19.2 tag on integrations-core for the list of changes on the Core Checks
Release on: 2020-05-04
This release is only an Agent v7 release, as Agent v6 is not affected by the undermentioned bug.
- Fix panic in the dogstatsd standalone package when running in a containerized environment.
Release on: 2020-04-30
- Please refer to the 7.19.0 tag on integrations-core for the list of changes on the Core Checks
Default logs-agent to use HTTPS with compression when possible. Starting from this version, the default transport is HTTPS with compression instead of TCP. The usage of TCP is kept in the following cases:
- logs_config.use_tcp is set to true
- logs_config.socks5_proxy_address is set, because socks5 proxies are not yet supported in HTTPS with compression
- HTTPS connectivity test has failed: at agent startup, an HTTPS test request is sent to determine if HTTPS can be used
To force the use of TCP or HTTPS with compression, logs_config.use_tcp or logs_config.use_http can be set to true, respectively.
- The Agent is now available on Chocolatey for Windows
- Add
--full-sketchesoption to agent check command that displays bins information - Support logs collection from Kubernetes log files with old Kubernetes versions (< v1.10).
- Expose the new JMXFetch rate with metrics method to test JMX based checks.
- The
ac_includeandac_excludeauto-discovery parameters now support thekube_namespace:prefix to collect or discard logs and metrics for whole namespaces in addition to thename:andimage:prefixes to filter on container name and image name. - EKS Fargate containers now appear in the live containers view. All processes running inside the EKS Fargate Pod appear in the live processes view when shareProcessNamespace is enabled in the Pod Spec.
- Add the ability to change log_level at runtime. The agent command
has been extended to support new operation. For example to set
the log_level to debug the following command should be used:
agent config set log_level debug, all runtime-configurable settings can be listed usingagent config list-runtime. The log_level may also be fetched using theagent config get log_levelcommand. Additional runtime-editable setting can easily be added by following this implementation. - The
system-probeclassifies UDP connections as incoming or outgoing.
- Adds a new config option to the systemd core check. It adds the ability to provide a custom mapping from a unit substate to the service check status.
- The systemd core check now collects and submits the systemd version as check metadata.
- Add
host_provider_idtag to Kubernetes events; for AWS instances this is unique in contrast to the Kubernetes nodename currently provided with thehosttag. - On Windows, now reports system.io.r_await and system.io.w_await. Metrics are reported from the performance monitor "Avg. Disk sec/Read" and "Avg. Disk sec/Write" metrics.
- Allow setting
is_jmxat the instance level, thereby enabling integrations to utilize JMXFetch and Python/Go. - The authentication token file is now only created
when the agent is launched with the
agent startcommand It prevents command such asagent statusto create an authentication token file owned by a wrong user. - Count of successful DNS responses are tracked for each connection.
- Network information is collected when the agent is running in docker (host mode only).
- Make sure we don't recognize
sha256:...as a valid image name and add fallback to .Config.Image in case it's impossible to mapsha256:...to a proper image name - Extract env, version and service tags from Docker containers
- Extract env, version and service tags from ECS Fargate containers
- Extract env, version and service tags from kubelet
- Log configurations of type
filenow accept a new parameter that allows to specify whether a log shall be tailed from the beginning or the end. It aims to allow whole log collection, including events that may occur before the agent first starts. The parameter is namedstart_positionand it can be set toendorbeginning, the default value isend. - Resolve Docker image name using config.Image in the case of multiple image RepoTags
- The agent configcheck command output now scrubs sensitive data and prevents API keys, password, token, etc. to appear in the console.
- Errors that arise while loading checks configuration files are now send with metadata along with checks loading errors and running errors so they will show up on the infrastructure list in the DataDog app.
- Remove cgroup deps from Docker utils, allows to implement several backends for Docker utils (i.e. Windows)
- On Windows, for Python3, add additional C-runtime DLLs to fix missing dependencies (notably for jpype).
- Fix 'check' command segfault when running for more than 1 hour (which could happen when using the '-b' option to set breakpoint).
- Fix panic due to concurrent map access in Docker AD provider
- Fix the
flarecommand not being able to be created for the non-core agents (trace, network, ...) when running in a separated container, such as in Helm. A new option,--local, has been added to theflarecommand to force the creation of the archive using the local filesystem and not the one where the core agent process is in. - Fix logs status page section showing port '0' being used when using the default HTTPS URL. The status page now show 443.
- Fix S6 behavior when the core agent dies. When the core agent died in a multi-process agent container managed by S6, the container stayed in an unhealthy half dead state. S6 configuration has been modified so that it will now exit in case of core agent death so that the whole container will exit and will be restarted.
- On Windows, fixes Process agent memory leak when obtaining process arguments.
- When a DNS name with ".local" is specifed in the parameter DDAGENTUSER_NAME, the correctly finds the corresponding domain.
- Fix an issue where
conf.yaml.examplecan be missing fromAdd a checkmenu in the Web user interface. - process-agent and system-probe now clean up their PID files when exiting.
- When the HTTPS transport is used to send logs, send the sourcecategory as the
sourcecategory:tag instead ofddsourcecategory:, for consistency with other transports.
- All Agents binaries are now compiled with Go
1.13.8 - JMXFetch upgraded to 0.36.1. See 0.36.1 and 0.36.0
- The
statsd_metric_namespaceoption now ignores the following metric prefixes:airflow,confluent,hazelcast,hive,ignite,jboss,sidekiq
- Fix conntrack issue where a large batch of deletion events was killing the goroutine polling the netlink socket.
- On Debian and Ubuntu-based systems, remove system-probe SELinux policy to prevent install failures.
Release on: 2020-03-13
- Please refer to the 7.18.0 tag on integrations-core for the list of changes on the Core Checks
- APM: Traces containing spans without an operation name will automatically be assigned the "unnamed_span" name (previously "unnamed-span").
- On MacOS, the Aerospike integration is no longer available since version 3.10 of the aerospike-client-python library is not yet available on this platform.
- On MacOS, the IBM WAS integration is no longer available since it relies on the lxml package which currently doesn't pass Apple's notarization tests.
- On Windows, the embedded Python will no longer use the PYTHONPATH environment variable, restricting its access to the Python packages installed with the Agent. Set windows_use_pythonpath to true to keep the previous behavior.
- Adding new "env" top level config option. This will be added to the host tags and propagated to the trace agent.
- Add SystemD integration support for SUSE.
- APM: Add support for calculating trace sublayer metrics for measured spans.
- APM: Add support for computing trace statistics on user-selected spans.
- APM: add support for version as another tag in trace metrics.
- Add docker.uptime, cri.uptime, and containerd.uptime metrics for all containers
- Add a warning in the logs-agent section of the agent status to incite users to switch over HTTP transport.
- Send a tag for any
servicedefined in theinit_configorinstancessection of integration configuration, with the latter taking precedence. This applies to metrics, events, and service checks.
The min_collection_interval check setting has been relocated since Agent 6/7 release. The agent import command now include in the right section this setting when importing configuration from Agent 5.
Add new config parameter (dogstatsd_entity_id_precedence) to enable DD_ENTITY_ID presence check when enriching Dogstatsd metrics with tags.
Add an option to exclude log files by name when wildcarding directories. The option is named
exclude_paths, it can be added for each custom log collection configuration file. The option accepts a list of glob.The status output now shows checks' last execution date and the last successful one.
On debian- and rhel-based systems, system-probe is now set up so that it can run in SELinux-enabled environments.
On Windows, a step to set the
siteparameter has been added to the graphical installer.Added support for inspecting DNS traffic received over TCP to gather DNS information.
Review the retry strategy used by the agent to connect to external services like docker, kubernetes API server, kubelet, etc. In case of failure to connect to them, the agent used to retry every 30 seconds 10 times and then, to give up. The agent will now retry after 1 second. It will then double the period between two consecutive retries each time, up to 5 minutes. So, After 10 minutes, the agent will keep on retrying every 5 minutes instead of completely giving up after 5 minutes. This change will avoid to have to restart the agent if it started in an environment that remains degraded for a while (docker being down for several minutes for example.)
Adds message field to the ComponentStatus check of kube_apiserver_controlplane.up service check.
Add a config option
ec2_use_windows_prefix_detectionto use the EC2 instance id for Windows hosts on EC2.Templates used for the agent status command are now embedded in the binary at compilation time and thus original template files are not required anymore at runtime.
Upgrade
pip-toolsandwheeldependencies for Python 2.Improve interpolation performance during conversion from Prometheus and OpenMetrics histograms to ddsketch
Allow sources for the Logs Agent to fallback to the
servicedefined in theinit_configsection of integration configuration to match the global tag that will be emitted.Stop doing HTML escaping on agent status command output in order to properly display all non-alphanumeric characters.
Upgrade embedded Python 3 to 3.8.1. Link to Python 3.8 changelog: https://docs.python.org/3/whatsnew/3.8.html
Note that the Python 2 version shipped in Agent v6 continues to be version 2.7.17 (unchanged).
Removing an RPM of the Datadog Agent will no longer throw missing files warning.
The agent config command output now scrubs sensitive data and prevents API keys, password, token, etc. from appearing in the console.
Add support for the EC2 instance metadata service (IMDS) v2 that requires to get a token before any metadata query. The agent will still issue unauthenticated request first (IMDS v1) before switching to token-based authentication (IMDS v2) if it fails.
system-probe no longer needs to be enabled/started separately through systemctl
- The submit_histogram_bucket API now accepts long integers as input values.
- ignore "origin" tags if the 'dd.internal.entity_id' tag is present in dogstatsd metrics.
- On Windows 64 bit, fix calculation of number of CPUS to handle machines with more than 64 CPUs.
- Make
systemdcore check handle gracefully missingSystemStateattribute. - Ignore missing docker label com.datadoghq.ad.check_names instead of showing error logs.
- The jmx and check jmx command will now properly use the loglevel provided with the deprecated --log-level flag or the DD_LOG_LEVEL environment var if any.
- Fix docker logs when the tailer receives a io.EOF error during a file rotation.
- Fix process-agent potentially dividing by zero when no containers are found.
- Fix process-agent not respecting logger configuration during startup.
- Errors mentioning the authentication token are now more specific and won't be obfuscated anymore.
- Upgrade embedded openssl to
1.1.1d, pyopenssl19.0.0and postgresql client lib to9.4.25. - Add the Go version used to build Dogstatsd in its version command.
- Update s6-overlay to v1.22.1.0 in docker images
- JMXFetch upgraded to 0.35.0
- Following the upgrade to Python 3.8, the Datadog Agent version
>= 6.18.0running Python 3 and version>= 7.18.0are now enforcing UTF8 encoding while running checks (and while running pdb debugger with -b option on the check cli command). Previous versions of the Agent were already using this encoding by default (depending on the environment variables) without enforcing it.
Release on: 2020-02-26
This is a Windows-only release.
- On Windows, fixes the Agent 7 installation causing the machine to reboot if the C runtime was upgraded when in use.
Release on: 2020-02-20
- Please refer to the 7.17.1 tag on integrations-core for the list of changes on the Core Checks
- Fix a panic in the log agent when the auto-discovery reports new containers to monitor and the agent fails to connect to the docker daemon. The main setup where this happened is on ECS Fargate where the ECS auto-discovery is watching for new containers and the docker socket is not available from the datadog agent.
- Fix DNS resolution for NPM when the system-probe is running in a container on a non-host network.
Release on: 2020-02-04
- Please refer to the 7.17.0 tag on integrations-core for the list of changes on the Core Checks
Change agents base images to Debian bullseye
Starting with this version, the containerized Agent never chooses the OS hostname as its hostname when it is running in a dedicated UTS namespace. This is done in order to avoid picking container IDs or kubernetes POD names as hostnames, since these identifiers do not reflect the identity of the host they run on.
This change only affects you if your agent is currently using a container ID or a kubernetes POD name as hostname. The hostname of the agent can be checked with
agent hostname. If the output stays stable when the container or POD of the agent is destroyed and recreated, you’re not impacted by this change. If the output changes, it means that the agent was unable to talk to EC2/GKE metadata server, it was unable to get the k8s node name from the kubelet, it was unable to get the hostname from the docker daemon and it is running in its dedicated UTS namespace. Under those conditions, you should set explicitly define the host name to be used by the agent in its configuration file.
- Add logic to support querying the kubelet through the APIServer to monitor AWS Fargate on Amazon EKS.
- Add mapping feature to dogstatsd to convert parts of dogstatsd/statsd metric names to tags using mapping rules in dogstatsd using wildcard and regex patterns.
- Resource tag collection on ECS.
- Add container_mode in journald input to set source/service as Docker image short name when we receive container logs
- Add kube_node_role tag in host metadata for the node role based on the
kubernetes.io/rolelabel. - Add cluster_name tag in host metadata tags. Cluster name used is read from config if set by user or autodiscovered from cloud provider or Kubernetes node label.
- The Agent check command displays the distribution metrics. The Agent status command displays histogram bucket samples.
- The system-probe will augment network connection information with DNS names gathered by inspecting local DNS traffic.
- Users can now use references to capture groups in mask sequence replacement_placeholder strings
- Do not apply the metric namespace configured under
statsd_metric_namespaceto dogstatsd metrics prefixed withdatadog.tracer. Tracer metrics are published with this prefix.
- APM: The trace-agent now correctly applies
log_to_console,log_to_syslogand all other syslog settings. - Make the log agent continuously retry to connect to docker rather than giving up when docker is not running when the agent is started. This is to handle the case where the agent is started while the docker daemon is stopped and the docker daemon is started later while the datadog agent is already running.
- Fixes #4650 [v7] Syntax in /readsecret.py for Py3
- Fixes an issue in Docker where mounting empty directories to disable docker check results in an error.
- Fixes the matching of container id in Tagger (due to runtime prefix) by matching on the 'id' part only
- Fix the node roles to host tags feature by handling the other official Kube way to setting node roles (when multiple roles are required)
- Properly check for "true" value of env var DD_LEADER_ELECTION
- It's possible now to reduce the risk of missing kubernetes tags on initial logs by configuring "logs_config.tagger_warmup_duration". Configuring "logs_config.tagger_warmup_duration" delays the send of the first logs of a container. Default value 0 seconds, the fix is disabled by default. Setting "logs_config.tagger_warmup_duration" to 5 (seconds) should be enough to retrieve all the tags.
- Fix eBPF code compilation errors about
asm gotoon Ubuntu 19.04 (Disco) - Fix race condition in singleton initialization
- On Windows, fixes registration of agent as event log source. Allows agent to correctly write to the Windows event log.
- On Windows, when upgrading, installer will fail if the user attempts to assign a configuration file directory or binary directory that is different from the original.
- Add logic to support docker restart of containers.
- Fix a Network Performance Monitoring issue where TCP connection direction was incorrectly classified as
outgoingin containerized environments. - Fixed a few edge cases that could lead to events payloads being rejected by Datadog's intake for being too big.
- Upgrade embedded dependencies:
curlto7.66.0,autoconfto2.69,procpsto3.3.16 - JMXFetch upgraded to 0.34.0
- Bump embedded Python 3 to 3.7.6
Release on: 2020-01-06
- Please refer to the 7.16.1 tag on integrations-core for the list of changes on the Core Checks
- UnixODBC software dependency bumped to 2.3.7 to address CVE-2018-7409.
Release on: 2019-12-18
This release introduces major version 7 of the Datadog Agent, which starts at v7.16.0. The only change from Agent v6 is that v7 defaults to Python 3 and only includes support for Python 3. Before upgrading to v7, confirm that any custom checks you have are compatible with Python 3. See this guide for more information.
Except for the supported Python versions, v7.16.0 and v6.16.0 have the same features.
Please refer to the 7.16.0 tag on integrations-core for the list of changes on the Core Checks
- Add support for SysVInit on SUSE 11.
- Add information on endpoints inside the logs-agent section of the agent status.
Add Python 3 linter results to status page
Log a warning when the hostname defined in the configuration will not be used as the in-app hostname.
Add
ignore_autodiscovery_tagsparameter config check.In some cases, a check should not receive tags coming from the autodiscovery listeners. By default
ignore_autodiscovery_tagsis set to false which doesn't change the behavior of the checks. The first check that will use it iskubernetes_state.Adds a new
flare_stripped_keysconfig setting to clean up additional configuration information from flare.Adding a new config option
exclude_gce_tags, to configure which metadata attribute from Google Cloud Engine to exclude from being converted into host tags.Extends the docker and containerd checks to include an open file descriptors metric. This metric reports the number of open file descriptors per container.
Allow the Agent to schedule different checks from different sources on the same service.
- APM: Added a fallback into the SQL obfuscator to handle SQL engines that treat backslashes literally.
- The default list of sensitive keywords for process argument scrubbing now uses wildcards before and after.
- On Windows process agent, fix problem wherein if the agent is unable to figure out the process user name, the process info/stats were not sent at all. Now sends all relevant stats without the username
- On windows, correctly deletes python 3 precompiled files (.pyc) in the event of an installation failure and rollback
- Logs: tailed files discovered through a configuration entry with
wildcard will properly have the
dirnametag on all log entries. - Fix small memory leak in
datadog_agent.set_external_tagswhen an empty source_type dict is passed for a given hostname. - Carry a custom patch for jaydebeapi to support latest jpype.
- Check that cluster-name provided by configuraiton file are compliant with the same rule as on GKE. Logs an error and ignore it otherwise.
- JMXFetch upgraded to 0.33.1
- JQuery, used in the web base agent GUI, has been upgraded to 3.4.1
Release on: 2019-11-27 This release was published for Windows on 2019-12-09.
- Collect IP address from containers in awsvpc mode
- Reintroduce legacy checks directory to make legacy AgentCheck import path
(
from checks import AgentCheck) work again. - Systemd integration points are re-ordered so that
dbusis used in preference to the systemd private API at/run/systemd/private, as per the systemd documentation. This prevents unnecessary logging to the system journal when datadog-agent is run without root permissions.
- Bump embedded Python to 2.7.17.
Release on: 2019-11-05
- Please refer to the 6.15.0 tag on integrations-core for the list of changes on the Core Checks
- Add persistent volume claim as tag (
persistentvolumeclaim:<pvc_name>) to StatefulSets pods. - APM: On SQL obfuscation errors, a detailed explanation is presented when DEBUG logging level is enabled.
- APM: SQL obfuscation now supports queries with UTF-8 characters.
- Augment network data with DNS information.
- Add an option to disable the cluster agent local fallback for tag collection (disabled by default).
- DNS lookup information is now included with network data via system-probe.
- Add support for the XX:+UseContainerSupport JVM option through the jmx_use_container_support configuration option.
- The Cluster Agent can now collect stats from Cluster Level Check runners to optimize its dispatching logic and rebalance the scheduled checks.
- Add a new python API to store and retrieve data. datadog_agent.write_persistent_cache(key, value) persists the data in value (as a string), whereas datadog_agent.read_persistent_cache(key) returns it for usage afterwards.
- Adding support for compression when forwarding logs through HTTPS. Enable it by following instructions here
- Migrate the api version of the Deployment and DaemonSet kubernetes objects to apps/v1 as older bersions are not supported anymore in k8s 1.16.
- Running the command check jmx now runs once JMXFetch with the with-metrics command instead of just displaying an error.
- Add options
tracemalloc_whitelistandtracemalloc_blacklistfor allowing the use of tracemalloc only for specific checks. - APM: a warning is now issued when important HTTP headers are omitted by clients.
- The system-probe will no longer log excessively when its internal copy of the conntrack table
is full. Furthermore, the artificial cap of 65536 on
system_probe_config.max_tracked_connections, which controlled the maximum number of conntrack entries seen by the system-probe has been lifted. - Allow filtering of event types,reason and kind at query time. Make the event limit configurable. Improve the interaction with the ConfigMap to store the Resource Version.
- The agent will now try to flush data to the backend when before exiting (from DogStatsD and checks). This avoid having metrics gap when restarting the agent. This behavior can be disable through configuration, see aggregator_stop_timeout and forwarder_stop_timeout.
- Expose metrics for the cluster level checks advanced dispatching.
- Implement API that allows Python checks to send metadata using
the
inventoriesprovider.
- The ddagentuser no longer has write access to the process-agent binary on Windows
- Avoid the tagger to log a warning when a docker container is not found.
- Use
pkg_resourcesto collect the version of the integrations instead of importing them. - On Windows, allow the uninstall to succeed even if the removal of the ddagentuser fails for some reason.
- APM: double-quoted strings following assignments are now correctly obfuscated.
- APM: Fixed a bug where an inactive ratelimiter would skew stats.
- Fix an issue where the node agent would not retry to connect to the cluster agent for tag collection.
- Fix the appearrance of the status bar icon when using dark mode on macOS
- The process-agent and system-probe agents should ignore SIGPIPE signals.
- Fix the behavior of the diagnose command that would not consider default configuration location when run independently
- Fix a bug where the agent would crash when using the docker autodiscovery config provider.
- Do not permit sending events at their first timestamp.
- Fix tag support for NTP check.
- Fixes a typo in the windows service related commands for the process agent CLI. Was previously referencing trace-agent.
- On Windows, properly installs on Read Only Domain Controller. Adds rights to domain-created user in local GPOs.
- Behavioral change on the forwarder healthcheck such that full queues will not label the forwarder as unhealthy. Networking or endpoint issues are not representative of an unhealthy agent or forwarder.
- The agent is now more resilient to incomplete responses from the kubelet
- On Linux, preserve the script /opt/datadog-agent/embedded/bin/2to3 that relies on the python 2 interpreter, alongside the python 3 one.
- Fix a possible race in autodiscovery where checks & log collection would be wrongly unscheduled.
- Minor memory leaks identified and fixed in RTLoader.
- On Windows, fixes installation logging to not include certain sensitive data (specifically api key and the ddagentuser password)
- Fixed a few edge cases that could lead to service checks payloads being rejected by Datadog's intake for being too big
- Use pylint directly for py3 validation, removing dependency on a7.
- JMXFetch upgraded from 0.31.0 to 0.32.1.
- JMXFetch upgraded from 0.32.1 to 0.33.0 https://github.com/DataDog/jmxfetch/blob/master/CHANGELOG.md#0330--2019-10-10
Release on: 2019-12-05
This release is only available for Windows.
- On windows, fixes problem where Agent would intermittently fail to install on domain-joined machine, when another Agent was already installed on the DC.
Released on: 2019-10-29
This release is only available for Windows.
- On Windows, allows the install to succeed successfully even in the event that the user was not cleaned up successfully from a previous install.
- On Windows, do not remove the home folder of the Agent's user (dd-agent-user) on uninstall.
Release on: 2019-09-26
- Disable debug log lines for the 'hostname' command since it's directly called by some Agent components. Fixes hostname resolution issues for APM and Live Process.
Release on: 2019-09-16
- Please refer to the 6.14.0 tag on integrations-core for the list of changes on the Core Checks
- The GPG key used to sign the Agent RPM packages has been rotated. See the dedicated Agent documentation page to know how to make sure that the new Agent RPM packages can be installed on hosts.
- Update to the configuration of the systemd check:
unit_namesis now required and only matching units will be monitored,unit_regexesconfiguration has been removed. - Several metrics sent by the systemd check have been renamed. The integration is now stable.
- All integrations that make HTTP(S) requests now properly fall back to proxy settings defined in
datadog.yamlif none are specified at the integration level. If this is undesired, you can setskip_proxytotruein every instance config or in theinit_configfallback.
- APM: add support for container tagging. It can be used with any client tracer that supports it.
- APM: Incoming TCP connections are now measured in the datadog.trace_agent.receiver.tcp_connections metrics with a "status" tag having values: "accepted", "rejected", "timedout" and "errored".
- Allows the user to blacklist source and destination connections by passing IPs or CIDRs as well as port numbers.
- Docker label autodiscovery configurations are now polled more often by default.
- The Agent can now expose runner stats via the CLC Runner API Server, a remotely-accessible authenticated API server. The Cluster Agent can use these stats to optimize dispatching cluster level checks. The CLC Runner API Server is disabled by default, it must be enabled in the Agent configuration, also the cluster agent must be enabled since it's the only client of the server. By default, the server listens on 5005 and its host address must be set to the Agent Pod IP using the Kubernetes downward API.
- [preview] Checks can now send histogram buckets to the agent to be sent as distribution metrics.
- In macOS datadog-agent is now able to start/stop process-agent.
- The Agent now includes a Python 3 runtime to run checks. By default, the Python 2 runtime is used. See the dedicated Agent documentation page for details on how to configure the Agent to use the Python 3 runtime and how to migrate checks from Python 2 to Python 3.
- High-level RTLoader memory usage statistics exposed as expvars on the agent.
- Adding tracemalloc_debug configuration setting (Python3 only). Enables Tracemalloc memory profiling on Python3. Enabling this option will override the number of check runners to 1 to guarantee sequential execution of checks.
- For NTP check, add the option
use_local_defined_servers. Whenuse_local_defined_serversis true, use the ntp servers defined in the current host otherwise use the hosts defined in the configuration.
- Show configuration source for each check's instance in the "status" and the "configcheck" commands.
- Add a new invoke task,
rtloader.generate-docwhich generates Doxygen documentation for the rtloader directory and warns about documentation errors or warnings. - Allow the check command to display and/or store memory profiling data.
- For Windows, add a message when the user cannot perform the action in the systray.
- APM: The datadog.trace_agent.normalizer.traces_dropped metric now has a new reason payload_too_large which was confusingly merged with decoding_error.
- APM: Bind
apm_config.replace_tagsparameter toDD_APM_REPLACE_TAGSenvironment variable. It accepts a JSON formatted string of the form[{"name":"tag_name","pattern":"pattern","repl":"repl_str"}] - The default collection interval for host metadata has been reduced from 4 hours to 30 min.
- Collection interval for the default metadata providers ('host', 'agent_checks' and 'resources') can now be configured using the 'metadata_providers' configuration entry.
- Agent commands now honor the DD_LOG_LEVEL env variable if set.
- Distributions: Distribution payloads are now compressed before being sent to Datadog if the agent is built with either zlib or zstd.
- Configuration files for core checks in cmd/agent/dist/conf.d/ have been migated to the new configuration file norm. https://docs.datadoghq.com/developers/integrations/new_check_howto/#configuration-file
- When a valid command is passed to the agent but the command fails, don't display the help usage message.
- Add
private_socketconfiguration to the systemd check. Defaults to/run/systemd/private(or/host/run/systemd/privatewhen using Docker Agent). - Warnings returned by the Python 3 linter for custom checks are now logged in the Agent at the 'debug' level.
- Make NTP check less verbose when a host can't be reached. Warn only after 10 consecutive errors.
- Added detection of a network ID which will be used to improve destination resolution of network connections.
- Windows events will now display a full text message instead of a JSON object. When available, the agent will now enrich the events with status, human readable task name and opcode.
- On Windows, adds system.mem.pagefile.* stats, previously available only in Agent 5.
- The
--log-levelargument inagent checkandagent jmxcommands has been deprecated in favor of the DD_LOG_LEVEL env variable.
- APM: The
datadog.trace_agent.receiver.payload_refusedmetric now has language tags like its peer metrics. - The
agent jmxcommand now correctly takes into account the options in the init_config section of the JMXFetch integration configs - Escape message when using JSON log format. This, for example, fixes multiline JSON payload when logging a Exception from Python.
- Fix a bug, when a check have its init configuration before that all the tagger collector report tags.
- Fix spikes for
system.io.avg_q_szmetrics on Linux when the kernel counter was wrapping back to 0. - Fix system.io.* metrics on Linux that were off by 1 when the kernel counters were wrapping back to 0.
- Fixed placeholder value for the marathon entry point to match the new configuration file layout.
- Fix a
taggergoroutine race issue when adding a new entry in thetagger.Storeand requesting an entry in another goroutine. - Fix files descriptor leak when tailing a logs file with file rotation and the tailer is stuck for instance because of lost connectivity with the logs intake endpoint.
- The parameter
jmxis not supported with the commandcheck, thejmxcommand should be used instead. - Fixed NTP timeout not being used from the configuration.
- On Windows, correctly configure the config file if the path includes a space.
- When uninstalling the agent, remove ddagentuser home folder.
- APM: Fix incorrect
traces_droppedandspans_malformedmetric counts. - On Windows, "ddagentuser" (the user context under which the Agent runs), is now added to the "Event Log Readers" group, granting access to Security event logs.
- The Windows agent no longer depends on the Windows WMI service. If the WMI service stops for any reason, the Windows agent will no longer stop with it. However, any integrations that do use WMI (wmi_check and win32_event_log) will not be able to function until the WMI service restarts.
- Ignore the containerd startup script and the kubeconfig as part of the host metadata on GKE.
- JMXFetch upgraded to 0.31.0
- On Windows, during an uninstall, if the user context for the datadog agent is a domain user, the user will no longer be deleted even when the user was created by the corresponding install.
Release on: 2019-07-24
- Please refer to the 6.13.0 tag on integrations-core for the list of changes on the Core Checks
- The
portoption in the NTP check configuration is now parsed as an integer instead of a string.
- APM: add support for Unix Domain Sockets by means of the apm_config.receiver_socket configuration. It is off by default. When set, it must point to a valid sock file.
- APM: API emitted metrics now have a lang_vendor tag when the Datadog-Meta-Lang-Vendor HTTP header is sent by clients.
- APM: Resource-based rate limiting in the API can now be completely disabled by setting apm_config.max_memory and/or apm_config.max_cpu_percent to the value 0.
- Add support for environment variables in checks' config files using the format "%%env_XXXX%%".
- Add new systemd integration to monitor systemd itself and the units managed by systemd.
- The total number of bytes received by dogstatsd is now reported by the dogstatsd-udp/Bytes and dogstatsd-uds/Bytes expvar.
- Adds the ability to use DD_TAGS to set global tags in Fargate.
- Added a support for the new pod log directory pattern introduced in version 1.14 of Kubernetes to make sure the agent keeps on collecting logs after upgrade of a Kubernetes cluster.
- Add a kube_cronjob tag in the tagger. It applies to container metrics, autodiscovery metrics and logs.
- Change the prefix of entity IDs to make it easier to query the tagger without knowing what the container runtime is.
- APM: reduce memory usage in high traffic by up to 10x.
- APM: Services are no longer aggreagated in the agent, nor written to the Datadog API. Instead, they are now automatically extracted on the backend based on the received traces.
- APM: The default interval at which the agent watches its resource usage has been reduced from 20s to 10s.
- APM: Improved processing concurrency and as a result, CPU usage decreased by 20% in some scenarios.
- APM: Queued sender was rewritten to improve performance around scenarios where network problems are present.
- APM: Code clean up around configuration and writer.
- The datadog-agent version command now prints the version of Golang the agent was compiled with.
- Display Go version in output of status command
- Upgraded JMXFetch to 0.30.0. See https://github.com/DataDog/jmxfetch/releases/tag/0.30.0
- APM: the trace agent now lets through a wider variety of traces, automatically correcting some malformed traces instead of dropping them. The following fields are now replaced with reasonable defaults if invalid or empty and truncated if exceeding max length: span.service, span.name, span.resource, span.type. span.duration=0 is now allowed. Missing span start date now defaults to duration - now. The datadog.trace_agent.receiver.traces_dropped metric is now tagged with a reason tag explaining the reason it was dropped. There is a new datadog.trace_agent.receiver.spans_malformed metric also tagged by reason explaining how the span was malformed.
- Refactored permissions check in the integration command.
- Support Python 3 for the integration command.
- APM: The presampler has been rebranded as a "rate limiter" to avoid confusing it with other sampling mechanisms.
- APM: The "datadog.trace_agent.presampler_rate" metric has been deprecated in favor of "datadog.trace_agent.receiver.ratelimit".
- On Windows, quote the service name when registering service. Mitigates CVE-2014-5455. Note that since the Agent is not running as admin, even a successful attack would not give admin rights as specified in the CVE.
- Fix the tagger behavior returning None when no tags are present for the kubelet and fargate integration.
- APM: metrics generated by the processing function (such as *.traces_priority) now contain language specific tags.
- APM: Memory spikes when retry queue grows have been fixed.
- Fix 'vcruntime140.dll is being held in use by the following process'
- System-probe s6 services: ensure that the system-probe binary is bundled before trying to run it / stop it. This is to ensure that the s6-services definitions will be backward compatible with older builds that didn't have the system-probe yet.
- Fix a bug in the log scanning logic of the JMXFetch wrapper that would make JMXFetch hang if it logged a very large log entry
- Fixed an issue where logs collected from kubernetes using '/var/log/pods' would show up with a wrong format '{"log":"x","stream":"y","time":"z"}' on the logs explorer when using docker as container runtime.
- Fix TLS connection handshake that hang forever making the whole logs pipeline to be stucked resulting in logs not being tailed and file descriptor not being closed.
- On Windows, fixes bug in which Agent can't start if the Go runtime can't determine the ddagentuser's profile directory. This information isn't used, so shouldn't cause a failure
- The External Metrics Setter no longer stops trying to get metrics after 3 failed attempts. Instead, it will retry indefinitely.
- Removes an unused duplicate copy of the
system-probebinary from the Linux packages - The NTP check now properly uses the
portconfiguration option.
- Logs informing about check runs and payload submission are now displayed once every 500 events instead of every 20 events.
Release on: 2019-07-03
This release is only available on Windows and contains all the changes introduced in 6.12.0 and 6.12.1.
- Please refer to the 6.12.2 tag on integrations-core for the list of changes on the Core Checks
Release on: 2019-06-28
This release is not available on Windows.
- Please refer to the 6.12.1 tag on integrations-core for the list of changes on the Core Checks
- Fixed a bug in the kubelet and fargate integrations preventing the collection of the
kubernetes.cpu.*andkubernetes.memory.*metrics.
Some metrics from the kubelet and fargate integrations (kubernetes.cpu.* and kubernetes.memory.*) are missing for certain configurations.
A fix will be released in v6.12.1. Meanwhile if downgrading to 6.11.3 is not an option we recommend using the runtime metrics
(ex: docker.cpu.*, docker.mem.*, containerd.cpu.*, ...).
Release on: 2019-06-26
This release is not available on Windows.
- Please refer to the 6.12.0 tag on integrations-core for the list of changes on the Core Checks
APM: Log throttling is now automatically enabled by default when log_level differs from debug. A maximum of no more than 10 error messages every 10 seconds will be displayed. If you had it enabled before,
it can now be removed from the config file.
On Windows, the path of the embedded
python.exebinary has changed from%ProgramFiles%\Datadog\Datadog Agent\embedded\python.exeto%ProgramFiles%\Datadog\Datadog Agent\embedded2\python.exe. If you use this path from your provisioning scripts, please update it accordingly. Note: on Windows, to call the embedded pip directly, please use%ProgramFiles%\Datadog\Datadog Agent\embedded2\python.exe -m pip.Logs: Breaking Change for Kubernetes log collection - In the version 6.11.2 logic was added in the Agent to first look for K8s container files if /var/log/pods was not available and then to go for the Docker socket. This created some permission issues as /var/log/pods can be a symlink in some configuration and the Agent also needed access to the symlink directory.
This logic is reverted to its prior behaviour which prioritise the Docker socket for container log collection. It is still possible to force the agent to go for the K8s log files even if the Docker socket is mounted by using the logs_config.k8s_container_use_file' or `DD_LOGS_CONFIG_K8S_CONTAINER_USE_FILE. parameter. This is recommended when more than 10 containers are running on the same pod.
- A count named
datadog.agent.startedis now sent with a value of 1 when the agent starts. - APM: Maximum allowed CPU percentage usage is now configurable via DD_APM_MAX_CPU_PERCENT.
- Node Agent can now perform checks on kubernetes service endpoints. It consumes the check configs from the Cluster Agent API via the endpointschecks config provider. Versions 1.3.0+ of the Cluster Agent are required for this feature.
- Logs can now be collected from init and stopped containers (possibly short-lived).
- Allow tracking pod labels and annotations value change to update labels/annotations_as_tags. Make the explicit tagging feature dynamic (introduced in DataDog#3024).
- APM: the writer will now flush based on an estimated number of bytes in accumulated buffer size, as opposed to a maximum number of spans, resulting in better traffic and payload size management.
- APM: traces are not dropped anymore because or rate limiting due to performance issues. Instead, the trace is kept in a queue awaiting to be processed.
- Logs docker container ID when parse invalid docker log in DEBUG level.
- Set the User-Agent string to include the agent name and version string.
- Adds host tags in the Hostname section of the agent status command and the status tab of the GUI.
- Expose the number of logs processed and sent to the agent status
- Added a warning message on agent status command and status gui tab when ntp offset is too large and may result in metrics ignored by Datadog.
- APM: minor improvements to CPU performance.
- APM: improved trace writer performance by introducing concurrent writing.
- APM: the stats writer now writes concurrently to the Datadog API, improving resource usage and processing speed of the trace-agent.
- Extends the docker check to accommodate the kernel memory usage metric. This metric shows the cgroup current kernel memory allocation.
- Ask confirmation before overwriting the output file while using the dogstatsd-stats command.
- Do not ship autotools within the Agent package.
- The
datadog-agent integrationsubcommand is now capable of installing prereleases of official integration wheels - Upgraded JMXFetch to 0.29.1. See https://github.com/DataDog/jmxfetch/releases/tag/0.28.0, https://github.com/DataDog/jmxfetch/releases/tag/0.29.0 and https://github.com/DataDog/jmxfetch/releases/tag/0.29.1
- Added validity checks to NTP responses
- Allow the '--check_period' flag of jmxfetch to be overriden by the DD_JMX_CHECK_PERIOD environment variable.
- Ship integrations and their dependencies on Python 3 in Omnibus.
- Added a warning about unknown keys in datadog.yaml.
- APM: the yaml setting apm_config.trace_writer.max_spans_per_payload is no longer in use; writes are now based solely on accumulated byte size.
- Updated the DataDog/gopsutil library to include changes related to excessive DEBUG logging in the process agent
- The computeMem is only called in the check when we ensure that it does not get passed with an empty pointer. But if someone was to reuse it without checking for the nil pointer it could cause a segfault. This PR moves the nil checking logic inside the function to ensure it is safe.
- APM: Fixed a bug where normalize tag would not truncate tags correctly in some situations.
- APM: Fixed a small issue with normalizing tags that contained the unicode replacement character.
- APM: fixed a bug where modulo operators caused SQL obfuscation to fail.
- Fix issue on process agent for DD_PROCESS_AGENT_ENABLED where 'false' did not turn off process/container collection.
- Fix an error when adding a custom check config through the GUI when the folder where the config will reside does not exist yet.
- APM: on macOS, trace-agent is now enabled by default, and, similarly to other platforms, can be enabled/disabled with the apm_config.enabled config setting or the DD_APM_ENABLED env var
- Fix a bug where when the log agent is mis-configured, it temporarily hog on resources after being killed
- Fix a potential crash when doing a
configcheckwhile the agent was not properly initialized yet. - Fix a crash that could occur when having trouble connecting to the Kubelet.
- Fix nil pointer access for container without memory cgroups.
- Improved credentials scrubbing logic.
- The
datadog-agent integration showsubcommand now properly accepts only Datadog integrations as argument - Fix incorrectly reported IO metrics when OS counters wrap in Linux.
- Fixed JMXFetch process not being terminated on Windows in certain cases.
- Empty logs could appear when collecting Docker logs in addition to the actual container logs. This was due to the way the Agent handles the header Docker adds to the logs. The process has been changed to make sure that no empty logs are generated.
- Fix bug when docker container terminate the last logs are missing and partially recovered from restart.
- Properly move configuration files for wheels installed locally via the
integrationcommand. - Reduced memory usage of the flare command
- Use a custom patch for a costly regex in PyYAML, see yaml/pyyaml#301.
- On Windows, restore the
system.mem.pagefile.pct_freemetric
- The 'integration freeze' cli subcommand now only displays datadog packages instead of the complete result of the 'pip freeze' command.
- The Secrets Management feature is no longer in beta.
Release on: 2019-06-04
- Please refer to the 6.11.3 tag on process-agent for the list of changes on the Process Agent.
- Upgrade JMXFetch to 0.27.1
- APM: fixed a bug where secrets in environment variables were ignored.
Release on: 2019-05-23
- Add option cf_os_hostname_aliasing to send the OS hostname as an alias when using the BOSH agent on Cloud Foundry.
- Fixes problem in which Windows Agent wouldn't install on non-English machines due to assumption that "Performance Monitor Users" didn't need to be localized.
- Windows Installer is now more resilient to missing domain controller.
Important: 6.11.1 is not marked as latest for Windows: we are
investigating some cases where 6.11.0 and 6.11.1 are not installing correctly
on Windows.
Downloading datadog-agent-6-latest.amd64.msi will give you version 6.10.1.
Release on: 2019-05-06
- Please refer to the 6.11.0 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.11.1 tag on process-agent for the list of changes on the Process Agent.
- Change the prioritization between the two logic that we have to collect logs on Kubernetes. Now attempt first to collect logs on '/var/log/pods' and fallback to using the docker socket if the initialization failed.
- Fix a bug where short image name wouldn't be properly set on old docker versions
- Properly handle docker container logs in multiline mode in case of infrequence log messages, log file rotations or agent restart
Release on: 2019-04-17
- Please refer to the 6.11.0 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.11.0 tag on process-agent for the list of changes on the Process Agent.
- APM: move flush notifications from level "INFO" to "DEBUG"
- APM: logging format has been changed to match the format of the core agent.
- Metrics coming through dogstatsd with the following internal prefixes:
activemq,activemq_58,cassandra,jvm,presto,solr,tomcat,kafka,datadog.trace_agent,datadog.process,datadog.agent,datadog.dogstatsdare no longer affected by thestatsd_metric_namespaceoption. - Removed the internal ability to send logs to a specific logset at agent level.
- On Windows, the Datadog Agent now runs as a non-privileged user (ddagentuser by default) rather than LOCAL_SYSTEM. Please refer to our dedicated docs for more information
- The Windows installer will no longer allow direct downgrades; if a downgrade is required, the user must uninstall the newer version and install the older version.
- Secrets beta feature is now available on windows allowing users to pull secrets from secret management services.
- APM: JSON logging is now supported using the log_format_json: true setting.
- Collect container thread count and thread limit
- JMXFetch upgraded to 0.27.0. See 0.27.0 for more details.
- The agent now ignores pod that exited more than 15 minutes ago to reduce its resource footprint when pods are not garbage-collected. This is configurable with the kubernetes_pod_expiration_duration option.
- Now support CRI-O container runtime for log collection on Kubernetes.
- Automatically add a "dirname" tag representing the directory of logs tailed from a wildcard path.
- AutoDiscovery can now monitor unready pods. It looks for a new pod annotation "ad.datadoghq.com/tolerate-unready" which, if set to true will make AutoDiscovery monitor that pod regardless of its readiness state.
- Add the ability for the
datadog-agent checkcommand to have Python checks start an interactive debugging session. - Change the logging format to include the name of the logging agent instead of appending it in the agent container logs.
- Add /metrics to the bare endpoints the agent can access. This is required to support querying endpoints protected by RBAC, by kube-rbac-proxy for instance.
- APM: errors reported by the receiver's HTTP server are now shown in the logs.
- APM: slightly improved normalization error logs.
- On Windows, allows Agent to be installed to nonstandard directories. Uses APPLICATIONDATADIRECTORY to set the root of the configuration file tree, and PROJECTLOCATION to set the root of the binary tree. Please refer to the docs for more details
- In order to decrease the number of API DCA requests, The Agent now uses a different API endpoint to call the DCA's API only once in order to retrieve the Pods metadata.
- Host metadata payloads are now zlib-compressed
- Log file size and number of rotation is now configurable.
- Add a command dogstatsd-stats to the agent to get basic stats about the processed metrics.
- Support JSON arrays within environment variables, in addition to space separated values.
- On Google Compute Engine, the Agent now reports <instance_name>.<project_id> as a host alias instead of <hostname_prefix>.<prefix_id>, which improves the uniqueness and relevance of the host alias when the GCE instance has a custom hostname.
- The import command doesn't stop anymore when there is no
conf.dorauto_confdirectory. - Kubernetes event collection timeout can now be configured.
- Improve status page by splitting errors and warnings from the Logs agent
- Secrets are no longer decrypted in agent command when it's not needed (commands like hostname, launchgui, configuration ...). This reduce the number of times the 'secret_command_backend' executable will be called.
- Improved memory efficiency on hosts sending very high numbers of metrics.
- Resolve once the DNS name given by docker and try the associated IP to reach the kubelet. Prioritize HTTPS over HTTP to connect to kubelet. Prioritize communication using IPs over hostnames to spare DNS servers accross the cluster.
- Removal of largely unused go SNMP check. SNMP support still provided by the python variant.
- Fix an auto-discovery annotation value parsing limitation in version 6
compared to version 5.
Now,
ad.datadoghq.com/*.instancesannotation key supports value like[[{"foo":"bar1"}, {"foo":"bar2"}], {"name":"bar3"}] - The agent container will now output valid JSON when using JSON log format.
- APM: Multiple value "Content-Type" headers are now parsed correctly for media type in the HTTP receiver.
- APM: always reply with correct Content-Type in API responses.
- APM: when a span's resource is empty, the error "Resource can not be empty" will be returned instead of the wrong "Resource is invalid UTF-8".
- APM: sensitive information is now scrubbed from logs.
- APM: Fix issue with --version flag when API key is unset.
- APM: Ensure UTF-8 characters are not cut mid-way when truncating span fields.
- Metrics coming through dogstatsd with the following internal prefixes:
activemq,activemq_58,cassandra,jvm`, ``presto,solr,tomcat,kafka,datadog.trace_agent,datadog.process,datadog.agent,datadog.dogstatsdare no longer affected by thestatsd_metric_namespaceoption. - Fixes ec2 tags collection when datadog agent is deployed into a kubernetes cluster along with kube2iam.
- Fixes bug in which upgrading from agent5 doesn't correctly import the configuration
- Fix a race condition in gohai that could make the Agent crash while collecting the host's filesystem metadata
- Hostnames containing characters that are invalid for a filename no longer prevent the agent from generating a flare.
- Allow macOS users to invoke the datadog-agent integration command as root since the installation directory is owned by root.
- Change to a randomized exponential backoff in case of connection failure
- Ignore empty logs_dd_url to fall back on default config for logs agent.
- Detect and handle Docker logs with only header and empty content
- To mitigate issues with the hostname detection on AKS, hostnames gathered from the metadata endpoints of AWS, GCE, Azure, and Alibaba cloud are no longer considered valid if their length exceeds 255 characters.
- Bump embedded Python to 2.7.16
Release on: 2019-03-20
- Fix a race condition in Autodiscovery leading to some checks not being unscheduled on container exit
Release on: 2019-03-07
- APM: Mixing cases in apm_config.analyzed_spans and apm_config.analyzed_rate_by_service entries is now allowed. Service names and operation names will be treated as case insensitive.
- Refactor the
ContainerdUtilso that each call to thecontainerdapi has a dedicated timeout.
Release on: 2019-02-28
- Please refer to the 6.10.0 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.10.0 tag on process-agent for the list of changes on the Process Agent.
- Starting with this release, the changes on the Trace Agent are listed in the present release notes.
- The Agent now defaults to aliasing yaml.load and yaml.dump to yaml.safe_load and yaml.safe_dump for ALL checks as a defense-in-depth measure against CVE-2017-18342. The Datadog Agent does not use the vulnerable code directly. The effort to patch the PyYAML library guards against the accidental unsafe use of this library by custom checks and transitive dependencies. Specifically, the kubernetes client library v8.0.1 calls the unsafe yaml.load function, but the fix provided forces the use of yaml.safe_load by default. In this release of the Agent, kubernetes client library v8.0.1 is only used by the new kube_controller_manager integration. If for any reason you encounter problems with your custom checks, please reach out to support.
- Introduce pod and container tagging through annotations.
- Docker images are now signed with Content Trust to ensure their integrity when pulling
- Dogstatsd can now inject extra tags on a metric when a special entity tag is provided
datadog-agent integration installcommand allows to install a check from a locally available wheel (.whl file) with the added parameter--local-wheel.- JMXFetch upgraded to 0.26.1: introduces concurrent metric collection across
JMXFetch check instances. Concurrent collection should avoid stalling
healthy instances in the event of issues (networking, system) in
any of the remaining instances configured. A timeout of
jmx_collection_timeout(default 60s) is enforced on the whole metric collection run. See 0.25.0, 0.26.0 and 0.26.1. - Added the possibility to define global logs processing rules in datadog.yaml that will be applied to all logs, in addition to integration logs processing rules when defined.
- Consider static pods as ready, even though their status is never updated in the pod list. This creates the risk of running checks against pods that are not actually ready, but this is necessary to make autodiscovery work on static pods (which are used in standard kops deployments for example).
- Adds the device mapper logical volume name as a tag in the system.io infos.
- Extends the docker check to accommodate the failed memory count metric. This metric increments every time a cgroup hits its memory limit
- Add a
--jsonflag to thecheckcommand that will output all aggregator data as JSON. - [tagger] Add pod phase to kubelet collector
- The Agent logs now contains the relative file path (including the package) instead of only the filename.
- Each corecheck could now send custom tags using
the
tagsfield in its configuration file. - ECS: running the agent in awsvpc mode is now supported, provided it runs in a security group that can reach both the containers to monitor and the host via its private IP on port 51678
- The performance of the Agent under DogStatsD load has been improved.
- Improve memory usage when metrics, service checks or events contain many tags.
- APM: improve performance of NormalizeTag function.
- Use dedicated
datadog_checks_downloaderto securely download integrations wheels when using thedatadog-agent integration installcommand. - A warning is now displayed in the status when the connection to the log endpoint cannot be established
- When shutting the agent down, cancel ongoing python subprocess so they can exit as cleanly and gracefully as possible.
- Add of a "secrets" command to show information about decrypted secrets. We now also track the configuration's name where each secrets was found.
- Secrets are now resolved in environment variables.
- In order to ensure compatibility with systemd < 229,
StartLimitBurstandStartLimitIntervalhave been moved to the Service section of the service files. - Files are not tailed in reverse lexicographical order w.r.t their file names then dir name. If you have files /1/2017.log, /1/2018.log, /2/2018.log and logs_config.open_files_limit == 2, then you will tail /2/2018.log and /1/2018.log.
- Include
.ymlconfiguration files in the flare.
- Fix an issue where some auto-discovered integrations would not get rescheduled when the template was not containing variables
- Autodiscovery now removes children configurations when removing templates
- Fix the display of unresolved configs in the verbose output of the
configcheckcommand - Fix custom command line port configuration on configcheck and tagger-list CLI commands.
- When the secrets feature is enabled, fix bug preventing the
additional_endpointsconfig option from being read correctly - Fix "status" command JSON output to exclude non JSON header. The output of the command is now a valid JSON payload.
- APM: Fix a potential memory leak problem when the trace agent is stopped.
- Fixed a bug where logs forwarded by UDP would not be split because of missing line feed character at the end of a datagram. Now adding a line feed character at the end of each frame is deprecated because it is automatically added by the agent on read operations.
- Fix an issue where some kubernetes tags would not be properly removed.
- The Agent is now compiled with Go 1.11.5
- Custom checks default on safe pyyaml methods.
Release on: 2019-01-22
- Please refer to the 6.9.0 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.9.0 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.9.0 tag on process-agent for the list of changes on the Process Agent.
- On EC2 hosts that were upgraded from Agent 5.x using the install script or that have the
hostname_fqdnoption enabled if your hostname currently begins withip-ordomU-(default EC2 hostnames) your hostname will change to the EC2 instance ID. Example:ip-10-1-1-1.ec2.internal=>i-1234567890abcdef0. This is an effort to fix a bug in the hostname resolution that was introduced in the version 6.3 of the Agent. - Kubernetes logs integration is now automatically enabled if it can find
/var/log/pods. Iflogs_config.container_collect_allis not enabled, only pods with Datadog logs annotation will be collected. Iflogs_config.container_collect_allis enabled, logs for all pods (matchingac_excludeandac_includefilters if applicable) will be collected.
- Introduce a way to configure the cardinality level of tags that the tagger should return. This is split between two options - one for checks and one for dogstatsd. The three cardinality levels are High, Orchestrator, and Low. Checks get Low and Orchestrator-level tags by default Dogstatsd benefits from Low-card tags only by default.
- You can add extra listeners and config providers via the
DD_EXTRA_LISTENERSandDD_EXTRA_CONFIG_PROVIDERSenviroment variables. They will be added on top of the ones defined in thelistenersandconfig_providerssection of the datadog.yaml configuration file. - Adding native containerd check, based on the containerd socket.
- You can now see an extra instance id when displaying the Agent status depending on the check.
If the instance contains an attribute
nameornamespace, it will be displayed next to the instance id. - Added a new
container_cgroup_prefixoption to fix some cases where system slices were detected as containers. - Add
datadog-agent integration show [package]command to show information about an installed integration.
- AutoDiscovery can now monitor unready pods. It looks for a new pod annotation "ad.datadoghq.com/tolerate-unready" which, if set to true will make AutoDiscovery monitor that pod regardless of its readiness state.
- Add debug information about the secrets feature to the flare.
- On the
checkcommand, add a pause of 1sec between the 2 check runs when--check-rateis set. Allows some checks to gather more meaningful metric samples. - Docker disk IO metrics are now tagged by
device - Introduces an expvar reporting the number of dogstatsd packets per second processed if dogstatsd_stats_enable is enabled.
- Add an Endpoints section in the GUI status page and the CLI status command, listing all endpoints used by the agent and their api keys.
- Expose number of packets received for each dogstatsd listeners through expvar
- Better descriptions of the
installandfreezesubcommands of thedatadog-agent integrationcommand. - In the flare, try to redact api keys from other services.
- Support the
siteconfig option in the log agent. - Add ability for Python checks to submit trace logs.
datadog/dogstatsd image: gohai metadata collection is now disabled by default
If dogstatsd_stats_enable is indeed enabled, we should consume and report on the generated stats. Fixes stagnant channel and misleading debug statement.
Fix a hostname resolution bug on EC2 hosts which could happen when the
hostname_fqdnoption was enabled, and which made the Agent use a non-unique FQDN instead of the EC2 instance ID.Fix a bug with parsing of
trace.ignorein theimportcommand.Fixes bug in windows core checks where adding/removing devices isn't caught, so only devices present on startup are monitored.
Fix bug of the
datadog-agent integration installcommand that prevented moving configuration files when theconf.dfolder is a mounted directory.The
datadog-agent integration installcommand creates the configuration folder for an integration with the correct permissions so that the configuration files can be copied.On windows, fixes downgrades. Fix won't be apparent for an additional release, since the core fix occurs on install.
On Windows, further fixes when installation drive isn't c:. Fixes problem where logs was effectively hardcoded to use c: for programdata Fixes installation problem where process & trace service were using c:programdata... to find datadog.yaml regardless of installation dir
If upgrading from a prior version, the configuration file (datadog.yaml) may have incorrect data. It will be necessary to manually update those entries. For example confd_path: c:programdatadatadogconf.d will have to be changed to confd_path: d:programdatadatadogconf.d etc.
Removed the command arguments from the flare's container list to avoid collecting sensitive information
Fix a rare crash caused by a nil map dereference in the
gohailibraryReintroducing JMXFetch process lifecycle management on Linux. Adding JMXFetch healthcheck for docker environments.
Fix warning about unknown setting "StartLimitIntervalSecs" in the agent service file with systemd version <=229.
- The
datadog-agent integrationcommand is now GA. - On the packaged Linux Agent, the python interpreter is now built with the
-fPICflag. - JMXFetch upgraded to 0.24.1. See https://github.com/DataDog/jmxfetch/releases/tag/0.24.0 and https://github.com/DataDog/jmxfetch/releases/tag/0.24.1
- Log host metadata at debug level regardless of its size.
Release on: 2018-12-27
- Fix a bug that could send the last log multiple times when a container was not writing new logs
- Fix a panic that could occur if a container doesn't send logs for more than 30 sec and the timestamp of the last received log is corrupted
Release on: 2018-12-19
- Fix a panic that could occur when a container stopped while the agent was reading logs from it.
This is a container only release that fixes a bug introduced in 6.8.0 that was impacting the kubernetes integration.
Release on: 2018-12-17
- Please refer to the 6.8.1 tag on integrations-core for the list of changes on the Core Checks.
- Fixes the default
kubeletcheck configuration that was preventing the kubernetes integration from working properly
Please note that a critical bug has been identified in this release that would prevent the kubernetes integration from collecting kubelet metrics on containerized agents.
The severity of the issue has led us to remove the 6.8.0 images on dockerhub and to make the latest tag point to the 6.7.0 release.
If you have upgraded to this version of the containerized agent we recommend you downgrade to 6.7.0. Linux packages are not affected.
Release on: 2018-12-13
- Please refer to the 6.8.0 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.8.0 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.8.0 tag on process-agent for the list of changes on the Process Agent.
The Datadog Agent now automatically look for the container short image name to set the default value for the log source and service. The source is especially important as it triggers the automatic configuration of your platform with integration pipeline and facets. The Datadog Agent autodiscovery can still be used to override the default source and service with pod annotations or container labels.
- The agent now requires a cluster agent version 1.0+ to establish a valid connection
- JMX garbage collection metrics
jvm.gc.cms.countandjvm.gc.parnew.timewere renamed tojvm.gc.minor_collection_count,jvm.gc.major_collection_count,jvm.gc.minor_collection_time,jvm.gc.major_collection_timein 6.6 to be more meaningful. To ensure backward compatibility the change was reverted in this release and the new names put behind a config option. If you started relying on these new names please enable thenew_gc_metricsoption in your jmx configurations. An example can be found here. This flag will be enabled by default in a future major release.
- Enable docker config provider if docker.sock exists
- The new command
datadog-agent configprints the runtime config of the agent. - Adds eBPF-based network collection component called network-tracer.
- Add diagnosis to the agent for connectivity to the cluster agent
datadog-agent integration installcommand prevents a user from downgrading an integration to a version older than the one shipped by default in the agent.- Adding kerberos support with libkrb5.
datadog-agent integration installcommand moves configuration files present in thedatadirectory of the wheel upon successful installation
- Adding a default location on Windows for the file storing pointers to make sure we never lose nor duplicate any logs
- Add an option to the agent check command to run the check n times
- Set service and source to the docker short image name when container_collect_all flag is enabled and no label or annotation is defined
- Docker: the datadog/dogstatsd image now ships a healthcheck
- Improved consistency of the ECS and Fargate tagging
- Improve logging when python checks use invalid types for tags
- Added a
regiontag to Fargate containers, indicating the AWS region they run in - Adds system.cpu.interrupt, and system.mem.committed, system.mem.paged, system.mem.nonpaged, system.mem.cached metrics on Windows
- Add
permissions.logfile to the flare archive. - Add an agent go-routine dump to the flare as reported by the built-in pprof runtime profiling interface.
- The agent can now expose its healthcheck on a dedicated http port. The Kubernetes daemonset uses this by defaut, on port 5555.
- It's possible now to have different poll intervals for each autodiscovery configuration providers
- Improve Windows Event parsing. Event.EventData.Data fields are parsed as one JSON object. Event.EventData.Binary field is parsed to its string value
- Rename the Windows Event "#text" field to "value". This fixes the facet creation of those fields
- Add a
status.logand aconfig-check.logwith a basic message in the flare if the agent is not running or is unreachable. - Added support for wildcards to DD_KUBERNETES_POD_LABELS_AS_TAGS. For example, DD_KUBERNETES_POD_LABELS_AS_TAGS='{"*":"kube_%%label%%"}' will all pod labels as tags to your metrics with tags names prefixed by kube_.
- Removed support for logs_config.tcp_forward_port as it's no longer needed for other integrations.
- Configure error log when failing to run docker inspect to read as debug instead, as this log is duplicated by the tagger.
- Fix a bug where datadog-agent integration users could not test the --in-toto flag due to a filesystem permission issue.
- The cluster agent client init now fails as expected if the cluster agent URL is not valid
- Print correct error when the
datadog-agent integrationcommand fails after installing an integration - Fix build failure on 32bit armv7
- Fix a bug with Docker logging driver where logs would not be tailed after a log rotation when the option --log-opt max-file=1 was set.
- Display the correct timezone name in the status page.
- On Windows, the agent now properly computes the location of ProgramData for configuration files instead of using hardcoded values
- JMXFetch upgraded to 0.23.0. See https://github.com/DataDog/jmxfetch/releases/tag/0.23.0
- On linux, use the cgo dns resolver instead of the golang one. The will make the agent use glibc to resolve hostnames and should give more predictable results.
- Starting with this Agent release, all the Datadog integrations that are installed
with the
datadog-agent integration installcommand are reset to their default versions when the Agent is upgraded. This guarantees the integrity of the embedded python environment after the upgrade. - The
datadog-agent integrationcommand is now in Beta.
Release on: 2018-11-21
This release only ships changes to the trace-agent.
This release focuses on simplifying Trace Search configuration. APM Events can now be configured at the tracer level. Tracers will get updated in the near future to expose this option.
- Please refer to the 6.7.0 tag on trace-agent for the list of changes on the Trace Agent.
Release on: 2018-10-25
- Please refer to the 6.6.0 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.6.0 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.6.0 tag on process-agent for the list of changes on the Process Agent.
- JMX garbage collection metrics jvm.gc.parnew.time and jvm.gc.cms.count got renamed to jvm.gc.minor_collection_time and jvm.gc.major_collection_count on some JMX integrations. Since this change on the name of these 2 metrics may affect your dashboards and monitors, these metrics will also be sent under their older names in a later version of the Agent.
- Disk check support for the puppy agent on unix-like systems
- Support for the upcoming cluster-agent cluster-level checks feature,
via the
clusterchecksconfig provider - Add a new CRI core check that will send metrics about resource usage of your containers via the Container Runtime Interface.
- Support SysVinit on Debian note: some warnings can appear if you enable/disable the agent manually on a systemd system. They can be safely ignored
- The
datadog-agent integration installcommand will now check for compatibility withdatadog-checks-baseshipped with the agent. In case of mismatch, it will try to rollback to the previously installed integration version and exit with a failure. - Add
--in-totoflag todatadog-agent integrationcommand to enable in-toto - Add
--verboseflag todatadog-agent integrationcommand to enable verbose logging on pip and TUF - Docker image: running with a read-only root filesystem is now supported
- Add a setting to configure the interval at which configs should be polled for autodiscovery.
- Support a new config option,
site, that allows setting the Datadog site to which the Agent should send data.dd_urlis still supported and, when set, overridessite. - Display a warning in the agent status when too many logs are being tailed and the agent is not tailing them all. This happens with wildcards in path of the tailed files
- Dogstatsd supports removing the hostname on events and services checks as it did with metrics, by adding an empty
host:tag - Added new dogstatsd_tags variable which can be used to specify additional tags to append to all metrics received by dogstatsd.
- dogstatsd cleans up stale UNIX socket on startup.
- The ecs-agent's docker container name can now be set via the
ecs_agent_container_nameoption or theDD_ECS_AGENT_CONTAINER_NAMEenvvar for autodetection. - EKS pause containers are ignored by default
- All python and go checks support the new
empty_default_hostnameoption to send metrics with no hostname. This is used for cluster-level checks - All go checks now support the
min_collection_intervaloption, as python check already do - Added a
kubelet_wait_on_missing_containeroption to handle hosts where the kubelet's podlist is slow to update, leading to missing tags or failing Autodiscovery. Set it to 1 for a 1 second maximum wait - Add an option to enable protobuf communication with the Kubernetes apiserver
datadog-agent integrationcommand will not pull any of the integration's dependencies- More accurate tag extraction logic for Docker Swarm
- Added new command line properties to the Windows installer which allow for setting site specific configuration.
- Fix an issue preventing the exit logs of the agent from displaying the correct filename.
- Fix bug that occurs when checks labels/annotation are misconfigured and would prevent the logs of the container to be tailed
- Fix an issue causing the agent to stop when systemd-journald service is stopped or fails
- Fix deadlock when an config item under
logsis invalid - Fix system.mem.pct_usable implementation on Linux 3.14+ to match Datadog Agent 5
- Fix a potential race in the autodiscovery where a service would be removed before its config could be resolved (causing the agent to crash)
- Fixes crash on Windows when the agent encounters a malformed performance counter database
- Fixes config.Digest that was not stable depending on the oder of tags in the instance. It also did not take into account LogsConfig, this is fixed as well.
- Fix an issue where the log agent would prevent files from being log rotated on Windows
- Correctly pass the agent's proxy settings to pip when using the
datadog-agent integrationcommand with TUF enabled. - Recover from errors when connection to the docker socket is lost to continue tailing containers.
- When installing / updating wheels using the
datadog-agent integrationcommand, we replace the PyPI index with our own by default, in order to prevent accidental installation of Datadog or even third-party packages from PyPI. - Remove some undocumented power user options to the
datadog-agent integrationcommand to prevent accidental misconfiguration that may reduce security guarantees.
- JMXFetch upgraded to 0.21.0; Adds support for rmi registry connection over SSL and client authentication.
- Use autodiscovery in log-agent kubernetes integration
Release on: 2018-09-20
- Please refer to the 6.5.2 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.5.2 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.5.2 tag on process-agent for the list of changes on the Process Agent.
- Fix a crash in the logs package that could occur when a docker tailer initialization failed.
Release on: 2018-09-17
- Please refer to the 6.5.1 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.5.1 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.5.1 tag on process-agent for the list of changes on the Process Agent.
- Fix possible deadlocks that could occur when new docker sources
and services are pushed and:
- The docker socket is closed at agent setup
- The docker socket is not mounted
- The kubernetes integration is enabled
- Fix a deadlock that could occur when the logs-agent is enabled and the configuration parameter 'logs_config.container_collect_all' or the environment variable 'DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL' are set to true.
Released on: 2018-09-13
Please note that a critical bug identified in this release affecting container
log collection when the container_collect_all was set, would lead to an agent
deadlock. The severity of the issue has led us to remove the packages for the
affected platforms (Linux and Docker). If you have upgraded to this version,
on Linux or Docker we recommend you downgrade to 6.4.2.
- Please refer to the 6.5.0 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.5.0 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.5.0 tag on process-agent for the list of changes on the Process Agent.
- Autodiscovery: the
dockerandkubeletlisteners will retry on error, to support starting the agent before your container runtime (host install) - Bump the default number of check runners to 4. This has some concurrency implications as we will now run multiple checks in parallel.
- Kubernetes: to avoid hostname collisions between clusters, a new
cluster_nameoption is available. It will be added as a suffix to the host alias detected from the kubelet in order to make these aliases unique across different clusters. - Docker image: handle docker/kubernetes secret files with a helper script
- The Node Agent can rely on the Datadog Cluster Agent to collect Node Labels.
- Improved ECS fargate tagging:
- Honor the
docker_labels_as_tagsoption to extract custom tags - Make the
cluster_nametag shorter - Add the
short_imageandcontainer_idtags - Remove some noisy tags
- Fix a lifecycle issue that caused missing tags
- Honor the
- The live containers view can now retrieve containers directly from the kubelet, in order to support containerd and crio
- Kubernetes events: setting event host tags to the related hosts, instead of the host collecting the events.
- Added dedicated configuration parameters to send logs to a proxy by TCP. Note that 'logs_config.dd_url', 'logs_config.dd_port' and 'logs_config.dev_mode_no_ssl' are deprecated and will be unvailable soon, use the new parameters 'logs_config.logs_dd_url' and 'logs_config.logs_no_ssl' instead.
- Added the possibility to send logs to Datadog using the port 443.
- Add more environment variables to the flare whitelist
- When
dd_urlis set toapp.datadoghq.eu, the infra Agent also sends data to versioned endpoints (similar toapp.datadoghq.com) - Make all numbers on the status page more human readable (using unit and SI prefix when appropriate)
- Display hostname provider and errors on the status page
- Kubelet Autodiscovery: reduce logging when no change is detected
- On Windows, the hostname_fqdn flag will now be honored, and the host reported by Datadog will be the fully qualified hostname.
- Enable all configuration options to be set with env vars
- Tags generated from GCE metadata may now be omitted by using
collect_gce_tagsconfiguration option. - Introduction of a new bucketed scheduler to enable multiple check workers to increase concurrency while spreading the load over the collection interval.
- The 'status' command and 'status' page (in the GUI) now displays errors raised by the '__init__' method of a Python check.
- Exclude the rancher pause container in the agent
- On status page, allow users to know which instance of a check matches which yaml instance in configcheck page
- The file_handle check reports 4 new metrics for feature parity with agent 5
- The ntp check will now query multiple servers by default to be more
resilient to servers returning wrong offsets. A now config option
hostsis now available in the ntp check configuration file to allow users to change the list of ntp servers. - Tags and sources in the tagger-list command are now sorted to ease troubleshooting.
- To allow concurrent execution of subprocess calls from python, we now save the thread state and release the GIL to unblock the interpreter . We can reaquire the GIL and restore the thread state when the subprocess call returns.
- Add a new configuration option, named tag_value_split_separator, allowing the specified list of raw tags to have its value split by a given separator. Only applies to host tags, tags coming from container integrations. Does not apply to tags on dogstatsd metrics, and tags collected by other integrations.
Autodiscovery now enforces the ac_exclude and ac_include filtering options for all listeners. Please double-check your exclusion patterns before upgrading and add inclusion patterns if some autodiscovered containers match these.
The introduction of multiple runners for checks implies check instances may now run concurrently. This should help the agent make better use of resources, in particular it will help prevent or reduce the side-effects of slow checks delaying the execution of all other checks.
The change will affect custom checks not enforcing thread safety as they may, depending on the schedule, access unsynchronized structures concurrently with the corresponding data race ensuing. If you wish to run checks in a fully sequential fashion, you may set the check_runners option in your datadog.yaml config or via the DD_CHECK_RUNNERS to 1. Also, please feel free to reach out to us if you need more information or help with the new multiple runner/concurrency model.
For more details please read the technical note in the datadog.yaml.
Prometheus custom checks are now limited to 2000 metrics by default to provide users control over the maximum number of custom metrics sent in the case of configuration errors or input changes. This limit can be changed with the
max_returned_metricsoption in the check configuration.
- All Autodiscovery listeners now enforce the ac_exclude and ac_include filtering options, as described in the documentation.
- Fixed "logs_config.frame_size" override that would not be taken into account.
- collect io metrics for drives with path only (like: C:C0) on Windows
- Fix API_KEY validation for 'additional_endpoints' by using their respective endpoint instead of the main one all the time.
- Fix port ordering for the %%port_%% Autodiscovery tag on the docker listener
- Fix missing ECS tags under some conditions
- Change the name of the agent expvar from
aggregator/ServiceCheckFlushed)toaggregator/ServiceCheckFlushed - Fix an issue where logs wouldn't be ingested if the API key contains a trailing new line
- Setting the log level of the
checksubcommand using the-lflag was not setting the log level of python integrations. - Display embedded Python version in the status page instead of the version from the system Python.
- Fixes a bug causing kube_service tags to be missing when kubernetes_map_services_on_ip is false.
- The ntp check now handles negative offsets if the host time is in the future.
- Fix a possible index out of range panic in Dogstatsd origin detection
- Fix a verbose debug log caused by rescheduling services with no checks associated with them.
- JMXFetch upgraded to 0.20.2; ships updated FasterXML.
- Remove noisy and useless debug log line from contextResolver
Release on: 2018-08-13
- Please refer to the 6.4.2 tag on integrations-core for the list of changes on the Core Checks.
- The flare command does not collect the agent container's environment variables anymore
- Fixes an issue with docker tailing on restart of monitored containers. Previously, at each container restart the agent would re submit all logs. Now, on restart we use tracked offsets properly, and as a result submit only new logs
Release on: 2018-08-01
- Please refer to the 6.4.1 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.4.1 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.4.1 tag on process-agent for the list of changes on the Process Agent.
- Create packaging for google cloud launcher integration.
- Add options to exclude specific payloads from being sent to Datadog. In some environments, some of the gathered information is considered too sensitive to be sent to Datadog (i.e. IP addresses in events or service checks). This feature adds to option to exclude specific payload types from being sent to the backend.
- Collect container disk metrics less often in the docker check, decreasing its effect on performance when enabled.
- Autodiscovery now supports the %%hostname%% tag on the docker listener This tag will resolve to the containers' hostname value if present in the container inspect. It is useful if the container IP is not available or erroneous.
- Dogstatsd origin detection now supports container tagging for Kubernetes clusters running containerd or cri-o, in addition to the existing docker support
- This release ships full support of Kubernetes 1.3+
- OpenShift ClusterResourceQuotas metrics are now collected by the kube_apiserver check, under the openshift.clusterquota.* and openshift.appliedclusterquota.* names.
- Display the version for Python checks on the status page.
- Adding DD_EXPVAR_PORT to the configuration environment variables.
- On Windows, Specifically log to both the log file and the event viewer what initiated an agent shutdown. Also logs specific startup errors to both the log file and event viewer.
- The embedded Python has been bumped from 2.7.14 to 2.7.15
- Agent expvar metrics now have default values. Metrics like the number of
packets dropped by the agent or errors were previously not reported until a
first event occurred. This should make it easier to use the expvar
configuration
agent_stats.yaml. - Proxy settings can be configured through the environment variables
DD_PROXY_HTTP,DD_PROXY_HTTPSandDD_PROXY_NO_PROXY. These environment variables take precedence over theproxyoptions configured indatadog.yaml, and behave exactly the same way as these options. The standardHTTP_PROXY,HTTPS_PROXYandNO_PROXYare still honored but have known side effects on integrations, for simplicity we recommended using the new environment variables. For more information, please refer to our proxy docs - Update to distribution metrics algorithm with improved accuracy
- Added ECS pause containers to the default docker exclusion list
- Adding logging for when the agent fails to detect the origin of a packet in dogstatsd socket mode because of namespace issues.
- The
skip_ssl_validationconfiguration option can now be set through the relatedDD_SKIP_SSL_VALIDATIONenv var - The Agent will log failed healthchecks on query and during exit
- On Windows, provides installation parameter to set the cmd_port, the port on which the agent command interface runs. To be used if the default (5001) is already used by another program.
- The kube_service tag is now collected on Kubernetes 1.3.x versions. The matching uses a new logic. If it were to fail, reverting to the previous logic is possible by setting the kubernetes_map_services_on_ip option to true.
- The Kubernetes event collection timeout is now configurable
- Logs Agent: Added SOCKS5 proxy support. Use
logs_config: socks5_proxy_address: fqdn.example.com:portto set the proxy. - The diagnose output is now sorted by the diagnosis name
- Adding the status of the DCA (If enabled) in the Agent status command.
- If the environment variables that can be used to configure a proxy (
DD_PROXY_HTTP,DD_PROXY_HTTPS,DD_PROXY_NO_PROXY,HTTP_PROXY,HTTPS_PROXYandNO_PROXY) are present with an empty value (e.g.HTTP_PROXY=""), the Agent now uses this empty value instead of ignoring it and using lower-precedence options.
- Begin deprecating "Agent start" command. It is being replaced by "run". The "start" command will continue to function, with a deprecation notice
- 'app_key' value from the configuration is now redacted when creating a flare with the agent.
- Fixes presence of invalid UTF-8 characters when docker log message is greater than 16Kb
- Fix a possible agent crash due to a race condition in the auto discovery.
- Fixed an issue with jmxfetch not being killed on agent exit.
- Errors logged before the agent initialized the log module are now printed on STDERR instead of being silenced.
- Detect and handle Docker messages without header.
- Fixes installation, packaging scripts for OpenSUSE LEAP and greater.
- In the event of being unable to lock the dd-agent user (eg. dd-agent is an LDAP user) during installation, do not fail; print relevant warning.
- The leader election process is now restarted if the leader stops leading.
- Avoid Linux package installation failures when both the
initctlandsystemctlcommands are present but upstart is used as the init system
- The system information collected from gohai no longer includes network information when the agent is running in a container since the network information is for the the container and not the host itself.
- The ntp check now runs every 15 minutes by default to avoid over-loading the NTP server pools
- Added new command "run" to the agent. This command replaces the "start" command, to reduce ambiguity with the service lifecycle commands
Release on: 2018-07-17
- Please refer to the 6.3.3 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.3.3 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.3.3 tag on process-agent for the list of changes on the Process Agent.
- Add 'system.mem.buffered' metric on linux system.
- Fix the IO check behavior on unix based on 'iostat' tool:
- Most metrics are an average time, so we don't need to divide again by 'delta' (ex: number of read/time doing read operations)
- time is based on the millisecond and not the second
- Kubernetes API Server's polling frequency is now customisable.
- Use as expected the configuration value of kubernetes_metadata_tag_update_freq, introduce a kubernetes_apiserver_client_timeout configuration option.
- Fix a bug that led the agent to panic in some cases if
the
log_levelconfiguration option was set toerror.
Released on: 2018-07-05
- Please refer to the 6.3.2 tag on integrations-core for the list of changes on the Core Checks.
- The service mapper now groups the mappings of pods to services by namespace. This prevents kube_service tags from being erroneously applied to metrics for a pod not targeted by a service but has the same name as a pod in a different namespace targeted by that service.
- Fix a bug in dogstatsd metrics parsing where the Agent would leave the host tag empty instead of applying its hostname on metrics with a tag metadata field but no tags (i.e. the tags field is only one # character). Regression introduced in 6.3.0
- Replace invalid utf-8 characters by the standard replacement char.
Release on: 2018-06-27
- Please refer to the 6.3.1 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.3.1 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.3.1 tag on process-agent for the list of changes on the Process Agent.
- JMXFetch upgraded to 0.20.1; ships tagging bugfixes.
- Fixes panic when the agent receives an unsupported pattern in a log processing rule
- Fixes problem in 6.3.0 in which agent wouldn't start on Windows Server 2008r2.
- Provide the actual JMX check name as check_name in configurations provided to JMXFetch via the agent API. This addresses a regression in 6.3.0 that broke the instance: tag. Due to the nature of the regression, and the fix, this will cause churn on the tag potentially affecting dashboards and monitors.
Release on: 2018-06-20
- Please refer to the 6.3.0 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.3.0 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.3.0 tag on process-agent for the list of changes on the Process Agent.
- Add docker memory soft limit metric.
- Added a host tag for docker swarm node role.
- The import command now support multiple dd_url and API keys.
- Add an option to set the read buffer size for dogstatsd socket on POSIX system (SO_RCVBUF).
- Add support for port names in template vars for autodiscovery.
- Add a new "tagger-list" command that outputs the tagger content of a running agent.
- Adding Azure pause containers to the default image exclusion list
- Add flag histogram_copy_to_distribution to send histogram metric values as distributions automatically. Note that the distributions feature is in beta. An additional flag histogram_copy_to_distribution_prefix modifies the existing histogram metric name by adding a prefix, e.g. dist., to better distinguish between these values.
- Add docker & swarm information to host metadata
- "[BETA] Encrypted passwords in configurations can now be fetched from a secrets manager."
- Add docker ps -a output to the flare.
- Introduces a new redacting writer that will make sure anything written into the flare is scrubbed from credentials and sensitive information.
- The agent now supports setting/overriding proxy URLs through environment variables (HTTP_PROXY, HTTPS_PROXY and NO_PROXY).
- Created a new journald integration to collect logs from systemd. It's only available on debian distributions for now.
- Add kubelet version to container metadata.
- Adds support for windows event logs collection
- Allow overriding procfs path. Should allow to collect relevant host metrics in containerized environments. The override will affect python checks and will result in psutil using the overriding path.
- The fowarder will now spaw specific workers per domain to avoid slow down when one domain is down.
- ALPHA - Adding new tooling to securely upgrade integration packages/wheels from our private TUF repository. Please note any third party dependencies will still be downloaded from PyPI with no additional security validation.
- If your Agent is configured to use a web proxy through the
proxyconfig option or one of the*_PROXYenvironment variables, and the configured proxy URL starts with thehttps://scheme, the Agent will now attempt to connect to your proxy using HTTPS, whereas it would previously connect to your proxy using HTTP. If you have a working proxy configuration, please make sure your proxy URL(s) start with thehttp://scheme before upgrading to v6.3+. This has no impact on the security of the data sent to Datadog, since the payloads are always secured with HTTPS between your Agents and Datadog whateverproxyconfiguration you may use. - Docker image: we moved the default configuration from the docker image's default environment variables to the datadog-*.yaml files. This allows users to easily mount a custom datadog.yaml configuration file to set all options. If you already did so, you will need to update your datadog.yaml to include these new defaults. If you only used envvars, no change is needed.
- The agent now supports the environment variables "HTTP_PROXY", "HTTPS_PROXY" and "NO_PROXY". If set these variables will override the setting in datadog.yaml.
- Moves away from the community library for the kubernetes client in favor of the official one.
- The core Agent check Python code is no longer duplicated here and is instead pulled from integrations-core. The code now resides in the datadog_checks namespace, though the old checks, utils, etc. paths are still supported. Please update your custom checks accordingly. For more information, see https://github.com/DataDog/datadog-agent/blob/master/docs/agent/changes.md#python-modules
- Default config agent_stats.yaml used to collect go_expvar metrics from the Agent has been updated.
- Take into account empty hosts on metrics coming from dogstatsd, instead of ignoring them and applying the Agent's hostname.
- Decrease epsilon and increase incoming buffer size for improved accuracy of distribution metrics.
- Better handling of docker return values to avoid errors
- Fix log format when no log file is specified which cause the log date to not be correctly displayed.
- Configurations of unscheduled checks are now properly removed from the configcheck command display.
- The agent would send the source twice when protobuf enabled (default), once in the source field and once in tags. As a result, we would see the source twice in the app. This PR fixes it, by sending it only in the source field.
- Fix a bug on windows where the io check was reporting metrics for the
C:drive only. - Multiple config files can now be used for the same JMX based integration
- The auto-discovery mechanism can now properly discover multiple configs for one JMX based integration
- The JMXFetch process is now managed properly when JMXFetch configs are unscheduled through auto-discovery
- Fix a possible panic in the kubernetes event watcher.
- Fix panics within the agent when using non thread safe method from Viper library (Unmarshall).
- On RHEL/SUSE, stop the Agent properly in the pre-install RPM script on systems where
/libis not a symlink to/usr/lib. - To match the behavior of Agent 5, a flag has been introduced to make the
agent use
hostname -fon unix-based systems before tryingos.Hostname(). This flag is turned off by default for 6.3 and will be enabled by default in 6.4. The import command used to upgrade from the Agent5 to the Agent6 will enable this flag in the config. - Align docker agent's kubernetes liveness probe timeout with docker healthcheck (5s) to avoid too many container restarts.
- Fix kube_service tagging of kubernetes network metrics
- Fixed parsing issue with logs processing rules in autodiscovery.
- Prevent logs agent from submitting protocol buffer payloads with invalid UTF-8.
- Fixes JMXFetch on Windows when the
custom_jar_pathsand/ortools_jar_pathoptions are set, by using a semicolon as the path separator on Windows. - Prevent an empty response body from being marked as a "successful call to the GCE metadata api". Fixes a bug where hostnames became an empty string when using docker swarm and a non GCE environment.
- Config option specified in syslog_pem if syslog logging is enabled with TLS should be a path to the certificate, not a textual certificate in the configuration.
- Changes the hostname used for Docker events to be the hostname of the agent.
- Removes use of gopsutil on Windows. Gopsutil relies heavily on WMI; because the go runtime doesn't lock goroutines to system threads, the COM layer can have difficulties initializing. Solves the problem where metadata and various system checks can't initialize properly
- The agent is now compiled with Go 1.10.2
- The datadog/agent docker image now runs two collector runners by default
- The DEB and RPM packages now create the
dd-agentuser with no login shell (/sbin/nologinor/usr/sbin/nologin). The packages do not modify the login shell of thedd-agentuser if it already exists. - The scripts of the Linux packages now don't exit with errors when no supported init system is detected, and only print warnings instead
- On the status and check command outputs, rename checks'
MetricstoMetric Samplesto reflect that the number represents the number of samples submitted by the check, not the number of metrics after aggregation. - Scrub all logging output from credentials. Should prevent leakage of credentials in logs from 3rd-party code or code added in the future.
2018-05-23
- Please refer to the 6.2.1 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.2.1 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.2.1 tag on process-agent for the list of changes on the Process Agent.
- If the kubelet is not configured with TLS auth, the agent will fail to communicate with the API when it should still try HTTP.
- Fix collection of host tags pulled from GCP project (
project:andnumeric_project_id:tags) and GCP instance attributes. - A bug was preventing some jmx configuration options to be set from the jmx checks configs.
- The RPM packages now write systemd service files to /usr/lib/systemd/system/ (recommended path on RHEL/SUSE) instead of /lib/systemd/system/
2018-05-11
- Please refer to the 6.2.0 tag on integrations-core for the list of changes on the Core Checks.
- Please refer to the 6.2.0 tag on trace-agent for the list of changes on the Trace Agent.
- Please refer to the 6.2.0 tag on process-agent for the list of changes on the Process Agent.
- Introduce new docker cpu shares gauge.
- Add ability to configure the namespace in which the resources related to the kubernetes check are created.
- The kubelet check now honors container filtering options
- Adding Datadog Cluster Agent client in Node Agent. Adding support for TLS in the Datadog Cluster Agent API.
- Docker: set a default 5 seconds timeout on all docker requests to mitigate possible docker daemon freezes
- Connection to the ECS agent should be more resilient
- Add agent5-like JMXFetch helper commands to help with JMXFetch troubleshooting.
- The agent has been tested on Kubernetes 1.4 & OpenShift 3.4. Refer to https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/agent/README.md for installation instructions
- Extract creator tags from kubernetes legacy created-by annotation if the new ownerReferences field is not found
- The agent import command now handles converting options from the legacy kubernetes.yaml file, for agents running on the host
- The memory corecheck sends 2 new metrics on Linux:
system.mem.commit_limitandsystem.mem.committed_as - Added the possibility to filter docker containers by name for log collection.
- Added a support for docker labels to enrich logs metadata.
- Logs Agent: add a filename tag to messages with the name of the file being tailed.
- Shipping protobuf C++ implementation for the protobuf package, this should help us be more performant when parsing larger/binary protobuf messages in relevant integrations.
- Enable to set collect_ec2_tags from environment variable DD_COLLECT_EC2_TAGS
- The configcheck command now display checks in alphabetical orders and are no longer grouped by configuration provider
- Add average check run time to
datadog-agent statusand to the GUI. - Consider every configuration having autodiscovery identifier a template
- Implement a circuit breaker and use jittered, truncated exponential backoff for network error retries.
- Change logs agent configuration to use protocol buffers encoding and endpoint by default.
- Kubernetes 1.3 & OpenShift 3.3 are currently not fully supported: docker and kubelet integrations work OK, but apiserver communication (event collection, kube_service tagging) is not implemented
- Removing python PDH code bundled with the agent in favor of code already included in the integrations-core` repository and bundled with datadog_checks_base wheel. This provides a single source of truth for the python PDH logic.
- Fix a possible race condition in AutoDiscovery where configuration is identical on container churn and considered as duplicate before being de-scheduled.
- It is now possible to save logs only configuration in the GUI without getting an error message.
- Docker network metrics are now tagged by interface name as a fallback if a docker network name cannot be determined (affects some Swarm stack deployments)
- Dogstatsd now support listening on an IPv6 address when using
bind_hostconfig option. - The agent now fetches a hostname alias from kubernetes when possible. It fixes some duplicated host issues that could happen when metrics were using kubernetes host names, as the kubernetes_state integration
- Fix case issues in tag extraction for docker/kubernetes container tags and kubernetes host tags
- Fixes initialization of performance counter (Windows) to be able to better cope with missing counter strings, and non-english locales
- Bind the kubelet_tls_verify as an environment variable.
- Docker image: fix entrypoint bug causing the kubernetes_apiserver check to not be enabled
- Fixed an issue with collecting logs bigger than 4096 chars on windows.
- Fixes a misleading log line on windows for logs file tailing
- Fixed a concurrent issue in the logs auditor causing the agent to crash.
- Fix an issue for docker image name filtering when images contain a tag.
- On Windows, changes the configuration for Process Agent and Trace Agent services to be manual-start. There is no impact if the services are configured to be active; however, if they're disabled, will stop the behavior where they're briefly started then stopped, which creates excessive Windows service alert.
- API key validation logic was ignoring proxy settings, leading to situations where the agent reported that it was "Unable to validate API key" in the GUI.
- Fix EC2 tags collection when multiple marketplaces are set.
- Fixes collection of host tags from GCE metadata
- Fix Go checks errors not being displayed in the status page.
- Sanitize logged Datadog URLs when proxies are configured.
- Fix a race condition in the kubernetes service tagging logic
- Fix a possible panic when docker cannot inspect a container
- In the metrics aggregator, log readable context information (metric name, host, tags) instead of the raw context key to help troubleshooting
- Remove executable permission bits from systemd/upstart/launchd service definition files.
- Improved the flare credential removing logic to work in a few edge cases that where not accounted for previously.
- Make file tailing a little less verbose. We avoid logging at every iteration the different issues we encountered, instead we log them at first run only. The status command shows the up-to-date information, and can be used at anytime to troubleshoot such issues
- Adds collection of PDH counter information to the flare; saves the step of always asking the customer for this information.
- Improve logging for the metamap, avoid spammy error when no cluster level metadata is found.
2018-04-19
Our development staff observed that a local, unprivileged user had the ability to make an HTTP request to the /agent/check-config endpoint on the agent process that listens on localhost. This request would result in the local-users' ability to read Agent integration configurations. This issue was patched by enforcing authentication via a session token. Please upgrade your agent accordingly.
- The
/agent/check-configendpoint has been patched to enforce authentication of the caller via a bearer session token.
2018-04-16
- This release also includes changes to the trace agent. See 6.1.3 tag on trace-agent
- Fix a bug where the docker_network tag incorrectly appeared on non-network docker metrics and autodiscovery tags
- Fix the use of "docker restart" with the agent image
2018-04-05
- Fix some edge cases where flare could contain secrets if the secrets where encapsulated in quotes.
2018-03-29
- Fix a crash in the docker check when collecting sizes on an image with no repository tags.
- Fixes bug on Windows where, if configuration options are specified on the installation command line, invalid proxy options are set.
- Removed the read timeout for UDP connections causing the agent to stop forwarding logs after one minute of nonactivity.
- Updating the data type of the CPU of the task and the metadata name for Version to Revision.
- Add environment variable DD_ENABLE_GOHAI for setting option enable_gohai when running in a container.
2018-03-23
- Add Agent Version to flare form
- Add the DD_CHECK_RUNNERS environment variable binding
- Add the status command to the DCA.
- Docker check: ignore the new exec_die event type by default
- Extract the swarm_namespace tag for docker swarm containers, in addition to the already present swarm_service tag.
- Allow configuration of the enabled-state of process, logs, and apm to be specified on the installation command line for Windows.
- Add a jmx_use_cgroup_memory_limit option to set jmxfetch to use cgroup memory limits when calculating its heap size. It is enabled by default in the docker image.
- Add option to extract kubernetes pod annotations as tags, similar to labels
- Added an environment variable DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL to enable logs tailing on all containers.
- Adding the 'bind_host' option to configure the interface to bind by dogstatsd and JMX.
- Support setting tags as a YAML array in the logs agent integration configuration
- Fix docker memory metrics parsing from cgroup files
- Fix docker.mem.in_use metric computation
- When using the import script, change the group owner of configuration files to the dd-agent user.
- Fix a false positive in the collector-queue healthcheck
- The old docker_daemon check is now properly converted in the "import" command by default
- Docker check: fix event filtering for exec events
- Improve docker monitoring when the system is under a very high load. The agent might still temporarily miss a healthcheck, but will be able to run already scheduled checks, and recover once the spike ends
- Fixes the container startup on Fargate, where we tried and remove the same file twice, failing hard (stopping) on the second attempt.
- Fix flare failing on zipping individual components
- Fixed an issue where the import script would put an empty histogram aggregates and percentiles in datadog.yaml if they didn't exist in datadog.conf.
- Fix the build for platforms not supporting Gohai.
- Fixes flaw where Windows Performance counters were not properly initialized on non EN-US versions of windows
- Menu in system tray reports wrong version (6.0.0) for all versions of Agent. This fixes the system tray menu to report the correct version.
- Fixing clear passwords in "config-check.log" when sending a flare.
- Allow network proxy settings set on the Windows installation command line to be set in the registry, where they'll be translated to the configuration
- Accept now short names for docker image in logs configuration file and added to the possibility to filter containers by image name with Kubernetes.
- Fixes an issue that would prevent the agent from stopping when it was tailing logs of a container that had no logs.
- fixes an issue with wildcard tailing of logs files on windows
- Allow Linux package uninstallation to proceed without errors even on platforms that aren't supported by the Agent
- Fixes agent to run on Server "Core" versions
- Changes default precision of pdh-based counters from int to float. Fixes bug where fidelity of some counters is quite low, especially counters with values between 0 and 1.
- Adds back the removed system.mem.usable metric for Agents running on Windows.
- Avoid multiple initializations of the tagger subsystem
- Normalize support of nested config options defined with env vars.
- Make the check-rate command more visible when running "check` to get a list of metrics.
2018-03-12
- This release also includes bugfixes to the process agent. See diff.
- Fixed the issue preventing from having docker tags when collecting logs from containers.
- Fix docker metrics collection on Moby Linux hosts (default Swarm AMI)
2018-03-07
- Packaging issue in 6.0.1 resulted in the release of nightly builds for trace-agent and process-agent. 6.0.2 ships the stable intended versions.
2018-03-07
- Add information about Log Agent checks to the GUI General Status page.
- Run the service mapper on all the agents running the apiserver check. Exit before running the rest of the check if the agent is not the leader.
- Fixing docker network metrics collection for the docker check and the process agent on some network configurations.
- Replaces the system.mem.free metric with gopsutil's 'available' and splits the windows and linux memory checks. Previously this reported with a value of 0 and system.mem.used was reporting the same as system.mem.total
- ".pdh" suffix was added to system.io metrics on windows for side-by-side testing when changed the collection mechanism, and inadvertently left.
- Fix bug where global tags for PDH based python checks are not read correctly from the configuration yaml.
- IE does not support String.prototype.endsWith, add implementation to the string prototype to enable the functionality.
- remove .pdh suffix from system.io.wkb_s, system.io_w_s, system.io.rkb_s, system.io.r_s, system.io.avg_q_sz
- Fix GUI for JMX checks, they are now manageable from the web UI.
- Fix the launch of JMXFetch on windows and make multiplatform treatment of the launch more robust.
2018-02-27
- Fixes bug in agent hostname command, whereby the configuration library wasn't initialized. This caused agent hostname to use the default computed hostname, rather than the entry in the configuration file
2018-02-23
- Change the kubernetes leader election system to use configmaps instead of endpoints. This allows a simpler migration from Agent5, as Agent6 will not require additional permissions.
- Adds in the proc.queue_length and proc.count metrics with the windows version of the Agent.
- Process agent service should pass the configuration file argument to the executable when launching - otherwise service will always come up on reboots.
- Add the windows icon to the Infrastructure List for Agents installed on Windows machines.
- Fix Docker container
--pid=hostoperations. Previous RCs can cause host system instabilities and should not be run in pid host mode. - Windows: set correct default value for apm config to enabled, so that the trace agent is started by default
- Removes deprecated process_agent_enabled flag
- metrics.yaml is not a "configurable" file - it provides default metrics for checks and shouldn't be altered. Removed from the GUI configuration file list.
- Windows: gopsutil calls to the CPU module require COM threading model to be in multi-threaded mode, to guarantee it's safe to make those calls we load the python checks setting the right COM concurrency mode first. Once loaded we clear the concurrency mode and python checks that might use COM will set it as they need.
- Windows: make stop/restart of DatadogAgent service stop/restart dependent services accordingly
- Windows: Prevent system tray icon from being displayed more than once
- Windows: Make default start behavior of process-agent consistent with Linux Agent
- Windows: Fix the item launching the web-based GUI in the systray icon menu
- Windows: Process agent service now passes the configuration file argument to the executable when launching - otherwise service will always come up on reboots.
- Windows: Added developer documentation regarding the caveats of the COM concurrency model and its implications moving forward. The current state affects auto-discovery and dynamic scheduling of checks.
2018-02-22
- Adds windows systray icon. System tray icon includes menu options for starting/stopping/restarting services, creating a flare, and launching the browser-based GUI.
- allow auth token path to be set in the config file
- Implementation for disabling checks from the web UI
- Agent restart message on UI, clears after restart.
- Add SSL support & label joins for the prometheus check
- Fix the command-line flag parsing regression caused by a transitive dependency importing the
glog library.
agentflags should now behave as in beta9. - GUI had broken after the introduction of integrations as wheels this PR ensures we collect the full list of available integrations so we can enable the corresponding configurations from the UI.
- Fix an issue preventing logs-agent to tail container logs when docker API version is prior to 1.25
- Fix line miss issue that could happen when tailing new files found when scanning
- On windows
Automaticservices would fail to start across reboots due to a known go issue on 1.9.2: golang/go#23479 We now start windows services as delayed start automatic services (ie. they now start automatically after all other automatic services).
- The OSX build of the agent does not include the containers integrations as they are only supported on Linux for now. The Windows build already excluded them since beta1
- The
auth_tokenfile, used to store the api authentication token, is now only readable/writable by the user running the agent instead of inheriting datadog.yaml permissions.
2018-02-20
- Add namespace configuration for metric names for dogstatsd
- Rework autodiscovery label names to be consistent, still support the previous names
- Ships updated integrations from integrations-core 6.0.0-rc.2, including new
kubeletcheck - Add envvar bindings for docker/kubernetes custom tag extraction features
- Normal installations: APM now listens to localhost only by default, you need to set apm_config.apm_non_local_traffic = true to enable listening on the network
- Docker image: APM is now disabled by default, you need to set DD_APM_ENABLED=true to run the trace agent. It listens on all interfaces by default when running, you can set DD_APM_NON_LOCAL_TRAFFIC=false to only listen on localhost
- Don't try to match containers by image name if they provide an AD template via docker labels or pod annotations. This avoid scheduling double checks.
- Fix handling of the %%host%% and %%port%% autodiscovery tags
- The aggregator now discards metric samples with
NaNvalues. Also solves a serializing error on metric payloads. - Fixes bug whereby device tag was (correctly) removed from tags list, but device field was only added to the metric on the first run.
- Fix an issue unscheduling checks discovered through auto-discovery
- Upstart would indefinitely respawn trace and process agents even when exiting with a zero status code. We now explicitly define exit code 0 as a valid exit code to prevent respawn when the agents are disabled.
- Fix cases where empty host tags in the Agent
datadog.yamlconfiguration caused the host metadata payload parsing to fail in the backend. - Fix
resourcesmetadata collector so that its payload is correctly parsed in the backend even when empty. - Make sure we don't get stuck if the API server does not return events.
- make tagger more resilient to malformed docker events
- Removing vsphere and sqlserver from the blacklist. The former is available on all platforms, sqlserver is currently windows-only.
- The apm.yaml.default config file was removed on linux and the trace-agent.conf.example was removed on every other platform.
- Only enable the
resourcesmetadata collector on Linux by default, to match Agent 5's behavior.
2018-02-16
The execution of the main agent, trace-agent (APM), and process-agent processes is now orchestrated using systemd/upstart facilities on Linux.
On Linux and macOS, the trace-agent and process-agent now read their configuration from the main
datadog.yaml file (located, by default on Linux, at /etc/datadog-agent/datadog.yaml).
Changes implementation of IOStats in Windows to use the Performance Data Helper API, rather than WMI.
- Introducing the Datadog Process Agent for Windows
- Make the trace-agent read its configuration from datadog.yaml.
- Add a HTTP header containing the agent version to every transaction sent by the agent.
- Add a configcheck command to the agent API and CLI, it prints current loaded & resolved configurations for each provider. The output of the command is also added to a new config-check.log into the flare.
- Add option to output logs in JSON format
- Add a pod entity to the tagger.
- Introducing the service mapper strategy for the DCA. We periodically hit the API server, to get the list of pods, nodes and services. Then we proceed to match which endpoint (i.e. pod) is covered by which service. We create a map of pod name to service names, cache it and expose a public method. This method is called when the Service Mapper endpoint of the DCA API is hit. We also query the cache instead of the API Server if a cache miss happens, to separate the concerns.
- The kubelet request /pods is now cached to avoid stressing the kubelet API and for better performances
- Create a cluster agent util to query the Datadog Cluster Agent (DCA) API.
- Convert the format of histogram_aggregate when importing agent5 configs to agent6
- Add support for histogram_percentile when importing config options from agent5 to agent6
- Adds in a datadog.agent.running metric that showcases a value of 1 if the Agent is currently reporting to Datadog.
- Use the S6 light init in the docker image to start the process-agent and the trace-agent. This allows to remove the process orchestration logic from the infra-agent
- Add short_image tag to docker tagger collector
- Try to connect to Docker even /var/run/docker.sock is absent, to honor the $DOCKER_HOST environment variable.
- Get the docker server version in the debug logs for troubleshooting
- Add ECS & ECS Fargate metadata API connectivity diagnose
- Add optional forwarding of dogstatsd packets to another UDP server.
- Don't block the aggregator if the forwarder input queue is full (we now drop transaction).
- Add healthcheck to the forwarder. The agent will be unhealthy if the apikey is invalid or if transactions are dropped because all workers are busy.
- Make the number of workers used by the forwarder configurable and set default to 1.
- The agent has internal healthchecks on all subsystems. The result is exposed via the agent health command and used in the docker image as a healthcheck. The /probe.sh wrapper is provided for compatibility with agent5 and future-proofing.
- Adds the ability to import the Trace configuration options from datadog.conf to their respective datadog.yaml fields with the import command.
- Add jmx_custom_jars option to make sure they are loaded by jmxfetch
- Added ability to retrieve the hostname from the Kubernetes kubelet API.
- Support the new kubelet integration by providing the kubelet url and credentials to it
- Add source_component and namespace tags to Kubernetes events.
- Support the Kubernetes service name tags on containers. The tag used is kube_service. The default listener for kubernetes is now the kubelet instead of docker.
- Implementation of the Leader Election for the node agent.
- Add more details to the information displayed by the logs agent status.
- Log lines starting with the special character < were considered rfc5424 formatted, and any further formatting was skipped. This commit updates the detection rule to match logs starting with `<pri>version `, to reduce false positives
- Support tagging on Nomad 0.6.0+ clusters
- adds a core check and the core logic to query the kubernetes API server and format the events into Datadog events. Also adds the Control Plane status check.
- The process-agent uses the datadog.yaml file for activation and additional config.
- Support the new kubelet integration by providing the container tags to it
- Set configurable timeout for IPC api server
- Add support for collect_ec2_tags configuration option.
- Support the legacy default_integration_http_timeout configuration option.
- handle the "warning" value to log_level, translating it to "warn"
- Use systemd/upstart facilities on linux to orchestrate the agents execution. Rate limit process restarts such that 5 failures in a 10 second span will result in no further restart attempts.
- Increased the number of versions of Docker API that logs-agent support from 1.25 to 1.18
- If you run a Nomad agent older than 0.6.0, the nomad_group tag will be absent until you upgrade your orchestrator.
- Changed the attribute name to enable log collection from YAML configuration file from "log_enabled" to "logs_enabled", "log_enabled" is still supported.
- Properly listen for events emitted on OSes like CentOS 6 so the Agent starts on reboot
- Relieves CPU consumption of the WMI service by using PDH rather than WMI
- Updating custom error used to insure the collection of the token in the configmap datadogtoken.
- Fix docker event reconnection logic to gracefully wait if docker daemon is unresponsive
- The checks packaged with the new wheels method are in the default python
package site, already included in
sys.path. We therefore removed this path them from the locations that are appended to the default pythonsys.path - Strip hostname if running inside ECS Fargate and disable core checks (not relevant without host infos) Fix hostname caching consistency
- The agent/dogstatsd docker image now ships the appropriate binary
- Fix the uploaded file name of the flare archive.
- Fixed the structure of the Flare archive on all platforms.
- Fix the import proxy build conversion
- Fix the agent stop command
- Fixes a bug that caused the GUI to create a flare without a status file
- Fix DataDog#1159 where error was not explicit when the check had invalid configuration or code
- Fix to evaluate whether the DCA can query resources (events, services, nodes, pods) before running core component or checks. Logic allows for the DCA to run components independently if they are configured and we can query the associated resources.
- Only load and log the kube_apiserver check if KUBERNETES=yes is used.
- The collector will no longer block when the number of runners is lower than the number of long running check. We now start a new runner for those checks.
- Modify JMXFetch jar permissions to allow the agent to read it on osx.
- Deleted linux-specific configuration files on macosx to avoid polluting the logs and the web ui.
- Fix dogstatsd unix socket rights so every user could write to it.
- The
resourcesmetadata provider is now enabled by default in order to populate the "process treemap" visualization continuously (on host dashboards) - Decrease verbosity of
urllib3's logger (used by python checks through therequestsmodule) - Document the exclusion of dockercloud containers.
- The flare now includes a dump of whitelisted environment variables that can have an impact on agent behaviour. If no whitelisted envvar is found, the envvars.log file is not created in the flare.
- Upgraded Go runtime to 1.9.4 on Linux builds
- The linux packages now own the custom check directory (
/etc/datadog-agent/checks.d/) and the log directory (/var/log/datadog/) - Add an automated install script for osx.
2018-01-26
In this release, the order of precedence of the Custom Checks has changed. This may affect your custom checks. Please refer to the Upgrade Notes section for more details.
This release includes support for Datadog Logs for Windows.
- In this release, the Datadog Log feature is supported on all supported versions of windows.
- Support AD on Rancher 1.x by getting the container port through the image's exposed ports as a fallback mechanism
- APM / Process / Log agents can be enabled/disabled via the DD_*_ENABLED environment variables, see the agent docker image readme for details
- Add a dd-agent user in the docker image to prepare for running root-less
- Config parsing errors are now displayed in the output of the 'status' command and on the web ui.
- The DD_TAGS environment variable allows to set host tags, in addition to the tags option in datadog.yaml
- Added a section to the agent status to report live information about logs-agent
- Set the default "procfs_path" configuration to /host/proc when containerized to allow the network check to collect the host's network metrics. This can be overridden with the the DD_PROCFS_PATH envvar.
- Series for a common metric name will no longer be split among multiple transactions/payload. This guarantee that every point for a time T and a metric M will be bundled together when push to the backend. This allows some optimization on the backend side.
- Add a service listener for ECS Fargate, and a config provider. Also add the concept of ECSContainer and make it compatible with Docker containers so that the process agent can handle them.
- Having a separate type of container for ECS is not ideal, we will need to change how we represent containers to make it more generic. We also need to improve hostname handling, this will come in a follow up PR.
- Custom checks (located by default on Linux in
/etc/datadog-agent/checks.d/) now have a lower precedence than the checks that are bundled with the Agent. This means that a custom check with the same name as a bundled check will now be ignored, and the bundled check will be loaded instead. If you want to override a bundled check with a custom check, please use a new name for your custom check, and use that new name for the related yaml configuration file. - Tags in the DD_TAGS environment variable are now separated by spaces instead of commas in agent5
- Added a support for multi-line tailing with docker
- Fix the extraction of the environment variables in the pkg/tagger/collectors/docker_extract.go when the variable is like "KEY="
- Fix a nil-pointer segfault in docker event processing when an event is ignored
- Make yum revalidate the cache before installing the rpm package when using the install script.
- Added some missing spaces in logs
- Fixed build pipeline commenting flaky tests from logs tailer
- Fixed a bug on file tailing causing the logs-agent to reprocess multiple times the same data when restarted because of a wrong file offset management when lines are trimmed.
- Be more lenient when filtering containers using labels.
- Fixes an issue when pulling tags from a yaml config file with any integration that uses the PDHBaseCheck class
- Updated the shipped CA certs to latest
- For CircleCI use builder images in the datadog dockerhub repo.
2018-01-11
- Logs-agent now runs as a goroutine in the main agent process
- All docker event subscribers are multiplexed in one connection, reduces stress on the docker daemon
- The Agent can find relevant listeners on its host by using the "auto" listener, for now only docker is supported
- The Docker label AD provider now watches container events and only updates when containers start/die to save resources
- Add a new option, force_tls_12, to the agent configuration to force the TLS version to 1.2 when contactin Datatog.
- Reno and releasenotes are now mandatory. A test will fail if no releasenotes where added/updated to the PR. A 'noreno' label can be added to the PR to skip this test.
- [logs] Fix an issue when the hostname was not provided in datadog.yaml: the logs-agent logic uses the same hostname as the main agent
- [logs] Trim spaces from single lines
- Fix missing fields in forwarder logging entries
- Fix RancherOS cgroup mountpoint detection
- [linux packaging] Fix missing dd-agent script after upgrade, the fix will take effect on a fresh install of '>= beta.8' or upgrade from '>= beta.8'
- [logs] Do not send empty logs with multilines
- [flare] Fix command on Windows by fixing path of collected log files
- Fix path of logs collected by flare on Windows, was breaking flare command
- Remove resversion handling from podwatcher, as it's unused
- Refactor corecheck boilerplate in CheckBase
- [flare] Rename config file dumped from memory