Can't apply a monitor after setting a silence from the UI: is already unmuted

[iksaif]

Terraform Version

$ terraform -v
Terraform v0.11.10

Affected Resource(s)

• datadog_momnitor

Terraform Configuration Files

resource "datadog_monitor" "test" {
    name = "test"
    type = "metric alert"
    message = "test"

  query = "avg(last_5m):max:system.load.1{${var.common_filters},${var.datacenter_filters},node-role.kubernetes.io/logs-seed} by {host} > 100"

  thresholds {
      critical = 100
  }

  notify_no_data = false
}

Debug Output

N/A

Panic Output

N/A

Expected Behavior

When adding a manual downtime via the UI on one of the groups:

This happens:

Acquiring state lock. This may take a few moments...
datadog_monitor.logs_test: Refreshing state... (ID: xxxxx)
datadog_monitor.logs_test: Modifying... (ID: xxxxx)
  silenced.%:                        "1" => "0"
  silenced.host:i-xxxxxxxxxxxx: "0" => ""
Releasing state lock. This may take a few moments...

Error: Error applying plan:

1 error(s) occurred:

* datadog_monitor.logs_test: 1 error(s) occurred:

* datadog_monitor.logs_test: API error 400 Bad Request: {"errors":["metric alert \"[Logs] [us1.staging.dog] Logs test\" is already unmuted"]}

Actual Behavior

• My monitor gets updated
• My manually set silence does not get removed

Steps to Reproduce

1. terraform apply a grouped by monitor
2. Mute one of the groups
3. terraform apply again

Important Factoids

Reverting 17eea5f fixes the issue.

[bkabrda]

So the referenced commit fixed an issue where removing the silenced block from the monitor config, which would then not be removed from the actual monitor. In order to do that, it calls the /api/v1/monitor/<id>/unmute endpoint (without JSON payload). When the referenced commit is reverted, doing the second terraform apply doesn't actually unmute it - the silenced attribute will stay the same - which is exactly the issue that the commit fixed.

The issue with calling the unmute endpoint without a payload is that it tries to unmute the * scope, which in the reproducer above is already unmuted, so the API returns a correct response.

The proper solution is calling the unmute endpoint with {"all_scopes": true} as JSON payload. This will ensure that all muted scopes are removed and works even if * is not among them.

Fixing this is not hard, but it will require a fix in the go-datadog-api library first, since that doesn't support attaching JSON payloads to calls to the unmute endpoint.

[bkabrda]

So in addition to the above, we should:

• Document that the above solution would disable any created datadog_downtime that has monitor_id set [1].
• Document usage of lifecycle, more specifically ignore_changes which would avoid touching any changes done by hand or by adding a downtime with monitor_id.

[1] https://www.terraform.io/docs/providers/datadog/r/downtime.html#monitor_id