Document how to create DD monitor like New Relic's "alert if X is over Y for entire time period"

[robrap]

A/C:

• Options documented (include the following to the extent it is known: pros/cons, appropriateness for manual vs. Terraform config, etc.)
  • Documenting at https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/1581023295/Options+for+Datadog+time-period+APM+monitors
  • Link to from other alerting doc, unless this goes directly in the how-to: https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/1008500757/How+to+configure+edX+services+for+Datadog#Alert-Policies
• Communicate updated doc as-appropriate.

Proposal is to create a doc that shares what we know, and link to it from our how-to docs.

Datadog Support ticket asking for ideas, as well as to submit a feature request: https://help.datadoghq.com/hc/en-us/requests/1994936 (private link)

[robrap]

[inform] Copying a Slack request to @julianajlk:

Could you:

• Add me to your support ticket, and
• Have them convert it to a feature request (if you haven't already)?
  • We should note that what was super simple in NR has taken lots of discussion and no agreement on the best way to handle this situation.

[timmc-edx]

Ideas posted in Slack:

Troy:

maybe try smoothing the metric first by applying a rolling average window of 5 minutes, then calculate the min() of that over 1 hour?

if the threshold is crossed, then that means the rolling average 5-minute window was consistently crossing the threshold for the entire hour

only tricky thing about my approach is that the moving average functions only operate on # of intervals, so if you use a metric with a 10 second interval, 20 invervals (via ewma_20) would be about 3.3 minutes.

Ray:

I think if you clampMax(), you can trigger if the average over that time is that max value.

Because average can't go above that threshold value, you ensure you hit that threshold for the entire period if average is at the threshold

Troy notes that this requires coordinating the same number in two places. (Might be OK for Terraform-managed metrics.)

Juliana got some advice from DD support:

this case a rollup might be the best solution. Rollups should usually be avoided in monitor queries, because of the possibility of misalignment between the rollup interval and the evaluation window of the monitor. The start and end of rollup intervals are aligned to UNIX time, not to the start and end of monitor queries. Therefore, a monitor may evaluate (and trigger on) an incomplete rollup interval containing only a small sample of data. To avoid this issue, delay the evaluation of your monitor by (at least) the length of the setup rollup interval. See Rollups in monitors
For your monitors to evaluate as expected, you need to add the following configurations:

• 15minutes (900 seconds) avg rollup function
• Evaluation window of 1 hour
• Monitor evaluation delay of 900 seconds

For example. It's currently 9:30 am, the monitor will evaluate from 8:15 to 9:15, it'll evaluate the average of four 15 minutes segments. The average of those 4 segments, needs to be below threshold for the monitor to trigger.

[robrap]

[inform] @julianajlk also initiated a feature request with DD on https://help.datadoghq.com/hc/en-us/requests/1754847 (which you probably won't be able to see).

[timmc-edx]

Datadog Support notes that Metrics monitors support evaluation over an entire time period. So this capability may be present for some monitor types and not others.

[julianajlk]

Linking here some Datadog metrics monitor docs, the chart of "Definitions", highlighting the below -

"any single value" in max aggregation method vs. "all points in the evaluation window" in min method, specifically "For monitors that alert when below the threshold, the max and min behavior is reversed."

[robrap]

@timmc-edx:

1. I'm moving this to "Ready for Development" to get it on our board, and I'm assigning it to you, because I think you have the most experience, and would be able to complete this the fastest.
2. I adjusted the AC to have this more about moving what we know today to a doc, so we can close out this ticket.

[timmc-edx]

I created a wiki page with the results of my experiments: https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/1581023295/Options+for+Datadog+time-period+APM+monitors

Unfortunately, none of the suggestions seem to work for APM monitors.

[timmc-edx]

Blocked until after winter break; revisit Jan 6 by opening a ticket with DD to ask for a workable option.

[robrap]

We weren't able to find a solution for sustained alerts on APM data.

• One idea is creating a metric from the APM data (https://app.datadoghq.com/apm/traces/generate-metrics) and then using the metric-based solution for sustained alerts.
• Additionally, we should open a feature request around supporting sustained alert capabilities for APM data.

[timmc-edx]

I've filed a Datadog Support ticket asking for ideas, as well as to submit a feature request: https://help.datadoghq.com/hc/en-us/requests/1994936

[robrap]

Reminder: @timmc-edx will fold this ticket into the following issue(s), and to pre-groom them so anyone (including Ray) could pick them up.

• Recreate sustained alerts for Arch-BOM in Datadog #830
• Adjust apdex monitors after NR is turned off #673

@timmc-edx: Since there will be a relationship between docs, and experimentation, and closing all the tickets, I propose you consolidate down to a single ticket and close this one and one of the others as duplicates. Note: You can now "Close as duplicate" in Github.

[timmc-edx]

We've documented some options, but experimenting with them and communicating the results will now be part of #830 -- A/C have been copied over and expanded.