Add ability to have monthly alerting history indices instead of daily.

[stafot]

Is your feature request related to a problem? Please describe.
Right now the opendistro alerting plugin which is embed on the AWS ES Service creates an .opendistro-alerting-alert-history-{DATE} index daily. Also uses the default number of shards which is not optimal for this use case.

Describe the solution you'd like
I want to be able to customise this functionality and be able to change the "rotation" to weekly or monthly and also be able to change the number of shards right now uses 5 shards for a very small index while in this case 1 shard (for my use case) would be more than enough. ref. https://aws.amazon.com/blogs/database/get-started-with-amazon-elasticsearch-service-how-many-shards-do-i-need/. My current daily indices are ~300kb.

Describe alternatives you've considered

Additional context

I am using the latest available version of AWS ES service (7.1.1)
Updated: Situation remains the same after upgrade to 7.4

[stafot]

Following up the issue I was able to change the number of shards in a newly created by using a template. Although would be nice to be able to customize upon setup of the plugin.
So the problem that remains actually is the ability to have weekly, monthly or any other rotation the user prefers.

FYI: the template that fixes the one portion of this feature request.

{
  "order": 0,
  "index_patterns": [
    ".opendistro-alerting-alert-history-*"
  ],
  "settings": {
    "index": {
      "number_of_shards": "1"
    }
  },
  "aliases": {}
}

[dbbaughe]

I believe we have a opendistro.alerting.alert_history_max_age setting. This would allow you to specifiy the age used for the rollover condition which currently defaults to 24 hours.

This might not be exposed to the customer on the AmazonES service.
@qreshi, could we verify if this is exposed on the service?

[stafot]

@dbbaughe Yes i can confirm this could be a reasonable workaround to mitigate my issue, so I will apply it if available. But still having the ability to preset different cycles than the current in the initial setup could be nice enhancement. I 'll let you know if your suggestion worked for and then you can keep or close my feature request at your will.

[dbbaughe]

Hey @stafot,

Can you clarify what you mean by preset different cycles?
Do you mean an option somewhere on the Kibana Alerting plugin to easily select?
Any "preset" that I can see us offering is just us internally changing these below settings on behalf of you.

The rollover settings we have exposed are:

opendistro.alerting.alert_history_max_age - default 24h
The max_age condition used in the rollover request

opendistro.alerting.alert_history_max_docs - default 1000
The max_docs condition used in the rollover request

opendistro.alerting.alert_history_rollover_period - default 12h
How often the rollover API is called with the above conditions.
If the above conditions are false then it won't rollover.

And in 7.4+ I believe we also have

opendistro.alerting.alert_history_retention_period - default 30d
Alert history indices over this time value will be deleted.

opendistro.alerting.alert_history_enabled - default true
Enable/Disable alert history in case you don't want it.

[stafot]

@dbbaughe By preset mean lets say a checkbox with options daily, weekly, monthly, yearly which automatically will set alert_history_max_age value accordingly and ideally makes the appropriate naming conventions for each option. So for example if you choose monthly retention the name of the history index will contain only month and year not actual date number. Make any sense?

[dbbaughe]

Hi @stafot,

Sure, how important is the name reflecting the daily, weekly, monthly, yearly option though?

The functionality of having the index rollover being daily, weekly, monthly, yearly would be simple to add with the name just being the full date of when it was "created".

Changing the name to reflect the chosen "preset" though complicates things when switching between presets. Going from smaller to larger to smaller etc. in terms of granularity causes weird behavior and probably unexpected behavior.

So to sum up I think you have two asks:

1. Have selectable presets on Kibana Alerting to allow to choose when Alerting history indices should be rolled over to make it easier (daily, weekly, monthly, yearly, etc.)
2. Change the the name of the Alerting history indices to reflect the chosen preset

Will #1 without #2 be confusing or would you find having #1 alone be fine for now?

[stafot]

@dbbaughe Both options are totally acceptable. 1 is fine. 2 is ideal. Tomorrow i will try your first suggestion and I will let you know if worked as expected.

[stafot]

@dbbaughe Unfortunately what you suggested is not allowed on AWS ES service.
This part of API is not exposed to customer.
Request:

PUT /_cluster/settings
{
    "persistent" : {
        "opendistro.alerting.alert_history_max_age" : "30d"
    }
}

Response:

{
  "Message": "Your request: '/_cluster/settings' payload is not allowed."
}

/cc @qreshi

I didn't find any alternative way to do it with index template. I assume I can't.

[dbbaughe]

Hey @stafot,

It looks like these settings are not exposed on the service.

We can expose these as the first step and look into making it easier with things like presets.

[stafot]

@dbbaughe It would be great.

[stafot]

Since we upgraded to AWS ES 7.7 I noticed that the behavior changed. Now the alerting history indices are not changing daily. Although I can't find which is the default behavior now. Is there a size threshold or something else? I would appreciate any info.
My assumption is that it will change on 1000 docs based on your previous comments. Is this assumption correct?
/cc @dbbaughe @qreshi

[qreshi]

Hi @stafot,

The alerting history settings are the same as the open source ones as of AWS ES 7.7.

Meaning that history indices are rolled over every 30 days or 1000 documents (whichever happens first) and are retained for at most 60 days.

I believe the 'Alerting Settings' section of the AWS docs mentions we use the default values as well. If you do see any discrepancy between the open source settings and the AWS ES ones, feel free to call it out so we can have the AWS documentation reflect it.

[stafot]

@qreshi Thank you very much, I ll monitor this and come back to you if any discrepancy get noticed.

[stafot]

Feel free to mark this as done, as the functionality I asked for, now exists. There could be some minor UX improvements as mentioned already but the core requirements of this issue fulfilled.