Increase resiliency of logging alerts

[robrap]

We had an issue where we were using the following for an event bus alert: SELECT * FROM Log WHERE message RLIKE r'Error producing event to event bus.*' LIMIT MAX, but the error message had presumably changed to: Error delivering message to Kafka event bus. This stopped alerts from firing.

1. Ensuring there is a custom attribute to look for would be more resilient to these types of changes.
2. However, logging alerts are preferred because a errors would not be missed due to transaction sampling.
3. Could a combo of alerts help? Maybe this would result in redundant alerts, or maybe the custom attribute alert would only fire if it detected more errors than the logs?

A/C:

• The previous event production alerting on New Relic which uses TransactionError should be queried off of an error.class that is custom for producing errors.
• A new alert on New Relic for consuming TransactionError should be off of a error.class that is custom for event consumption errors.
• Communication with the Open edX community about possible suggestions to match this pattern.

Implementation Details:

• Create a new ProducingException(or better name) exception class.
• Update record_producing_error to reraise the previous error as a ProducingException exception.
• poll_indefinitely has a call to record_exception that should be refactored to use record_producing_error instead.
• Create a new ConsumingException (or better name) exception class.
• Update record_event_consuming_error to reraise the previous error as a ConsumingException error.
• • Update the alert conditions on New Relic once these changes have been deployed.

[robrap]

Labeled with event-bus, but might affect other types of log alerting if we have any.

[dianakhuang]

I decided against changing the exception handling in poll_indefinitely, because it (a) doesn't have the same context information that the other call has and would therefore be confusing, and (b) since it doesn't reraise the exception, it will still send across the original exception information, which is what we want.

[robrap]

We did complete this ticket as written (I think), but it turns out that we are still using the logging alerts, and those alerts should be made more resilient by using a tag or something other than an error message to be more resilient to change. We may want to create a new ticket for this.

Note: I'm also closing this ticket, which was marked as Done, but not closed.