feat(tf): add log metric and alert for detecting signup throttling

[m90]

Ticket: https://phabricator.wikimedia.org/T335963

This PR just adds the changes to the module, which will then need to be consumed in a second PR.

[tarrow]

Suggested change

	alignment_period = "300s"
	alignment_period = "88200s" # 24.5 hours

Probably want to make this similar to the SQL alerting failure alignment period or this will auto resolve after 5mins (unless people keep trying to make accounts) and we'll probably ignore it

[m90]

Probably want to make this similar to the SQL alerting failure

SQL alerting is where I copied the value from

wbaas-deploy/tf/modules/monitoring/replication-lag-alert.tf

Line 31 in 0a5b6f6

alignment_period = "300s"

I also think a shorter value might be better as it will bring us closer to telling us how many users are experiencing throttling, instead of telling us "it's broken right now, not sure how many people are affected".

That being said, I gotta admit I wonder if those alerts are the right primitive to use for such a thing to begin with. Is there anything in GCP monitoring that is designed for being notified of point-in-time events that cannot be "resolved" in an automatically measurable fashion? I took a brief look, but couldn't find anything.

[m90]

I think what we want is something that works in the way "Sentry" would work, which seems to be close to this https://cloud.google.com/error-reporting/docs/

However, it inverses control and the application would need to push the error to the service. Not sure right now.

[m90]

It looks as if we already have Error Reporting enabled via Stack Driver, we just never use it for anything application level: https://console.cloud.google.com/errors?referrer=search&project=wikibase-cloud

[m90]

Unfortunately it seems terraform doesn't support creating notifications from the Error Reporting service yet hashicorp/terraform-provider-google#12068

[tarrow]

SQL alerting is where I copied the value from

Ah, I was unclear. I was thinking about

wbaas-deploy/tf/modules/monitoring/sql-backup-failure-alert.tf

Line 26 in 0a5b6f6

duration = "88200s"

which is another (and functionally different) sql alert.

I think we should try using this error reporting thing; I agree that log based metrics exceeding a count doesn't feel like the right concept for this type of alerting (if it happens even once we would like to know).

[m90]

So how would we do this? Close this PR, update api to report and see what it brings us?

[tarrow]

I think that makes sense; let's try to not totally lose these commits though

[m90]

PR in api wbstack/api#620