Feature backoff and delay

[bskeggs]

This PR adds two optional arguments to the gcp_logging_collect recipe: backoff and delay. These are introduced to help manage GCP's logging api limits, which I have regularly exceeded when using dftimewolf.

Both are disabled by default, so should not impact existing dftimewolf workflows.

Backoff is a boolean value, which if True, will retry log collection if the GCP logging API quota is exceeded. Due to the way the logging API handles subsequent requests after exceeding an the API limit (#679), a new cloud logging client must be created for the re-try. This means the collection must start again. If no delay was configured, the collection will be retried with a delay of 1s between each request. If a delay was configured, the collection will be retried with a delay of 2x the set value.

Delay is the number of seconds to delay between each API request.

When dftimewolf if run without delay or backoff arguments, and the API limit is exceeded, the output will look like this:

poetry run dftimewolf gcp_logging_collect 'projectname' 'timestamp >= "2022-11-30T00:00:00Z" AND timestamp <= "2022-12-08T08:00:00Z"'
  [ dftimewolf       ] Debug log: /tmp/dftimewolf-run-20221202_081136_l_1y_kd9.log
  [ GCPLogsCollector ] backoff False
  [ GCPLogsCollector ] delay 0.0
  [ GCPLogsCollector ] Hit quota limit requesting GCP logs.
  [ GCPLogsCollector ] Exponential backoff was not enabled, so query has exited.
  [ GCPLogsCollector ] The collection is most likely incomplete.
  [ GCPLogsCollector ] Downloaded logs to /tmp/tmpjfwq0jhv.jsonl

When backoff and delay are specified, the output will look like this:

poetry run dftimewolf gcp_logging_collect 'projectname' 'timestamp >= "2022-11-30T00:00:00Z" AND timestamp <= "2022-12-08T08:00:00Z"' --backoff --delay 1
Messages
  [ dftimewolf       ] Debug log: /tmp/dftimewolf-run-20221202_081057_1ph_z9ws.log
  [ GCPLogsCollector ] backoff True
  [ GCPLogsCollector ] delay 1.0
  [ GCPLogsCollector ] Hit quota limit requesting GCP logs.
  [ GCPLogsCollector ] Retrying in 60 seconds with a slower query rate.
  [ GCPLogsCollector ] Due to the GCP logging API, the query must restart from the beginning
  [ GCPLogsCollector ] Setting up new logging client.
  [ GCPLogsCollector ] Restarting query with an API request rate of 1 per 2s
  [ GCPLogsCollector ] Downloaded logs to /tmp/tmpfr1nz782.jsonl

Please note that I had to break up the bulk of the code in the 'Process' function into distinct functions, however most of the code base is the same. I had to break it up because I needed a way to re-call the functions to generate a new logging client, and set a new output path, once the api limit was exceeded.

[ramo-j]

Thanks for the PR! Logic looks solid to me, just a few nits, and noting the failing automated checks.

Also noting that we don't have unit tests for this module, so raised #681.

[bskeggs]

Thanks for the review @ramo-j . I've addressed your suggestions and push the changes. Please let me know if there is anything else I can do regarding the automated checks.

[bskeggs]

Should be good to go on the linting tests now @ramo-j , thanks!

[ramo-j]

Here and in other recipes: Any arguments to be interpolated (that is, @arg_name) into a recipe need to be added into the "args" section. In this recipe, that would be filter_expression, backoff and delay.

Also, do we want to change up the filter expression here anyway? I didn't author the relevant module, so I'm not sure why the original filter_expression takes the form it does, but I assume there must have been a reason. How does changing this affect log collection? I notice data/recipes/gcp_logging_collect.json has the form you are changing to, so I assume it's probably fine, but we should make sure we're not breaking any expected behaviour.

[ramo-j]

As per my other comment, you'll just need to copy these additions into the other recipes you've changed.