fill/find behavior around ERRORing epoch-task reports

[placer14]

There is currently an implicit behavior in find to omit reports which have resulted in an ERROR status. The rationale being that these will likely always produce the same ERRORing result and we should save lily from performing the same work again with no benefit. However, I’ve noticed that there are processing records with ERRORs which also show as FILLED in the gap reports. This suggests that some errors could be transient and the premise that performing the same work again will result in the same outcome are not true.

Further, ERROR reports are effectively gaps in dataset and find should not treat these reports any differently from other types of gaps. If the cause of an ERROR is eventually resolved, using find to identify that epoch-task as a candidate to later fill is required to fix the bug in our data (and not just in the software).

[frrist]

Gap fill uses the indexer to fill gaps. When the indexer encounters an error (example) it persists it to the visor processing reports table and continues rather than returning an error. (which fill checks)

Gap Find doesn't treat errors as gaps for the reasons you've stated. An unstated expectation of errors in the processing reports table was that they would be fixed and filled in manually by a developer. The pattern for doing this doesn't exist, gap find/fill could be used to achieve it, and what that looks like could be decided here.

[placer14]

IMO, let's just leverage our existing fill/find to do that work for us. Can't think of any reason why we might want a different tool for this job. And if you agree, then find will probably need to mark ERRORs as GAPs for fill to do it's work.

I don't think there's any design work or discussion needed with the above approach, it keeps concerns tidy, and the potential cost is some occasional extra work (from lily) where fill could (re)produce an error again.

The thing to note here is the transient nature of these errors. Results will vary from one execution to the next.

[placer14]

#779 is the implementation of the above approach if we want to accept that.