TipSetIndexer task result persistence race condition

[frrist]

Describe the bug:

The TipSetIndexer's TipSet method may return before all task outputs have been persisted to storage. Its usage in WalkChain can lead to Walker's Run method returning before persistence has completed and can therefore cause Visor to exit before all extracted data has been persisted.

go routine responsible for persisting task outputs: https://github.com/filecoin-project/sentinel-visor/blob/dcc3c5769986613c465a67e9d54b0d11a559ceda/chain/indexer.go#L266-L293

I think the fix could be as simple as moving the WaitGroup instantiation calls and wg.Wait() call outside of the go routine -- see 9fbd5a9 for details

Steps to Reproduce:

I ran into this while in my vector bench marking branch frrist/vector-builder.
To Repo using the aforementioned branch:

1. ensure you have a lotus datastore at ~/.lotus
2. run visor --log-level=info vector build --from=446000 --to=446010 --tasks=blocks,messages --vector-file=./visor-vectors/vector.json
3. run ./visor --log-level=info vector execute --vector-file=./visor-vectors/vector.json
4. repeat step 3 until you get a panic... race conditions are fun

Visor Version: dcc3c57

[iand]

I don't think your suggested fix is correct. The TipSet method is designed to return before persistance is complete: the goal is to persist the results of an extraction while the next tipset extraction is executing. We then have 30 seconds to extract plus 30 seconds to persist (which is mostly IO or idle wait time). Your change would mean the extraction and persistance have to happen within the 30 second epoch of a single tipset.

t.persistSlot acts as a semaphore so that only one extraction + persistance pair is in flight at any one time.

Can you add the panic backtrace to the issue?

[frrist]

reopening as the current fix his a hack