refactor: Replace "GlobalStatus" state management

[cyrossignol]

This removes the legacy GlobalStatus data structure used for polling for GUI information from core state and replaces it with the "push" pattern employed by other GUI features where we can avoid the locks. It fixes intermittent block height lag in the GUI during the initial sync.

As part of the change, I rewrote the basic miner status container into a formal API that manages thread safety internally. This also centralizes several pieces of duplicated logic into one location and enables localization for miner alert messages.

[jamescowens]

@cyrossignol your change to a flag treatment bears a lot of similarity to what I was thinking with the miner status in https://github.com/jamescowens/Gridcoin-Research/commits/optimize_global_status, but of course your PR goes much farther in straightening this out.

[jamescowens]

I am going to go through the locking on this very carefully to make sure we have actually solved the problem and not just pushed the problem around.

[jamescowens]

I like it. You moved my age-based caching scheme to the previously unused NotifyBlocks signal to coarsen the updates, and they are push style from the core rather than pull from the GUI so there is no lock contention. I think that does it. :)

[jamescowens]

My only major comment on this is that I think we are still going to get lock contention although less than before. We generate signals from the core on a change of state that passes the aging filter, which propagate via the Qt signal queue to the GUI and end up calling ClientModel::updateNumBlocks(), which is in a different (GUI thread) and then has to fight with the running core thread to get a lock on cs_main. I think it is better to put this in a core data class (containing num_blocks, difficulty, and netweight), do the filtering directly in that class being called from the GridcoinServices() function (which already posseses the lock on cs_main). The Core class for those should also have its own internal lock to protect the CACHED values during access of this data class by the GUI. So this core class is internally thread-safe via an internal lock for accessing the cached values, but requires an external cs_main lock (already taken) to have the cache values refreshed FROM the core (in the core thread context).

I also think GRC::GetEstimatedTimetoStake() should be moved back from ClientModel::updateMinerStatus(bool staking, double coin_weight) to the above mentioned core class. While GetEstimatedTimetoStake is certainly dependent on miner status and can be filtered by miner status changed, it is more dependent on core state (because of the change in difficulty).

My guess is that you should move the age filter to this core class to wrap the above functions.

The above would essentially elimate all lock contention, because the heavy functions that require a cs_main lock will be executed in a thread that already has that lock in place. The internal class thread safety lock to protect access of the cached values in from the GUI side based on signals will be cheap and nearly instantaneous.

(I know this basically means a return of sorts of the global state class, but this is a different form of it.)

What do you think?

I will fetch your branch and tinker with this a little...

[jamescowens]

search_timestamp (hehe... you changed the name...)

[jamescowens]

I will post a PR on your branch tomorrow with some draft thoughts on this.

[cyrossignol]

@jamescowens As you noted, in regard to this PR, the queries that the GUI needs to synchronize threads to fetch core state for are:

• Current difficulty
• Estimated network weight
• Estimated staking frequency
• Estimated chain length (peer block count)

Current difficulty is a trivial calculation--we can probably push it to the GUI along with the chain tip info without any significant overhead.

Since the next two are an interpretation of the recent chain history, and the last metric is a rather wild guess, I didn't really see a need to handle those perfectly for the overview fields. In the uncommon instances where an in-sync node misses the lock in the GUI thread, the smoothed values won't be significantly off, and the next block or miner pass will very likely resolve the discrepancy. The high rate of cs_main locking during the initial sync can degrade the accuracy of those values, but the numbers fluctuate so intensely while syncing that the information is practically meaningless anyway.

I'm reluctant to reintroduce core state to cache the approximate values. I don't see any harm if the GUI exhibits a slight lag for these for a few moments every now and then. In fact, I think that skipping a cycle is better when the node is busy. We avoid additional thread safety complexity, the overhead of the mutex is tiny compared to the scans that these routines perform, and updating the values will hold up the main thread no matter which thread these execute in.

[jamescowens]

Some counterpoints to your comments:

1. I am not talking about resurrecting the monstrosity that is core state. In essence we are talking about where the following fields need to be:
   mutable std::atomic<int64_t> m_last_update_ms;
   mutable std::atomic m_cached_num_blocks;
   mutable std::atomic m_cached_num_blocks_of_peers;
   mutable std::atomic<int64_t> m_cached_best_block_time;
   mutable std::atomic m_cached_difficulty;
   mutable std::atomic m_cached_net_weight;
   mutable std::atomic m_cached_etts_days;
2. In relation to the above, you have placed those in the clientmodel for use by the GUI, but this precludes updating those fields from the core directly, which in some instances is the correct thing to do... I will get to that in a moment.
3. The issue that I noted that was a performance issue was not just one missed cycle, but up to several minutes where there were no updates on the main screen during a sync. In the original code this is due to your introduction of the try lock at

void BitcoinGUI::updateGlobalStatus()
{
LogPrint(BCLog::MISC, "BitcoinGUI::updateGlobalStatus()");

// This is needed to prevent segfaulting due to early GUI initialization compared to core.
if (miner_first_pass_complete)
{
    TRY_LOCK(cs_main, locked_main);

    if (!locked_main) {
        return;
    }

to relieve the locking contention on the GUI versus the core. This isn't a problem when the wallet is caught up and the core state is only changing once every 90 seconds or so, but what we were seeing is that the try-locks were falling through on many cycles when the core was busy.
4. The above issue is not alleviated in the current design. Essentially the same problem occurs at

void ClientModel::updateNumBlocks(int height, int64_t best_time)
{
m_cached_num_blocks = height;
m_cached_best_block_time = best_time;

{
    TRY_LOCK(cs_main, lockMain);

    if (lockMain) {
        m_cached_num_blocks_of_peers = ::GetNumBlocksOfPeers();
        m_cached_difficulty = GRC::GetCurrentDifficulty();
        m_cached_net_weight = GRC::GetEstimatedNetworkWeight() / 80.0;

        emit difficultyChanged(getDifficulty());
    }
}

emit numBlocksChanged(getNumBlocks(), getNumBlocksOfPeers());

}

5. The values that are computed by these functions are not approximate in the sense that they do represent the correct values for the chain at the height that they are computed. I am not worried about a signal delay in slowly changing state when we are in sync. I am worried about when we are syncing and we are doing ~1000 blocks/sec.
6. People have come to expect that the staking fields are updated on a regular basis during sync (a few seconds between updates). You could argue we could solve this by doing the same thing for the staking section that we do for the researcher section... blank it out with a message that the "wallet is not in sync...", but I do not think that will be well received, and we can't do that for the blocks field anyway, because then there would be no visible indication of syncing progress.

[jamescowens]

As I said, I am posting a PR with a sketch of what I am talking about later today, and I would like you to look at it with an open mind to my points.

[jamescowens]

@cyrossignol I posted a PR to your branch with my thoughts. Maybe there is something of value in there. You had already forced pushed after I had based my branch, so it is based on the original PR. I didn't rebase it because all the changes I made are in the last commit.

[iFoggz]

utACK;

Excellent work. Spent some time looking through it and liking what I am seeing. Glad these areas of much needed improvements are being undertaken and look forward to testing stage.

[cyrossignol]

Thanks, @jamescowens. Sorry about the force-push...wasn't thinking. I'll take a look at your PR here in a bit!

...and we can't do that for the blocks field anyway, because then there would be no visible indication of syncing progress.

To clarify: as implemented in this PR, the blocks counter will always update (immediately for in-sync, or at each MODEL_UPDATE_DELAY interval during inital block download). Only the estimate fields defer updates when busy.

[jamescowens]

Yes. With my approach all four update, because they are driven from the call to...

// This is fine to call each invocation of GridcoinServices. It is internally caching, and will
// only actually update at MODEL_UPDATE_DELAY intervals.
g_blockchain_status.UpdateBlockChainStatus(pindexBest->nHeight, pindexBest->GetBlockTime());

in GridcoinServices().

My real point is summed up thusely... if you are going to convert this to a push style approach go all the way with it, and push while you still possess the cs_main lock. Your implementation is a partial push/partial pull approach. (The signal and the blockheight is pushed, but the others are pull...)

[jamescowens]

I think both implementations suffer from the trigger after finishing sync problem... When the core finishes sync, depending on exactly where the model update interval last fell, the last signal could be a number of blocks behind the actual core state at the end of sync. I saw 20 blocks behind in playing around with this. This will not true up until the next block is received and processed, because there is nothing from the core to trigger the state change until that happens. This means the out-of-sync indicator stays lit and the block counter is "stuck" slightly behind the head until that next block is received.

[jamescowens]

One other advantage of my approach is that the g_blockchain_status cached member fields could be used by rpc calls to lessen the load on the core. I haven't looked at that carefully yet.

[cyrossignol]

My real point is summed up thusely... if you are going to convert this to a push style approach go all the way with it, and push while you still possess the cs_main lock. Your implementation is a partial push/partial pull approach. (The signal and the blockheight is pushed, but the others are pull...)

@jamescowens I was balancing the pushes for cheap/trivial values with pulls for the expensive operations that the GUI could defer while a node is busy. In other words, I didn't want to run the updates for net weight, etts, etc., immediately if the core still has work to do.

When the core finishes sync, depending on exactly where the model update interval last fell, the last signal could be a number of blocks behind the actual core state at the end of sync. I saw 20 blocks behind in playing around with this. This will not true up until the next block is received and processed, because there is nothing from the core to trigger the state change until that happens. This means the out-of-sync indicator stays lit and the block counter is "stuck" slightly behind the head until that next block is received.

Interesting... I haven't seen this happen in my tests. In this PR, the GUI skips the condition for the model update interval as soon as the core reports that the node is in-sync, so every block is processed after that point (well before the sync finishes). Will take a closer look. I think we need to work a check for IsInitialBlockDownload() into cyrossignol#6 somewhere.

One other advantage of my approach is that the g_blockchain_status cached member fields could be used by rpc calls to lessen the load on the core.

I agree—the caches in your PR could speed up the performance of getmininginfo rather well. However, it might not be appropriate for an RPC call to return stale information cached below the model update interval threshold—an RPC result should always be an exact snapshot, no?

[jamescowens]

I think it depends on the rpc call and the nature of the info. If you take the position that these metrics are estimates and not exact, then you shouldn't care about 4 seconds for etts, for example. The two heaviest calls here are by far the GetEstimatedNetworkWeight (which is really a call to GetAverageDifficulty over a 40 block interval) and the GetEstimatedTimetoStake() (the heaviest in the whole group), and I think a 4 sec lag for these is acceptable in rpc, but it may not be worth the bother. Unless you are literally pelting the wallet with some sort of stress test (which I have done before), the juice is probably not worth the squeeze.

One last point on my suggestions. To me, conceptually the items
mutable std::atomic<int64_t> m_last_update_ms;
mutable std::atomic m_cached_num_blocks;
mutable std::atomic m_cached_num_blocks_of_peers;
mutable std::atomic<int64_t> m_cached_best_block_time;
mutable std::atomic m_cached_difficulty;
mutable std::atomic m_cached_net_weight;
mutable std::atomic m_cached_etts_days;
are core parameters. If you are going to the bother to store the miner parameters in g_miner_status as the miner does its work (in the miner thread), it seems to me these deserve to be in a small peer class to that in status.h/cpp and not relegated into the clientmodel of the GUI.

I haven't run a side by side of your latest versus this BTW. I need to do that tonight.

[jamescowens]

See my comments on my PR to your branch....cyrossignol#6

[jamescowens]

Looking at them side by side, with your latest version of this PR, I think it is close enough to stick with your design. Hmm.... Actually difficulty is not updating on yours. It is staying at 0. Let's see when it actually updates. Maybe it is a region on testnet where the difficulty is really below 0.000. Yep. that looks to be the case.

[cyrossignol]

I think it depends on the rpc call and the nature of the info. If you take the position that these metrics are estimates and not exact, then you shouldn't care about 4 seconds for etts, for example...

Agreed.

One last point on my suggestions. To me, conceptually the items[...]are core parameters. If you are going to the bother to store the miner parameters in g_miner_status as the miner does its work (in the miner thread), it seems to me these deserve to be in a small peer class to that in status.h/cpp and not relegated into the clientmodel of the GUI.

That's a fair argument—I agree with your point here too. Since the miner status is purely for presentation, I did peek from the opposite perspective while I worked on this to see whether the MinerStatus state could make more sense as a GUI implementation with direct APIs for RPC (it doesn't, of course).

Anyway, my observation is really just pedantic for the sake of discussing design philosophy. Realistically, the benefits of the miner status state far outweigh an academic separation of concerns. We may be able to justify the BlockChainStatus cache too if we can remove the overhead from the main thread needed to keep the cache fresh for nodes that will never use it. I'm not trying to shoot the idea down—it could be an enhancement for the future.

[cyrossignol]

Rebased and changed the current difficulty field to a pushed value so that it stays fresh along with block count.

[jamescowens]

Let's leave the design as is for now. We can always resurrect my idea if needed. The overhead of g_blockchain_status.UpdateBlockChainStatus can be removed entirely by wrapping it externally with a conditional that checks fQtActive I think.

[jamescowens]

Tested again sync from zero on testnet and mainnet. Looks good. Merging.