[Merged by Bors] - in memory index with atx data for consensus layer

[dshulyak]

closes: #4164

this change introduces index with minimal data about identities which is required for consensus. the goal is to have predictable latency on the codepath that are sensitive to it, such as proposals and even more hare messages. the number of issues with that will only grow when sql db size won't fit in page cache (so querying vrf nonce and atx data will hit the disk often).

cached data per identity:

• weight
• height and base height
• vrf nonce
• malicious boolean

the data is added to the cache either on startup or when new atx is received. nonce may be loaded from previous atx if it wasn't included in this one. when block from epoch X is applied, we will evict epochs starting from X-capacity. index is configured to fit same number of epochs that fit in the tortoise window, this enables to self-healing during this period.

internally we store data indexed by epoch for simple eviction, and each epoch has an index by atx.

memory estimate for 1_000_000 identities is ~140MB

[codecov]

Codecov Report

Merging #5013 (fbf6667) into develop (6da9e61) will increase coverage by 0.1%.
Report is 2 commits behind head on develop.
The diff coverage is 93.1%.

@@            Coverage Diff            @@
##           develop   #5013     +/-   ##
=========================================
+ Coverage     77.7%   77.9%   +0.1%     
=========================================
  Files          262     264      +2     
  Lines        31112   31375    +263     
=========================================
+ Hits         24204   24461    +257     
- Misses        5403    5410      +7     
+ Partials      1505    1504      -1     

Files	Coverage Δ
atxsdata/data.go	100.0% <100.0%> (ø)
atxsdata/warmup.go	100.0% <100.0%> (ø)
common/types/verified_activation.go	100.0% <100.0%> (ø)
mesh/mesh.go	64.3% <100.0%> (+0.1%)	⬆️
proposals/eligibility_validator.go	97.7% <100.0%> (-2.3%)	⬇️
sql/database.go	73.3% <ø> (+1.6%)	⬆️
proposals/handler.go	88.2% <97.7%> (+0.7%)	⬆️
datastore/store.go	70.3% <75.0%> (-2.3%)	⬇️
node/node.go	65.0% <81.2%> (+<0.1%)	⬆️
activation/handler.go	76.7% <81.2%> (+0.4%)	⬆️
... and 1 more

... and 2 files with indirect coverage changes

[dshulyak]

bors try

[bors]

try

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

• systest-status
• ci-status

[dshulyak]

bors try

[bors]

try

Build failed:

• ci-status

[dshulyak]

will create a separate to follow to adjust those places in a different pr

where it will be used?

- no atx queries in ballot/proposal handlers (implemented in this change)
- no identity and atx queries in hare
- no atx queries in block generator
- atx queries during mesh hash resolution

will allow to

- drop atxs data from tortoise
- drop lru cache for atxs and identities

[dshulyak]

bors try

[bors]

try

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

• ci-status
• systest-status

[dshulyak]

bors try

[bors]

try

Build failed:

• ci-status

[dshulyak]

bors try

[bors]

try

Build failed:

• ci-status

[dshulyak]

bors merge

[bors]

Pull request successfully merged into develop.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

• ci-status
• systest-status