Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat:config:force existing users to opt into new defaults #10488

Merged
merged 13 commits into from
Mar 20, 2023

Conversation

ZenGround0
Copy link
Contributor

Related Issues

This is the change to 1) force users to explicitly set splitstore config by uncommenting the default, 2) for new nodes do the uncommenting automatically so they will opt into the default automatically

Proposed Changes

  • Add a notion of uncommenting filters to config update. Then apply a filter on EnableSplitstore in the default full node config
  • Add a notion of loadability checks to the lotus config. Then apply a check ensuring #EnableSplitstore = is not in the config to ensure that users with old configs will reset their config explicity.

Additional Info

  • The patterns introduced here can hopefully be used in other sensative default change situations
  • We can probably revert the splitstore specific filters and loadability checks in a few months when the community is adjusted to the new defaults

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

@ZenGround0 ZenGround0 requested a review from a team as a code owner March 15, 2023 21:35
@jennijuju jennijuju added this to the v1.21.0 milestone Mar 16, 2023
node/config/load.go Outdated Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
Comment on lines 83 to 85
poisonStr := "#EnableSplitstore =" // splitstore must be set explicitly
if strings.Contains(cfgRaw, poisonStr) {
return xerrors.Errorf("Config contains disallowed string %s, refusing to load. Please explicitly set EnableSplitstore. Set it to false to keep chain management unchanged from March 2023 set it to true if you want to discard old state by default", poisonStr)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if I just want the default? What if I comment out one EnableSplitstore line but keep another?

Also, this won't this cause every single node to fail to start on upgrade?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if I just want the default?
won't this cause every single node to fail to start on upgrade?

The point of this change (idea originating here is:

  1. If you want the default and you have an existing, unset config: too bad. You have to set it explicitly or your node won't start. This way users won't accidentally overwrite data.
  2. If you already set your existing config explicitly then your config values won't be changing so that's ok and your node starts.
  3. If you start a new node then the default will be set explicitly for you as part of this PR's change to the default config and your node will start.

What if I comment out one EnableSplitstore line but keep another?

This is a good point I hadn't thought of. What I would like to be doing instead is parsing the TOML and making sure that EnableSplitstore field is set there. I'll go back and take a look.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point of this change (idea originating here is:

Can we just warn loudly and let the user fix their config if they want? If they don't, isn't that their problem? It's not like non-splitstore configs aren't supported...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤷 this honestly isn't something I thought of or particularly care about as a node operator myself. Other node operators appear to a lot and data loss is scary so it makes some sense. @jennijuju as the progenitor of this change could you weigh in?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess going back to the original message it looks like @jennijuju is listening to the advice of glif on this one

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im pretty sure what actually happens is that the universal store will just stay static with all the old state ( the same way the ~130 Gigs of snapshot stay around forever) but the new state will begin to be trimmed. That's still bad but its a much easier problem to recover from. I think I'll test this and if communicate it which will hopefully lead to some more chill.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping this thread alive but the original concern is addressed in the latest commit. We now use regex to ensure that an uncommented EnableSplitstore = preceded only by whitespace exists in the config which should be the correct check.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ribasushi I've confirmed that datastore/chain will stay untouched with default discard splitstore so all of the old historical data will stick around even if we don't get config dance right

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just warn loudly and let the user fix their config if they want? If they don't, isn't that their problem? It's not like non-splitstore configs aren't supported...

We are trying to be more proactive here to avoid full archival node operators to lose their state by mistake. (both zondax and glif have raised valid concerns imho)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm suggesting the other way around. Warn loudly if the user hasn't configured anything, but leave the default the way it is for existing nodes. But I guess we'll need some form of hack either way.

node/config/load.go Outdated Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
@ZenGround0 ZenGround0 force-pushed the feat/safe-splitstore-default branch from 034c9cb to f1ec8d6 Compare March 16, 2023 03:13
@ZenGround0 ZenGround0 requested a review from Stebalien March 16, 2023 05:06
node/config/load.go Outdated Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
node/config/load.go Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
cmd/lotus/config.go Outdated Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
node/config/load.go Outdated Show resolved Hide resolved
@ZenGround0 ZenGround0 requested review from Stebalien and magik6k March 16, 2023 07:02
Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely needs some testing

I don't love that we're forcing users to take action on this, tho on the other hand it seems like without splitstore most non-fullnodes require regular maintenance anyways, and splitstore nodes will have this explicitly set already.

There will be some disruption to user flow, but I don't think it will be too bad - and it can likely be mitigated by clear comms in changelog / announcements.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add some unit tests around EnableSplitstore handling?

node/config/load.go Outdated Show resolved Hide resolved
@ZenGround0 ZenGround0 requested review from jennijuju and magik6k March 18, 2023 03:22
Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, feel free to merge whenever.

Ideally we'd give this a try on a real node too.

node/config/load_test.go Outdated Show resolved Hide resolved
@ZenGround0
Copy link
Contributor Author

Thanks

Ideally we'd give this a try on a real node too.

My mainnet node just crashed so this is a good opportunity.

@ZenGround0 ZenGround0 merged commit 43da108 into master Mar 20, 2023
@ZenGround0 ZenGround0 deleted the feat/safe-splitstore-default branch March 20, 2023 16:19
dumikau pushed a commit to protofire/lotus that referenced this pull request Jun 8, 2023
* Fix the overzealous fix (filecoin-project#10366)

Co-authored-by: zenground0 <[email protected]>

* chore: deps: update to actors v10.0.0

* build: v1.20.0: release prep (filecoin-project#10368)

* add upgrade epoch and bump version

* Update changelog

---------

Co-authored-by: Aayush <[email protected]>

* feat: stmgr: cache migrated stateroots

* feat: vm: switch to the new exec trace format (filecoin-project#10372)

This is now "FVM" native. Changes include:

1. Don't treat "trace" messages like off-chain messages. E.g., don't
include CIDs, versions, etc.
2. Include IPLD codecs where applicable.
3. Remove fields that aren't filled by the FVM (timing, some errors,
code locations, etc.).

* fix: EthAPI: Make newEthBlockFromFilecoinTipSet faster and correct

* feat: update renew-sectors with FIP-0045 logic

* Hide `lotus-worker set` command

Hide the `lotus-worker set` command and print that this command will be deprecated.

* chain prune hot -- hotstore online gc

* remove accidental add

* fix: EthAPI: Correctly get parent hash

* make debugging windowPoSt-failures human readable

* make gen

* indent output

* post worker sched: Filter out disabled workers correctly

* itests: Test PoSt worker RPC error handling

* post worker sched: Retry on alternative worker on RPC errors

* make gen

* itests: Wait for both workers in TestWindowPostWorkerDisconnected

* Unbump api version as its not necessary

* deps: Update go-jsonrpc to v0.2.2

* Add command to trigger moving GC of hotstore manually

* Timing info and updated docs

* fix: EthAPI: Make newEthBlockFromFilecoinTipSet faster and correct

* fix: EthAPI: Correctly get parent hash

* v1.20.1 prep

* fix: state: short-circuit genesis state computation

* fix: state: short-circuit genesis state computation

* update changelog

* Begin account for size during walks

* Track size of dags relevant to compaction

* Plan out moving GC limiting

* GC respects target for max hotstore space

* lint

* rpcenc: More reliably failing TestReaderRedirectDrop

* rpcenc: Propagate closeOnce in beginPost

* fix: eth API: correct gateway restrictions, drop unimplemented methods

* refactor: EthAPI: Drop unnecessary param from newEthTxReceipt

* feat(deps): update to go-data-transfer v2, still wip

* chore(deps): update go-libipfs

* chore(deps): update to get to compile

* update minimum Go version to 1.19.7

* use go-libipfs/files instead of ipfs/go-ipfs-files

* fix drand test

* Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml

* Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml

* Update service_developer_bug_report.yml

* Update service_developer_bug_report.yml

* Update service_developer_bug_report.yml

* un-un-update filecoin-ffi

* try to add statenetworkname method

* add method implementation

* add MPoolGetNonce

* add two more methods: StateCall and StateDecodeParams

* fix logging

* deprecate MarketListRetrievalDeals, make gen

* Fix up approximation and logging

* run make gen

* small doc patch on how to make gen after api changes

* fix TestPrintGroupInfo

* Eth API: make block parameter parsing sounder.

* Eth API: fail when requesting future epochs.

* add a test to verify block parameter soundness.

* register rcmgr metrics

* replace make all with make deps according to review

* fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413)

* cid key size

* computing szPurge leads to deadlock and unneeded, remove

* Review Response

* Lint

* lint

* docsgen-cli

* only register metrics once

* lint

* fix: eth API: return correct txIdx around null blocks (filecoin-project#10419)

* fmt.

* lint

* feat:splitstore:Splitstore enabled by default (filecoin-project#10429)

Discard mode
---------

Co-authored-by: zenground0 <[email protected]>

* Review Response

* fix lint.

* Configur buffer and threshold

* lint

* make gen

* fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds.

* switch to ChainAPI#ChainGetTipSetByHeight.

* review comments

* simplify test.

* feat: chain: make fetching tipset by height 1000x faster

* feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416)

However, we're leaving the default at 1.25x for backwards compatibility, for now.

Also:

1. Actually use the configured replace fee ratio.
2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2),
can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot.

fixes filecoin-project#10415

* try to add statenetworkname method

* add method implementation

* add MPoolGetNonce

* add two more methods: StateCall and StateDecodeParams

* run make gen

* Eth API: make block parameter parsing sounder.

* Eth API: fail when requesting future epochs.

* add a test to verify block parameter soundness.

* fmt.

* fix lint.

* fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds.

* switch to ChainAPI#ChainGetTipSetByHeight.

* simplify test.

* fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413)

* fix: eth API: return correct txIdx around null blocks (filecoin-project#10419)

* feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416)

However, we're leaving the default at 1.25x for backwards compatibility, for now.

Also:

1. Actually use the configured replace fee ratio.
2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2),
can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot.

fixes filecoin-project#10415

* make gen

* change log

* fiix bakcport error filecoin-project#10427

* retract v1.20.2

* go mod tidy

* make gen

* feat: cli: Add an EVM command to fetch a contract's bytecode

* feat: eth API: reject masked ID addresses embedded in f410f payloads

We'll never get an actor/account deployed to one of these
addresses (although we might get a placeholder). However, converting
such an address to an f4 address is definitely wrong.

* chore: update the FFI for release (filecoin-project#10435)

Specifically, this updates the proofs libraries and enables CUDA by default.

* feat: stmgr: skip tipset execution when possible

* feat: eth: optimize receipt reading

This optimizes the eth APIs (except the fee history one) to lookup the
tipset state/receipts instead of computing the state.

* feat: api: optimize ChainGetParentReceipts

Read the receipts all at once instead of fetching them one-by-one.

* test: eth: reduce chances of chain-reorgs affecting the test

Now that this API is _much_ faster, we're more likely to "catch up" to
the head faster than it can stabilize. I'm pretty sure the test was
intended to be written this way anyways.

* gateway: fix: drop overzealous guard on MsigGetVested

* eth_feeHistory: migrate to using TipSetState.

* Eth API: drop support for 'pending' block parameter.

After transitioning from using StateCompute to loading receipts,
we can no longer handle the 'pending' block without forcing
computation. Eth Core Devs are evaluating a proposal to remove
support on their end too.

* Eth API: make net_version return the chain ID.

* fix imports.

* remove superfluous filter.

* drop irrelevant test.

* Eth API: make eth_getTransactionByBlock* ops return ErrUnsupported.

* fix: itest: avoid failing the test when we race the miner

I've been seeing quite a few tests failing here, so I'm hoping this will
deflake them a bit.

* feat: eth: Avoid StateCompute in EthTxnReceipt lookup (filecoin-project#10460)

Instead:

1. Use the receipt from the message search.
2. Re-compute the gas fees that would have been charged.

fixes filecoin-project#10418

Co-authored-by: raulk <[email protected]>

* refactor: update cache to the new generic version (filecoin-project#10463)

- Adds type safety.
- Reduces allocations.
- Fixes the drand cache (was storing by value, but retrieving by pointer)

* fix: gateway: correctly apply the fee history lookback max

This fix ensures that the full fee-history range falls into the
allowable lookback rang.

* Plumb through a proper Flush() method on all blockstores

* feat: consensus: log ApplyBlock timing/gas stats

* fix: stmgr: don't attempt to lookup genesis state (filecoin-project#10472)

* fix: stmgr: don't attempt to lookup genesis state

* stmgr: More correct fix for genesis state compute

* Revert "Eth API: drop support for 'pending' block parameter."

This reverts commit 9412753.

* fix: build: drop drand incentinet servers

* feat: gateway: export StateVerifierStatus

* chore: update go-libp2p-pubsub to v0.9.2

* chore: update ffi to increase execution parallelism (filecoin-project#10480)

This changes the default validation parallelism from 1 to 4, and makes
it configurable with a LOTUS_FVM_CONCURRENCY environment variable.

* fix connection gater integration test

* chore: update go-libp2p to v0.26.2

* make mod-tidy ci happy

* update go-libp2p-pubsub to v0.9.3

* update go-jsonrpc to v0.2.3

* ethrpc: Buffer sub messages if subscriber is slow

* make lint happy

* feat: apply gateway lookback limit to eth API lookback

This change:

1. Introduces new "limited" API endpoints for EthGetTransactionByHash
   and EthGetTransactionReceipt that accept lookback-limits.
2. Implements the gateway version of these API endpoints by calling the
   limited variants with the default message search lookback limit.

fixes filecoin-project#10412

* fix: actually apply limit

* feat: expose limited apis on the gateway

* fix: eth: handle a potential divide by zero in receipt handling

This isn't really possible to hit on-chain at this piont (message won't
be accepted) but we might as well be extra careful.

fixes filecoin-project#10471

* Don't enforce walking receipt tree during compaction

* Demote now common logs (filecoin-project#10516)

Co-authored-by: zenground0 <[email protected]>

* feat:config:force existing users to opt into new defaults (filecoin-project#10488)

* Config default does not comment out EnableSplitstore

* Loadability check

* Remove test used for debugging

* regexp for properly safe check that config is set

* regexp for safely matching the EnableSpitstore field in the config

* Add instructions for undeleting config and remind users to set splitstore false for full archive

* UpdateConfig small docs and functional opts

* make gen

* Lint

* Fix

* nil pointer check on validate

* Unit testing of EnableSplitstore cases

* Address Review

---------

Co-authored-by: zenground0 <[email protected]>

* release v1.21.0-rc1 prep

* chore: build: Update Changelog for v1.21.0

chore: build: Update Changelog for v1.21.0

* Add RPC API highlight

Add RPC API highlight

* Add callout to exchanges

Add callout to exchanges about the execution traces. Also moving the `execution trace format changes` section to above the contributors section.

* fix: miner: correctly count sector extensions

* fix: miner: call ExtendSectorExpiration2

* chore: all: bump go-libipfs

(cherry picked from commit f48c626)

* Initialize with same length as partition

Initialize the postParam.Partitions slice with the same length as i.Partitions before iterating over it in the loop.

* build: docker: Update GO-version

build: docker: Update GO-version

* build: release: v1.21.0-rc2

build: release: v1.21.0-rc2

* Update go.mod

Update go.mod

* Update changelog for v1.21.0-rc2

Update changelog for v1.21.0-rc2

* perf: eth: gas estimate set applyTsMessages false (filecoin-project#10546)

* have gas estimate call callInternal with applyTsMessages = false and other calls with applyTsMessages=true for gas caclulation optimization

* set applyTsMessages = true in CallWithGas call in shed

* update test with new callwithgas api optimization for eth call

* Update chain/stmgr/call.go

Co-authored-by: Łukasz Magiera <[email protected]>

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* make sure that even if we arent apply ts messages we grab ts messages from the particular user who is requesting gas estimation

---------

Co-authored-by: Jiaying Wang <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: Ubuntu <[email protected]>

* feat: supply: drop genesis market locked funds

* feat: supply: only grab genesis msig locks for writes

* Update Changelog with backports

Update Changelog with backports

* Bump version in changelog

Bump version in changelog

* Add longer validation time cause to changelog

Add longer validation time cause to changelog

* refactor: stop using deprecated io/ioutil

* Fix error when injecting StateManager

* new actor bundle, fix ci, upgrade fx

* fix fx version

* upgrade go version circle ci

* make linter happy :)

* more go version fixing

* replace deprecated mac build

* removing support for macos build

* Fix ctrl-c for GOOD

---------

Co-authored-by: ZenGround0 <[email protected]>
Co-authored-by: zenground0 <[email protected]>
Co-authored-by: Aayush Rajasekaran <[email protected]>
Co-authored-by: Jiaying Wang <[email protected]>
Co-authored-by: Jennifer Wang <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: Travis Person <[email protected]>
Co-authored-by: Steven Allen <[email protected]>
Co-authored-by: Phi <[email protected]>
Co-authored-by: beck <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: hannahhoward <[email protected]>
Co-authored-by: Marten Seemann <[email protected]>
Co-authored-by: Arsenii Petrovich <[email protected]>
Co-authored-by: Arsenii Petrovich <[email protected]>
Co-authored-by: Raúl Kripalani <[email protected]>
Co-authored-by: ychiao <[email protected]>
Co-authored-by: Travis Person <[email protected]>
Co-authored-by: raulk <[email protected]>
Co-authored-by: Peter Rabbitson <[email protected]>
Co-authored-by: vyzo <[email protected]>
Co-authored-by: Jorropo <[email protected]>
Co-authored-by: Mikers <[email protected]>
Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Henrique Moniz <[email protected]>
dumikau pushed a commit to protofire/lotus that referenced this pull request Jun 8, 2023
…ed (filecoin-project#144)

* use go-libipfs/files instead of ipfs/go-ipfs-files

* fix drand test

* Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml

* Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml

* Update service_developer_bug_report.yml

* Update service_developer_bug_report.yml

* Update service_developer_bug_report.yml

* un-un-update filecoin-ffi

* try to add statenetworkname method

* add method implementation

* add MPoolGetNonce

* add two more methods: StateCall and StateDecodeParams

* fix logging

* deprecate MarketListRetrievalDeals, make gen

* Fix up approximation and logging

* run make gen

* small doc patch on how to make gen after api changes

* fix TestPrintGroupInfo

* Eth API: make block parameter parsing sounder.

* Eth API: fail when requesting future epochs.

* add a test to verify block parameter soundness.

* register rcmgr metrics

* replace make all with make deps according to review

* fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413)

* cid key size

* computing szPurge leads to deadlock and unneeded, remove

* Review Response

* Lint

* lint

* docsgen-cli

* only register metrics once

* lint

* fix: eth API: return correct txIdx around null blocks (filecoin-project#10419)

* fmt.

* lint

* feat:splitstore:Splitstore enabled by default (filecoin-project#10429)

Discard mode
---------

Co-authored-by: zenground0 <[email protected]>

* Review Response

* fix lint.

* Configur buffer and threshold

* lint

* make gen

* fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds.

* switch to ChainAPI#ChainGetTipSetByHeight.

* review comments

* simplify test.

* feat: chain: make fetching tipset by height 1000x faster

* feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416)

However, we're leaving the default at 1.25x for backwards compatibility, for now.

Also:

1. Actually use the configured replace fee ratio.
2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2),
can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot.

fixes filecoin-project#10415

* try to add statenetworkname method

* add method implementation

* add MPoolGetNonce

* add two more methods: StateCall and StateDecodeParams

* run make gen

* Eth API: make block parameter parsing sounder.

* Eth API: fail when requesting future epochs.

* add a test to verify block parameter soundness.

* fmt.

* fix lint.

* fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds.

* switch to ChainAPI#ChainGetTipSetByHeight.

* simplify test.

* fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413)

* fix: eth API: return correct txIdx around null blocks (filecoin-project#10419)

* feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416)

However, we're leaving the default at 1.25x for backwards compatibility, for now.

Also:

1. Actually use the configured replace fee ratio.
2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2),
can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot.

fixes filecoin-project#10415

* make gen

* change log

* fiix bakcport error filecoin-project#10427

* retract v1.20.2

* go mod tidy

* make gen

* feat: cli: Add an EVM command to fetch a contract's bytecode

* feat: eth API: reject masked ID addresses embedded in f410f payloads

We'll never get an actor/account deployed to one of these
addresses (although we might get a placeholder). However, converting
such an address to an f4 address is definitely wrong.

* chore: update the FFI for release (filecoin-project#10435)

Specifically, this updates the proofs libraries and enables CUDA by default.

* feat: stmgr: skip tipset execution when possible

* feat: eth: optimize receipt reading

This optimizes the eth APIs (except the fee history one) to lookup the
tipset state/receipts instead of computing the state.

* feat: api: optimize ChainGetParentReceipts

Read the receipts all at once instead of fetching them one-by-one.

* test: eth: reduce chances of chain-reorgs affecting the test

Now that this API is _much_ faster, we're more likely to "catch up" to
the head faster than it can stabilize. I'm pretty sure the test was
intended to be written this way anyways.

* gateway: fix: drop overzealous guard on MsigGetVested

* eth_feeHistory: migrate to using TipSetState.

* Eth API: drop support for 'pending' block parameter.

After transitioning from using StateCompute to loading receipts,
we can no longer handle the 'pending' block without forcing
computation. Eth Core Devs are evaluating a proposal to remove
support on their end too.

* Eth API: make net_version return the chain ID.

* fix imports.

* remove superfluous filter.

* drop irrelevant test.

* Eth API: make eth_getTransactionByBlock* ops return ErrUnsupported.

* fix: itest: avoid failing the test when we race the miner

I've been seeing quite a few tests failing here, so I'm hoping this will
deflake them a bit.

* feat: eth: Avoid StateCompute in EthTxnReceipt lookup (filecoin-project#10460)

Instead:

1. Use the receipt from the message search.
2. Re-compute the gas fees that would have been charged.

fixes filecoin-project#10418

Co-authored-by: raulk <[email protected]>

* refactor: update cache to the new generic version (filecoin-project#10463)

- Adds type safety.
- Reduces allocations.
- Fixes the drand cache (was storing by value, but retrieving by pointer)

* fix: gateway: correctly apply the fee history lookback max

This fix ensures that the full fee-history range falls into the
allowable lookback rang.

* Plumb through a proper Flush() method on all blockstores

* feat: consensus: log ApplyBlock timing/gas stats

* fix: stmgr: don't attempt to lookup genesis state (filecoin-project#10472)

* fix: stmgr: don't attempt to lookup genesis state

* stmgr: More correct fix for genesis state compute

* Revert "Eth API: drop support for 'pending' block parameter."

This reverts commit 9412753.

* fix: build: drop drand incentinet servers

* feat: gateway: export StateVerifierStatus

* chore: update go-libp2p-pubsub to v0.9.2

* chore: update ffi to increase execution parallelism (filecoin-project#10480)

This changes the default validation parallelism from 1 to 4, and makes
it configurable with a LOTUS_FVM_CONCURRENCY environment variable.

* fix connection gater integration test

* chore: update go-libp2p to v0.26.2

* make mod-tidy ci happy

* update go-libp2p-pubsub to v0.9.3

* update go-jsonrpc to v0.2.3

* ethrpc: Buffer sub messages if subscriber is slow

* make lint happy

* feat: apply gateway lookback limit to eth API lookback

This change:

1. Introduces new "limited" API endpoints for EthGetTransactionByHash
   and EthGetTransactionReceipt that accept lookback-limits.
2. Implements the gateway version of these API endpoints by calling the
   limited variants with the default message search lookback limit.

fixes filecoin-project#10412

* fix: actually apply limit

* feat: expose limited apis on the gateway

* fix: eth: handle a potential divide by zero in receipt handling

This isn't really possible to hit on-chain at this piont (message won't
be accepted) but we might as well be extra careful.

fixes filecoin-project#10471

* Don't enforce walking receipt tree during compaction

* Demote now common logs (filecoin-project#10516)

Co-authored-by: zenground0 <[email protected]>

* feat:config:force existing users to opt into new defaults (filecoin-project#10488)

* Config default does not comment out EnableSplitstore

* Loadability check

* Remove test used for debugging

* regexp for properly safe check that config is set

* regexp for safely matching the EnableSpitstore field in the config

* Add instructions for undeleting config and remind users to set splitstore false for full archive

* UpdateConfig small docs and functional opts

* make gen

* Lint

* Fix

* nil pointer check on validate

* Unit testing of EnableSplitstore cases

* Address Review

---------

Co-authored-by: zenground0 <[email protected]>

* release v1.21.0-rc1 prep

* chore: build: Update Changelog for v1.21.0

chore: build: Update Changelog for v1.21.0

* Add RPC API highlight

Add RPC API highlight

* Add callout to exchanges

Add callout to exchanges about the execution traces. Also moving the `execution trace format changes` section to above the contributors section.

* fix: miner: correctly count sector extensions

* fix: miner: call ExtendSectorExpiration2

* chore: all: bump go-libipfs

(cherry picked from commit f48c626)

* Initialize with same length as partition

Initialize the postParam.Partitions slice with the same length as i.Partitions before iterating over it in the loop.

* build: docker: Update GO-version

build: docker: Update GO-version

* build: release: v1.21.0-rc2

build: release: v1.21.0-rc2

* Update go.mod

Update go.mod

* Update changelog for v1.21.0-rc2

Update changelog for v1.21.0-rc2

* perf: eth: gas estimate set applyTsMessages false (filecoin-project#10546)

* have gas estimate call callInternal with applyTsMessages = false and other calls with applyTsMessages=true for gas caclulation optimization

* set applyTsMessages = true in CallWithGas call in shed

* update test with new callwithgas api optimization for eth call

* Update chain/stmgr/call.go

Co-authored-by: Łukasz Magiera <[email protected]>

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* make sure that even if we arent apply ts messages we grab ts messages from the particular user who is requesting gas estimation

---------

Co-authored-by: Jiaying Wang <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: Ubuntu <[email protected]>

* feat: supply: drop genesis market locked funds

* feat: supply: only grab genesis msig locks for writes

* Update Changelog with backports

Update Changelog with backports

* Bump version in changelog

Bump version in changelog

* Add longer validation time cause to changelog

Add longer validation time cause to changelog

* refactor: stop using deprecated io/ioutil

* Fix error when injecting StateManager

* new actor bundle, fix ci, upgrade fx

* fix fx version

* upgrade go version circle ci

* make linter happy :)

* more go version fixing

* replace deprecated mac build

* removing support for macos build

* Fix ctrl-c for GOOD

* bypass mpool rm for set membership

---------

Co-authored-by: Marten Seemann <[email protected]>
Co-authored-by: Jiaying Wang <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: Arsenii Petrovich <[email protected]>
Co-authored-by: zenground0 <[email protected]>
Co-authored-by: Arsenii Petrovich <[email protected]>
Co-authored-by: Raúl Kripalani <[email protected]>
Co-authored-by: ychiao <[email protected]>
Co-authored-by: Aayush Rajasekaran <[email protected]>
Co-authored-by: ZenGround0 <[email protected]>
Co-authored-by: Travis Person <[email protected]>
Co-authored-by: Travis Person <[email protected]>
Co-authored-by: Steven Allen <[email protected]>
Co-authored-by: Jennifer Wang <[email protected]>
Co-authored-by: raulk <[email protected]>
Co-authored-by: Peter Rabbitson <[email protected]>
Co-authored-by: vyzo <[email protected]>
Co-authored-by: Phi <[email protected]>
Co-authored-by: Jorropo <[email protected]>
Co-authored-by: Mikers <[email protected]>
Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Henrique Moniz <[email protected]>
Co-authored-by: Denis Kolegov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants