Transaction calldata gas cost reduction with total calldata limit

[vbuterin]

Decrease transaction calldata gas cost, and add a limit of how much total transaction calldata can be in a block.

[eth-bot]

All tests passed; auto-merging...

(pass) eip-4488.md

• passed!

(pass) assets/eip-4488/gas_and_calldata_sample.csv

• file assets/eip-4488/gas_and_calldata_sample.csv is associated with EIP 4488; because there are also changes being made to EIPS/eip-4488.md all changes to corresponding assets are also allowed

[axic]

@MicahZoltu would argue to not have any external links, and potentially for this particular case to include the CSV file in the assets directory 😅

I'm happy with the external links though as they are not critical (in case they go dead), but while they are active they add some extra flavour.

[MicahZoltu]

You know me so well 🥲

[MicahZoltu]

Suggested change

Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. 90% of all transactions ([sample](http://vitalik.ca/files/misc_files/gas_and_calldata_sample.csv) taken from blocks `13500000, 13501000 ... 13529000`) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.

Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. At the time of writing this, 90% of all transactions taken from blocks `13500000, 13501000 ... 13529000`) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.

As @axic indicated, I think this document would be great to attach in the discussion, but I think it adds fairly minimal value long-term (to a reader in 5 years). I think just making the statement is enough, and if someone in 5 years really wants to they can dig up the data themselves or read the discussion history. If you think it really should be included, then please put it in ../assets/eip-4488/gas_and_calldata_sample.csv and link accordingly.

[jonathanbier]

Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. 90% of all transactions (sample taken from blocks 13500000, 13501000 ... 13529000) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.

Is the multi-dimentional problem also for users and wallets, not just block producers?

[MicahZoltu]

It is only a problem for block producers.

[jonathanbier]

Why don’t wallets estimating the fee also have this issue?

[MicahZoltu]

Please move this discussion to the discussions-to link: https://ethereum-magicians.org/t/eip-4488-transaction-calldata-gas-cost-reduction-with-total-calldata-limit/7555

[axic]

Not a needed section:

Suggested change

## Reference Implementation

[spartucus]

Bringing more possibilities to Rollups. :)

[MicahZoltu]

Format/layout looks good. My only complaint is the external links. Other editors may approve to merge this with the links included, but I take a fairly hard lined stand against them.

[MicahZoltu]

Suggested change

A natural alternative proposal is to decrease `NEW_CALLDATA_GAS_COST` without adding a limit. However, this presents a security concern: today, the average block size [is 60-90 kB](https://etherscan.io/chart/blocksize), but the _maximum_ block size is `30M / 16 = 1,875,000` bytes (plus about a kilobyte of block and tx overhead). Simply decreasing the calldata gas cost from 16 to 3 would increase the maximum block size to 10M bytes. This would push the Ethereum p2p networking layer to unprecedented levels of strain and risk breaking the network; some previous live tests of ~500 kB blocks a few years ago had already taken down a few bootstrap nodes.

A natural alternative proposal is to decrease `NEW_CALLDATA_GAS_COST` without adding a limit. However, this presents a security concern: at the time of writing this, the average block size is 60-90 kB, but the _maximum_ block size is `30M / 16 = 1,875,000` bytes (plus about a kilobyte of block and tx overhead). Simply decreasing the calldata gas cost from 16 to 3 would increase the maximum block size to 10M bytes. This would push the Ethereum p2p networking layer to unprecedented levels of strain and risk breaking the network; some previous live tests of ~500 kB blocks a few years ago had already taken down a few bootstrap nodes.

I think there may be value in including this link in the discussion thread, but in 5-10 years the link will likely be a 404 and it almost certainly won't contain useful data for the context of someone reading this EIP at that time so it shouldn't be included in the EIP itself.

[fbartho]

Maybe the right place for this link is as a comment linking to the statement -- directly on this PR.

Presuming the PR is merged, we don't need to go back and verify the block-size statement again, since potentially new sites would provide that data when a future reader looks it up.

[MicahZoltu]

Suggested change

Historically, the Ethereum protocol community has been suspicious of multi-dimensional resource limit rules (in this case, gas and calldata) because such rules greatly increase the complexity of the block packing problem that proposers (today miners, post-merge validators) need to solve. Today, proposers can generate blocks with near-optimal fee revenue by simply choosing transactions in highest-to-lowest order of priority fee. In a multi-dimensional world, proposers would have to deal with multi-dimensional constraints. [Multi-dimensional knapsack problems](https://en.wikipedia.org/wiki/Knapsack_problem#Multi-dimensional_knapsack_problem) are much more complicated than the single-dimensional equivalent, and well-optimized proprietary implementations built by pools may well outperform default open source implementations.

Historically, the Ethereum protocol community has been suspicious of multi-dimensional resource limit rules (in this case, gas and calldata) because such rules greatly increase the complexity of the block packing problem that proposers (today miners, post-merge validators) need to solve. Today, proposers can generate blocks with near-optimal fee revenue by simply choosing transactions in highest-to-lowest order of priority fee. In a multi-dimensional world, proposers would have to deal with multi-dimensional constraints. Multi-dimensional knapsack problems are much more complicated than the single-dimensional equivalent, and well-optimized proprietary implementations built by pools may well outperform default open source implementations.

Someone who wants to know what the Multi-dimensional knapsack problem is can search the internet for it just like they could for any other term they are unfamiliar with in this document. Separately (and unrelated to this PR), Wikipedia is pure evil now.

[ademidun]

Hey @MicahZoltu just out of curiosity, why do you think that about Wikipedia?

[MicahZoltu]

Their editorial "staff" (technically volunteers I think) has been captured by groups with strong biases and an unwillingness to present a factual and neutral view that doesn't align with their personal world view.

Try to make technical corrections to cryptocurrency related Wikipedia pages if you want to feel what it is like personally. As an example, a while back I tried to make corrections to a page about an open source dapp and all of my edits (purely technical corrections based on the source code) were rejected because mainstream media hadn't reported on them, and thus they claimed I was synthesizing information.

[MicahZoltu]

Suggested change

Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. 90% of all transactions ([sample](http://vitalik.ca/files/misc_files/gas_and_calldata_sample.csv) taken from blocks `13500000, 13501000 ... 13529000`) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.

Additionally, the stipend mechanism makes the two-dimensional optimization problem even less of an issue in practice. At the time of writing this, 90% of all transactions taken from blocks `13500000, 13501000 ... 13529000`) have <300 bytes of calldata. Hence, if a naive transaction selection algorithm overfills the calldata of a block that the proposer is creating, the proposer will still be able to keep adding transactions from 90% of their mempool.

As @axic indicated, I think this document would be great to attach in the discussion, but I think it adds fairly minimal value long-term (to a reader in 5 years). I think just making the statement is enough, and if someone in 5 years really wants to they can dig up the data themselves or read the discussion history. If you think it really should be included, then please put it in ../assets/eip-4488/gas_and_calldata_sample.csv and link accordingly.

[MicahZoltu]

Suggested change

However, even these fees are too expensive for many users. The long-term solution to the long-term inadequacy of rollups by themselves has always been [data sharding](https://github.com/ethereum/consensus-specs#sharding), which would add ~1-2 MB/sec of dedicated data space for rollups to the chain. However, data sharding will still take a considerable amount of time to finish implementing and deploying. Hence, a short-term solution to further cut costs for rollups, and to incentivize an ecosystem-wide transition to a rollup-centric Ethereum, is desired.

However, even these fees are too expensive for many users. The long-term solution to the long-term inadequacy of rollups by themselves has always been data sharding, which would add ~1-2 MB/sec of dedicated data space for rollups to the chain. However, data sharding will still take a considerable amount of time to finish implementing and deploying. Hence, a short-term solution to further cut costs for rollups, and to incentivize an ecosystem-wide transition to a rollup-centric Ethereum, is desired.

I'm generally OK with linking to other Ethereum specifications, but I prefer to avoid it when possible in case those specifications move to a new location. In this case, I think that there is some value in linking here, but not enough to overcome my personal burden of need to warrant an external link. Understanding data sharding isn't critical to understanding this EIP and it is a relatively easy term to search the internet for if you are uncertain about it.

[fbartho]

I tend to disagree on this suggested removal, as the official nature of this link isn't an "external" link in the same way that linking to a different site would be; perhaps directly linking to a specific commit's variant of the standard would help?

Additionally, the URL itself has value for future readers, as it gives them some useful keywords for finding the right section of a hypothetically moved spec.

[yoavw]

This is indeed the optimal strategy for miners - to drop rollup transactions in favor of more execution-oriented transactions. Isn't there a risk that it'll actually hurt rollups?

At high congestion times (e.g. big NFT sales) rollup transactions will be constantly dropped, and they'll have to compensate for the lack of execution gas by paying a higher total fee. Theoretically it should drive rollup gas fees to the current cost, except that it also limits the calldata size. The additional constraint might require them to pay an even higher fee to outbid other rollups competing on the same calldata space.

[MicahZoltu]

I recommend making this comment in the discussions-to link found in the EIP header. PR comments are very transient and not a good place for discussions about the EIP content.

[yoavw]

Done

[stevengcook]

Add a rule that a block is only valid if sum(len(tx.calldata) for tx in block.txs) <= BASE_MAX_CALLDATA_PER_BLOCK + len(block.txs) * CALLDATA_PER_TX_STIPEND.

Would it not be easier (and slightly less variable max calldata per block) if only transactions with more than CALLDATA_PER_TX_STIPEND are counted against BASE_MAX_CALLDATA_PER_BLOCK?
Assuming 300 is the stipend per block, transactions with <= 300 calldata don't provide extra room for transactions that use > 300 calldata.

[MicahZoltu]

@stevengcook I recommend asking this question in the discussions-to forum thread (at top of EIP).

[stevengcook]

@stevengcook I recommend asking this question in the discussions-to forum thread (at top of EIP).

my bad, thanks!

[alita-moore]

A critical exception has occurred (cc @alita-moore): Message: not found

I will fix this today, btw

[hydrogenbond007]

Rollup builders are gonna love this + some chains that don't support blob txns are using this proposal for research!