Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store fil value in db as numeric type #244

Closed
olizilla opened this issue Nov 18, 2020 · 3 comments · Fixed by #327
Closed

Store fil value in db as numeric type #244

olizilla opened this issue Nov 18, 2020 · 3 comments · Fixed by #327
Assignees
Labels
kind/enhancement Improvement to an existing feature

Comments

@olizilla
Copy link
Contributor

Description

We currently store FIL values as TEXT in the db. We often want to aggregate over the numeric value of the FIL, so lets store FIL as NUMERIC.

The type numeric can store numbers with a very large number of digits. It is especially recommended for storing monetary amounts and other quantities where exactness is required. Calculations with numeric values yield exact results where possible, e.g., addition, subtraction, multiplication. However, calculations on numeric values are very slow compared to the integer types, or to the floating-point types described in the next section.
https://www.postgresql.org/docs/12/datatype-numeric.html#DATATYPE-NUMERIC-DECIMAL

Caveat:

These [sentinel] tables are supposed to be the "raw" form, optimised for ingestion. The plan is to then create the queryable schema by joining and transforming into new tables or views.

Even so, I propose that we consider the raw, ingest form of the value column to be numeric rather than text.

Acceptance criteria

The value column is NUMERIC in all tables and views that record them.

Where to begin

  • Determine if there is any reason not to do this?
  • Write a migration.
@iand iand added the kind/enhancement Improvement to an existing feature label Nov 18, 2020
@iand
Copy link
Contributor

iand commented Nov 18, 2020

As far as I tell go-pg has no automatic support for big.Int/numeric types.

@frrist
Copy link
Member

frrist commented Nov 18, 2020

This leads me to believe there is undocumented support: go-pg/pg@d36b827.

@iand
Copy link
Contributor

iand commented Nov 20, 2020

Text fields that could be changed (since they are big.Int in lotus):

  • message.value
  • message.gas_fee_cap
  • message.gas_premium
  • derived_gas_outputs.value
  • derived_gas_outputs.gas_fee_cap
  • derived_gas_outputs.gas_premium
  • derived_gas_outputs.parent_base_fee
  • miner_locked_funds.locked_funds
  • miner_locked_funds.initial_pledge
  • miner_locked_funds.pre_commit_deposits
  • miner_fee_debts.fee_debt
  • miner_pre_commit_infos.pre_commit_deposit
  • miner_pre_commit_infos.deal_weight
  • miner_pre_commit_infos.verified_deal_weight
  • miner_sector_infos.deal_weight
  • miner_sector_infos.verified_deal_weight
  • miner_sector_infos.initial_pledge
  • miner_sector_infos.expected_day_reward
  • miner_sector_infos.expected_storage_pledge
  • chain_powers.total_pledge_collateral
  • chain_powers.total_raw_bytes_power
  • chain_powers.total_raw_bytes_committed
  • chain_powers.total_qa_bytes_power
  • chain_powers.total_qa_bytes_committed
  • power_actor_claims.raw_byte_powerchain_rewards
  • power_actor_claims.quality_adj_power
  • chain_rewards.effective_baseline_power
  • chain_rewards.cum_sum_baseline
  • chain_rewards.cum_sum_realized
  • chain_rewards.new_baseline_power
  • chain_rewards.new_reward_smoothed_position_estimate
  • chain_rewards.new_reward_smoothed_velocity_estimate
  • chain_rewards.total_mined_reward
  • chain_rewards.new_reward

Note message.gas_limit, derived_gas_outputs.gas_used and derived_gas_outputs.gas_limit are bigint (int64 in the lotus types package)

Other possibilities (need confirming):

  • message_gas_economy.base_fee - is a float at moment
  • message_gas_economy.gas_limit_total- bigint but is sum of other bigints so could overflow
  • message_gas_economy.gas_limit_unique_total - same as gas_limit_total

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement Improvement to an existing feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants