Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid ingesting binary, unused data #204

Closed
hsanjuan opened this issue Nov 4, 2020 · 0 comments · Fixed by #241
Closed

Avoid ingesting binary, unused data #204

hsanjuan opened this issue Nov 4, 2020 · 0 comments · Fixed by #241
Assignees
@frrist
Labels
kind/chore Technical debt due to be resolved P2 P2: Should be resolved

Comments

@hsanjuan
Copy link
Contributor

hsanjuan commented Nov 4, 2020 •
edited by placer14
Loading

We should be dropping as much information as we can when ingesting chain data.

The first example I have is the Proof fields in jsonb in the parsed_messages table, which for a full chain seems to amount to a lot of data. The messages table also stores the binary parameters (which is redundant if it is also stored as parsed data?).

There are binary columns like ticket and election_proof in the block_headers table. We are also storing the drand binary randomness value.

Acceptance Criteria

  • Remove ticket and election_proof from block_headers
  • Remove params from messages
  • Remove drand_entries entire
  • Remove return in receipts
@placer14 placer14 added the P2 P2: Should be resolved label Nov 12, 2020
@frrist frrist added the kind/chore Technical debt due to be resolved label Nov 13, 2020
@frrist frrist self-assigned this Nov 18, 2020
@frrist frrist mentioned this issue Nov 18, 2020
Merged
frrist added a commit that referenced this issue Nov 18, 2020
@frrist
chore: remove ingestion for binary and unused data
411f7cd 
- Remove the following tables:
  - public.drand_entries
- Remove the Following columns:
  - public.block_headers ticket
  - public.block_headers election_proof
  - public.messages params
  - public.receipts return

closes #204
placer14 added a commit that referenced this issue Nov 19, 2020
@frrist @placer14
chore: Avoid ingesting binary and unused data (#241)
d588dea 
* chore: remove ingestion for binary and unused data

- Remove the following tables:
  - public.drand_entries
- Remove the Following columns:
  - public.block_headers ticket
  - public.block_headers election_proof
  - public.messages params
  - public.receipts return

closes #204

* fix: Reorder migration to merge to master

Co-authored-by: Mike Greenberg <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@frrist frrist

Labels
kind/chore Technical debt due to be resolved P2 P2: Should be resolved
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

chore: Avoid ingesting binary and unused data filecoin-project/lily
3 participants
@placer14 @hsanjuan @frrist