Code

shim layer for reprocessing blocks#705

Code

Draft

darioush wants to merge 245 commits intomasterava-labs/coreth:masterfrom prefetch-stateava-labs/coreth:prefetch-stateCopy head branch name to clipboard

+3,762-91

Commits

Commits on Dec 10, 2024

• parallel state prefetcher

  darioush

  committed
  50899d6

• fix prefetching

  darioush

  committed
  7ff1e82

• reprocessing stuff

  darioush

  committed
  fd122d7

• use accept trie

  darioush

  committed
  089d677

• patch

  darioush

  committed
  5220cfb

• patch

  darioush

  committed
  9cdaa6e

• fix log

  darioush

  committed
  46ef608

• try

  darioush

  committed
  26125dd

• more stats

  darioush

  committed
  a5225c1

• quieter log

  darioush

  committed
  bdcbba9

• try again

  darioush

  committed
  d019ac5

• add cacher recovery

  darioush

  committed
  1179807

• prefetch serially

  darioush

  committed
  d8522ab

• make sure is working

  darioush

  committed
  2573353

• more like orig

  darioush

  committed
  3e99257

• add tape (UTs work)

  darioush

  committed
  c60eb31

• reprocess again

  darioush

  committed
  e9c4e9f

Commits on Dec 11, 2024

• snapshot tape

  darioush

  committed
  90066e6

• add back reprocess

  darioush

  committed
  db61d14

• include in replay

  darioush

  committed
  0ba565f

• add finalize

  darioush

  committed
  08ebb34

• try off metrics expensive

  darioush

  committed
  caf549a

• check rtime

  darioush

  committed
  0a3d36d

Commits on Dec 13, 2024

• shim layers for trie interception / use of kv backend: "stuff working"

  darioush

  committed
  a3b5918

• fix non-pruning / stacking views

  darioush

  committed
  25db320

• making sure it works for storage tries

  darioush

  committed
  b016b4b

Commits on Dec 14, 2024

• add snapshots

  darioush

  committed
  5b25442

• nit

  darioush

  committed
  e722e8f

• clarify interface

  darioush

  committed
  6d6fcc9

Commits on Dec 17, 2024

• fixes, support for start/stop

  darioush

  committed
  b963cf9

• "fix" races

  darioush

  committed
  17ca744

Commits on Dec 18, 2024

• try export blocks

  darioush

  committed
  02d577e

• fix

  darioush

  committed
  2e44670

• minor updates

  darioush

  committed
  8379845

• log avax assetID

  darioush

  committed
  3db0060

• refactor towards pluggable backend

  darioush

  committed
  97eb4c4

• refactor: separate snapshot verification

  darioush

  committed
  d500ff9

• not needed to drain acceptor queue

  darioush

  committed
  fe09f8b

• make it work with LegacyBackend again

  darioush

  committed
  28e19fe

• fix UTs

  darioush

  committed
  159f8f4

• patch

  darioush

  committed
  d42c028

Commits on Dec 19, 2024

• improvements

  darioush

  committed
  2568c3c

• inline handling of genesis to reprocess fn.

  darioush

  committed
  bdc8a23

• refactor more stuff into backend

  darioush

  committed
  55da582

• try to reprocess mainnet blocks in memory

  darioush

  committed
  d86fc2e

• use cmd flag for end

  darioush

  committed
  b581345

• try verify root

  darioush

  committed
  d3eca27

• try

  darioush

  committed
  23226a8

• try

  darioush

  committed
  83f9203

• try

  darioush

  committed
  a3854a4

• try

  darioush

  committed
  cda0386

• fix

  darioush

  committed
  7878f88

• add metadata store

  darioush

  committed
  e7252be

• track metadata, add flags

  darioush

  committed
  e83dc48

• off by one

  darioush

  committed
  19fddd1

• need to close mdb

  darioush

  committed
  2f655f5

• try

  darioush

  committed
  c0a6254

• try

  darioush

  committed
  a045983

• try

  darioush

  committed
  17ab510

• try

  darioush

  committed
  7d1e9f6

• try

  darioush

  committed
  f29efd6

• try

  darioush

  committed
  7f75d1e

• oops

  darioush

  committed
  5ebdd58

• try

  darioush

  committed
  a31dfb0

• try

  darioush

  committed
  69b9f59

• modify log

  darioush

  committed
  426baed

• add log

  darioush

  committed
  8ea65e7

• try

  darioush

  committed
  bbf32ce

• add hash

  darioush

  committed
  4361cfb

• try

  darioush

  committed
  1915247

• try

  darioush

  committed
  a306b81

• try

  darioush

  committed
  ce5b773

• try

  darioush

  committed
  514042a

• try

  darioush

  committed
  4745972

• try

  darioush

  committed
  515ce06

• add logging for atomic tx

  darioush

  committed
  6fea3bb

• use term string

  darioush

  committed
  9a940ba

• fix UTs

  darioush

  committed
  ac5230e

• fix UT again

  darioush

  committed
  a955bb8

• add snapshot iteration as test

  darioush

  committed
  9d3a9a4

• add merkledb flags

  darioush

  committed
  6369319

• change units

  darioush

  committed
  30105d4

• fix options

  darioush

  committed
  1ee79b5

• support upgrades

  darioush

  committed
  74f8e5b

• support skipUpgradeCheck

  darioush

  committed
  8237a65

Commits on Dec 20, 2024

• decouple prefetcher from snap

  darioush

  committed
  021fc39

• add usePersistedStartBlock

  darioush

  committed
  003ac7d

Commits on Dec 24, 2024

• add tapeRecorder

  darioush

  committed
  436cae0

• better logging w/ tape

  darioush

  committed
  e414e58

• reduce logs

  darioush

  committed
  fca800d

• formatting

  darioush

  committed
  02e97db

• fixes

  darioush

  committed
  9a4a636

• add file manager

  darioush

  committed
  a21a86e

• fix

  darioush

  committed
  718b2a2

• add create permissions

  darioush

  committed
  e277e99

• align files

  darioush

  committed
  ee1d9bc

• dont cap on pathdb

  darioush

  committed
  5ca3b36

• add option for trie clean cache

  darioush

  committed
  7172a14

Commits on Dec 25, 2024

• use tape representation

  darioush

  committed
  08d1cef

• add verification for tx end markers

  darioush

  committed
  a0bb20d

Commits on Dec 27, 2024

• some postprocessing

  darioush

  committed
  52d9c3d

• formatting

  darioush

  committed
  f704a65

• formatting

  darioush

  committed
  f38d9d5

• try

  darioush

  committed
  ee5080c

• update lastfile

  darioush

  committed
  48e37e0

• require again

  darioush

  committed
  0f870d9

• measure state size

  darioush

  committed
  79f5265

• fix

  darioush

  committed
  5b1b6c1

• term str

  darioush

  committed
  d6562f1

• try

  darioush

  committed
  c0dd1d9

• try

  darioush

  committed
  dfac973

• try

  darioush

  committed
  63135e7

• move legend

  darioush

  committed
  3643deb

• oops

  darioush

  committed
  2c6097c

• nits

  darioush

  committed
  32a97be

• fix

  darioush

  committed
  5081cdd

• DrainAcceptorQueue

  darioush

  committed
  c6c3c69

Commits on Dec 28, 2024

• try read cache

  darioush

  committed
  9fee652

• fix

  darioush

  committed
  fb9870b

• try

  darioush

  committed
  6296e0c

• only count first access

  darioush

  committed
  77609e5

• no need for legend

  darioush

  committed
  9ad00ba

• try

  darioush

  committed
  b0ce199

• less allocs

  darioush

  committed
  4e5c90d

• log % of state

  darioush

  committed
  abd6d17

• add details

  darioush

  committed
  42de2e0

• cmp fastcache

  darioush

  committed
  3f752f2

• add cache delete

  darioush

  committed
  8aac7d8

• try otter

  darioush

  committed
  4d1eb47

• reduce logs

  darioush

  committed
  3cda736

• try to figure out write recency

  darioush

  committed
  bc1899a

• keep deletes

  darioush

  committed
  5a5ab62

• count 0 -> 0 as "update" (vs deletion or addition

  darioush

  committed
  31ca4bb

• more logging

  darioush

  committed
  3969977

• stats

  darioush

  committed
  678ba40

• per tx

  darioush

  committed
  25d91ef

• log

  darioush

  committed
  22c1190

• try hst

  darioush

  committed
  280ecc0

• try

  darioush

  committed
  f0d5a66

• improve log

  darioush

  committed
  edc8f28

• try

  darioush

  committed
  44b2440

• try

  darioush

  committed
  b05e69b

• two hist

  darioush

  committed
  387a327

• oops

  darioush

  committed
  4fdcbac

• bump inf

  darioush

  committed
  2f57196

• try fix

  darioush

  committed
  b3028ae

• formatting

  darioush

  committed
  7b221ae

• fix

  darioush

  committed
  01523a8

• fix

  darioush

  committed
  cbdb568

• try

  darioush

  committed
  ea6b5bd

Commits on Dec 31, 2024

• try bufio

  darioush

  committed
  5718b88

• try

  darioush

  committed
  c7a4569

• try

  darioush

  committed
  b5616d1

• try

  darioush

  committed
  8590c8e

• try again

  darioush

  committed
  67acfe0

Commits on Jan 1, 2025

• try

  darioush

  committed
  5ee4152

• try

  darioush

  committed
  605aec2

• try

  darioush

  committed
  d4f7f9c

• try

  darioush

  committed
  69908df

• try

  darioush

  committed
  fd758a9

Commits on Jan 2, 2025

• basic ipc backend

  darioush

  committed
  41678e1

• remove verbose logs

  darioush

  committed
  cc48c83

• logEach for reprocess

  darioush

  committed
  0c3425d

• change

  darioush

  committed
  92e4e2c

Commits on Jan 3, 2025

• add writes from cache evicts

  darioush

  committed
  166b528

• try

  darioush

  committed
  4efc8c8

• fix

  darioush

  committed
  40d9078

• track metadata

  darioush

  committed
  cdec960

• fix start

  darioush

  committed
  e2fd0a5

• skip blocks

  darioush

  committed
  f837646

• try

  darioush

  committed
  ff6fc88

• try

  darioush

  committed
  666d506

• improve skip log

  darioush

  committed
  5edea85

• try

  darioush

  committed
  0f96133

• fix

  darioush

  committed
  cb7bdb5

• try

  darioush

  committed
  235a531

• add TestQueryBlock

  darioush

  committed
  e87059f

• try

  darioush

  committed
  7849e57

• try

  darioush

  committed
  61121ad

• try

  darioush

  committed
  9fb65dc

• try

  darioush

  committed
  6c99c19

• Merge branch 'master' of github.com:ava-labs/coreth into prefetch-state

  darioush

  committed
  10a2556

• fix

  darioush

  committed
  afce6e4

• check genesis

  darioush

  committed
  03838e2

• verbose

  darioush

  committed
  9c3c69a

• try

  darioush

  committed
  1655aa9

• try

  darioush

  committed
  12bed94

• log genesis

  darioush

  committed
  860b590

• try

  darioush

  committed
  c3cacaa

• too verbose

  darioush

  committed
  4b918c5

• nit

  darioush

  committed
  59c726c

• patch

  darioush

  committed
  61fcaa7

• only send unique updates to underlying storage

  darioush

  committed
  849269f

• formatting

  darioush

  committed
  73dc9ce

• ms

  darioush

  committed
  0c5f2b4

• update formatting for negative state growth

  darioush

  committed
  95a2bfb

• allow creartion of continuable db through pp

  darioush

  committed
  eb5df37

• handle genesis + write batch

  darioush

  committed
  13564f4

• try

  darioush

  committed
  3769b9b

• try

  darioush

  committed
  f2bf587

• try

  darioush

  committed
  0094036

• try

  darioush

  committed
  0f1e164

• snapshot

  darioush

  committed
  e230478

• Merge branch 'master' of github.com:ava-labs/coreth into prefetch-state

  darioush

  committed
  b9ae451

• add code export support

  darioush

  committed
  ec47722

• detect root divergance

  darioush

  committed
  b655494

• add log

  darioush

  committed
  51b20ff

• add iteration

  darioush

  committed
  f6ff6f1

• try

  darioush

  committed
  8498356

• try

  darioush

  committed
  73ea7e6

• try

  darioush

  committed
  0ff0a5b

• try

  darioush

  committed
  a006119

• try

  darioush

  committed
  47a4176

• try

  darioush

  committed
  f90888d

• try

  darioush

  committed
  946afa9

Commits on Jan 4, 2025

• oops

  darioush

  committed
  c1fcee6

• verify expectation

  darioush

  committed
  e158f03

• check

  darioush

  committed
  aa78bdb

• try

  darioush

  committed
  470c707

• try

  darioush

  committed
  e2b6679

• try

  darioush

  committed
  fe552dd

• remove log

  darioush

  committed
  fb80d8a

• try handle deletes

  darioush

  committed
  46e0d05

• preserve order

  darioush

  committed
  078a355

• try

  darioush

  committed
  c8e9a7b

• try

  darioush

  committed
  013c23a

• try

  darioush

  committed
  ad035d3

• try

  darioush

  committed
  4338bdc

• fix

  darioush

  committed
  9a8d485

• log ethdb prefix

  darioush

  committed
  ad58a06

• try

  darioush

  committed
  ebc817e

• fix

  darioush

  committed
  ea59141

• try

  darioush

  committed
  e4cae09

• ```oops

  darioush

  committed
  d63d2b9

• try

  darioush

  committed
  7b0efa8

• try

  darioush

  committed
  975b747

• try

  darioush

  committed
  2a60161

• try storage befor accounts

  darioush

  committed
  281139c

• undo logs

  darioush

  committed
  cbe40c2

• remove logs

  darioush

  committed
  6f0e942

Commits on Jan 7, 2025

• add option to continue from hash db

  darioush

  committed
  c0d0b14

• prefix option

  darioush

  committed
  b579c23

• try

  darioush

  committed
  e248b94

• try

  darioush

  committed
  7618b6c

• fix

  darioush

  committed
  5b60dd6