-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: shrink allocations when setting new elements #90
Conversation
`bytes.Buffer` will often over-allocate. Here, we copy to an exact-sized buffer to avoid this. We'll pay some extra copying cost, but we can at least avoid the allocation overhead with a buffer pool in most cases.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #90 +/- ##
==========================================
+ Coverage 52.94% 53.32% +0.38%
==========================================
Files 13 13
Lines 1309 1324 +15
==========================================
+ Hits 693 706 +13
- Misses 464 465 +1
- Partials 152 153 +1
|
}, | ||
} | ||
|
||
func cborToBytes(val cbg.CBORMarshaler) ([]byte, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, checking my understanding: I think what we're doing here is (1) using bytes.Buffer
to do efficient allocation of bytes when we don't know up-front what we'll need, (2) using a pool so we're not going to suffer from the consistent over-allocation bytes.Buffer
does every time we use one, and (3) copy the bytes into a fixed length slice so we end up with precisely what we need and can reuse the bytes.Buffer
without concern.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exactly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this looks good to me, and it works, and is faster for the heavy migration work, which is unexpected I think but good news
Co-authored-by: Rod Vagg <[email protected]>
Co-authored-by: Masih H. Derkani <[email protected]>
to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90
* deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi <[email protected]> Co-authored-by: Steven Allen <[email protected]>
* deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]>
* deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]>
* Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <[email protected]> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <[email protected]> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * #11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from #11723 Closes: #11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <[email protected]> Co-authored-by: parthshah1 <[email protected]> Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: ZenGround0 <[email protected]> Co-authored-by: zenground0 <[email protected]> Co-authored-by: forcedebug <[email protected]> Co-authored-by: Jiaying Wang <[email protected]> Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: Masih H. Derkani <[email protected]> Co-authored-by: Lee <[email protected]> Co-authored-by: Andrew Jackson (Ajax) <[email protected]> Co-authored-by: beck <[email protected]> Co-authored-by: 0x5459 <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Aarsh Shah <[email protected]>
* fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * 1 * relatable * add and delete layer * chore: curio: merge master (#11956) * Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <[email protected]> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <[email protected]> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * #11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from #11723 Closes: #11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <[email protected]> Co-authored-by: parthshah1 <[email protected]> Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: ZenGround0 <[email protected]> Co-authored-by: zenground0 <[email protected]> Co-authored-by: forcedebug <[email protected]> Co-authored-by: Jiaying Wang <[email protected]> Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: Masih H. Derkani <[email protected]> Co-authored-by: Lee <[email protected]> Co-authored-by: Andrew Jackson (Ajax) <[email protected]> Co-authored-by: beck <[email protected]> Co-authored-by: 0x5459 <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Aarsh Shah <[email protected]> * linter oops * gen cleanup * fix * named returns are confusing --------- Signed-off-by: forcedebug <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: LexLuthr <[email protected]> Co-authored-by: parthshah1 <[email protected]> Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: ZenGround0 <[email protected]> Co-authored-by: zenground0 <[email protected]> Co-authored-by: forcedebug <[email protected]> Co-authored-by: Jiaying Wang <[email protected]> Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: Masih H. Derkani <[email protected]> Co-authored-by: Lee <[email protected]> Co-authored-by: beck <[email protected]> Co-authored-by: 0x5459 <[email protected]> Co-authored-by: Aarsh Shah <[email protected]>
* Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <[email protected]> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <[email protected]> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * filecoin-project/lotus#11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from filecoin-project/lotus#11723 Closes: filecoin-project/lotus#11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <[email protected]> Co-authored-by: parthshah1 <[email protected]> Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: ZenGround0 <[email protected]> Co-authored-by: zenground0 <[email protected]> Co-authored-by: forcedebug <[email protected]> Co-authored-by: Jiaying Wang <[email protected]> Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: Masih H. Derkani <[email protected]> Co-authored-by: Lee <[email protected]> Co-authored-by: Andrew Jackson (Ajax) <[email protected]> Co-authored-by: beck <[email protected]> Co-authored-by: 0x5459 <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Aarsh Shah <[email protected]>
* Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <[email protected]> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <[email protected]> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * filecoin-project/lotus#11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from filecoin-project/lotus#11723 Closes: filecoin-project/lotus#11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <[email protected]> Co-authored-by: parthshah1 <[email protected]> Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: ZenGround0 <[email protected]> Co-authored-by: zenground0 <[email protected]> Co-authored-by: forcedebug <[email protected]> Co-authored-by: Jiaying Wang <[email protected]> Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: Masih H. Derkani <[email protected]> Co-authored-by: Lee <[email protected]> Co-authored-by: Andrew Jackson (Ajax) <[email protected]> Co-authored-by: beck <[email protected]> Co-authored-by: 0x5459 <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Aarsh Shah <[email protected]>
* fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * 1 * relatable * add and delete layer * chore: curio: merge master (#11956) * Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <[email protected]> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <[email protected]> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * filecoin-project/lotus#11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from filecoin-project/lotus#11723 Closes: filecoin-project/lotus#11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <[email protected]> Co-authored-by: parthshah1 <[email protected]> Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: ZenGround0 <[email protected]> Co-authored-by: zenground0 <[email protected]> Co-authored-by: forcedebug <[email protected]> Co-authored-by: Jiaying Wang <[email protected]> Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: Masih H. Derkani <[email protected]> Co-authored-by: Lee <[email protected]> Co-authored-by: Andrew Jackson (Ajax) <[email protected]> Co-authored-by: beck <[email protected]> Co-authored-by: 0x5459 <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Aarsh Shah <[email protected]> * linter oops * gen cleanup * fix * named returns are confusing --------- Signed-off-by: forcedebug <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: LexLuthr <[email protected]> Co-authored-by: parthshah1 <[email protected]> Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: ZenGround0 <[email protected]> Co-authored-by: zenground0 <[email protected]> Co-authored-by: forcedebug <[email protected]> Co-authored-by: Jiaying Wang <[email protected]> Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: Masih H. Derkani <[email protected]> Co-authored-by: Lee <[email protected]> Co-authored-by: beck <[email protected]> Co-authored-by: 0x5459 <[email protected]> Co-authored-by: Aarsh Shah <[email protected]>
* Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <[email protected]> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <[email protected]> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * filecoin-project/lotus#11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from filecoin-project/lotus#11723 Closes: filecoin-project/lotus#11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <[email protected]> Co-authored-by: parthshah1 <[email protected]> Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: ZenGround0 <[email protected]> Co-authored-by: zenground0 <[email protected]> Co-authored-by: forcedebug <[email protected]> Co-authored-by: Jiaying Wang <[email protected]> Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: Masih H. Derkani <[email protected]> Co-authored-by: Lee <[email protected]> Co-authored-by: Andrew Jackson (Ajax) <[email protected]> Co-authored-by: beck <[email protected]> Co-authored-by: 0x5459 <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Aarsh Shah <[email protected]>
* release: v1.26.3 (#11908) (#11915) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> * build: release: v1.27.0-rc1 (#11947) * chore: Set version as v1.27.0-rc1 Set version as v1.27.0-rc1, run make gen & make docsgen-cli * Update changelog Update changelog * Update changelog Update changelog based on feedback * Bump pubsub-dep Bump pubsub-dep * Prep v1.27.0-rc2 Prep v1.27.0-rc2 * Typo fixes, and more changelog updates Typo fixes, and more changelog updates * chore: remove unmaintained bootstrappers (#11983) * chore: remove unmaintained bootstrappers chore: remove unmaintained bootstrappers * Update mainnet.pi fixing typoed domain fixing typo for 1475.io 'bootstarp' -> 'bootstrap' * Update mainnet.pi apparently the actual hostname is typoed. so bootstarp it is. --------- Co-authored-by: smagdali <[email protected]> * chore: update go-data-transfer and go-graphsync * add ETH addrs API to Gateway (#11979) * fix: copy Flags field from SectorOnChainInfo Fixes: #11962 * feat: libp2p: Lotus stream cleanup (#11993) * set stream deadlines in Lotus * reduce timeout * whitelist bootstrappers * fix tests * Update changelog and version Update changelog and version * ci: deprecate circle ci in favour of github actions (#11786) * Update changelog Update changelog with the deprecate circle-ci * chore: update drand (#12021) * Update changelog / make docsgen Update changelog / make docsgen * chore: lint: update golangci lint config * remove and replace some linters * remove some exclusions * make all exclusions more explicit matches * chore: lint: fix lint errors with new linting config Ref: #11967 * chore: lint: address feedback from reviews * doc: eth: restore comment lost in linter cleanup Ref: #11968 * chore: libp2p: update to v0.34.1 (#12027) * update libp2p to v0.34.0 * fix libp2p err * fix imports * update go mod * update go mod * Update changelog Update changelog * go mod tidy go mod tidy * revert go version change (#12050) * Update changelog Update changelog * chore: backport #12054 to release/v1.27.0 branch (#12056) * chore: pin golanglint-ci to v1.58.2 (#12054) Fixes: #12044 * Add backport to changelog Add backport to changelog --------- Co-authored-by: Rod Vagg <[email protected]> * Fix where #12054 is placed in changelog Fix where #12054 is placed in changelog * Add trailing line in Makefile Add trailing line in Makefile * Go mod tidy Go mod tidy --------- Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: smagdali <[email protected]> Co-authored-by: Aarsh Shah <[email protected]> Co-authored-by: Piotr Galar <[email protected]>
* release: v1.26.3 (#11908) (#11915) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> * build: release: v1.27.0-rc1 (#11947) * chore: Set version as v1.27.0-rc1 Set version as v1.27.0-rc1, run make gen & make docsgen-cli * Update changelog Update changelog * Update changelog Update changelog based on feedback * Bump pubsub-dep Bump pubsub-dep * Prep v1.27.0-rc2 Prep v1.27.0-rc2 * Typo fixes, and more changelog updates Typo fixes, and more changelog updates * chore: remove unmaintained bootstrappers (#11983) * chore: remove unmaintained bootstrappers chore: remove unmaintained bootstrappers * Update mainnet.pi fixing typoed domain fixing typo for 1475.io 'bootstarp' -> 'bootstrap' * Update mainnet.pi apparently the actual hostname is typoed. so bootstarp it is. --------- Co-authored-by: smagdali <[email protected]> * chore: update go-data-transfer and go-graphsync * add ETH addrs API to Gateway (#11979) * fix: copy Flags field from SectorOnChainInfo Fixes: #11962 * feat: libp2p: Lotus stream cleanup (#11993) * set stream deadlines in Lotus * reduce timeout * whitelist bootstrappers * fix tests * Update changelog and version Update changelog and version * ci: deprecate circle ci in favour of github actions (#11786) * Update changelog Update changelog with the deprecate circle-ci * chore: update drand (#12021) * Update changelog / make docsgen Update changelog / make docsgen * chore: lint: update golangci lint config * remove and replace some linters * remove some exclusions * make all exclusions more explicit matches * chore: lint: fix lint errors with new linting config Ref: #11967 * chore: lint: address feedback from reviews * doc: eth: restore comment lost in linter cleanup Ref: #11968 * chore: libp2p: update to v0.34.1 (#12027) * update libp2p to v0.34.0 * fix libp2p err * fix imports * update go mod * update go mod * Update changelog Update changelog * go mod tidy go mod tidy * revert go version change (#12050) * Update changelog Update changelog * chore: backport #12054 to release/v1.27.0 branch (#12056) * chore: pin golanglint-ci to v1.58.2 (#12054) Fixes: #12044 * Add backport to changelog Add backport to changelog --------- Co-authored-by: Rod Vagg <[email protected]> * Bump version - make gen/make docsgen Bump version - make gen/make docsgen * Update changelog Update changelog * Bump NodeBuildVersion to v1.27.1-rc1 Bump NodeBuildVersion to v1.27.1-rc1 * Add Lotus-Miner / Curio related changes Add Lotus-Miner / Curio related changes * Update date and upgrade warnings Update date and upgrade warnings * fix: ci: do not use deprecated --debug goreleaser flag (#12086) * chore: deals: remove forgotten graphsync references (#12084) * chore: types: remove more items forgotten after markets (#12095) * chore: cleanup: remove more items forgotten after markets * .gz somehow reappeared after #11625 * fix: ETH RPC API: ETH Call should use the parent state root of the subsequent tipset (#11905) * fix eth call * tests * changes as per review * changes as per review * Update node/impl/full/eth.go Co-authored-by: Rod Vagg <[email protected]> * fix as per review --------- Co-authored-by: Rod Vagg <[email protected]> * Update changelog to RC2 Update changelog to RC2 * Make gen / make docsgen-cli Make gen / make docsgen-cli * chore: api: the Net API/CLI now remains only on daemon The only part of this repository that does lp2p is now lotus-daemon Remove the CommonNet type, used exclusively bu the CLI stack Adjust the rest of struct-memebership to match what went where End result best seen in diff of `documentation/en/api-v0-methods-miner.md` * Update changelog Update changelog * fix: events: sqlite db improvements (#12090) * fix: events: sqlite db improvements * fix unclosed multi-row query * tune options to limit wal growth Ref: #12089 * fix: events: use correct context for CollectEvents transaction * fix: events: close prepared read statement * fix: events: close initial query; handle lint failures * Update CHANGELOG.md * build: release: v1.27.1-rc2 (#12101) * fix: ci: do not use deprecated --debug goreleaser flag (#12086) * chore: deals: remove forgotten graphsync references (#12084) * chore: types: remove more items forgotten after markets (#12095) * chore: cleanup: remove more items forgotten after markets * .gz somehow reappeared after #11625 * fix: ETH RPC API: ETH Call should use the parent state root of the subsequent tipset (#11905) * fix eth call * tests * changes as per review * changes as per review * Update node/impl/full/eth.go Co-authored-by: Rod Vagg <[email protected]> * fix as per review --------- Co-authored-by: Rod Vagg <[email protected]> * Update changelog to RC2 Update changelog to RC2 * Make gen / make docsgen-cli Make gen / make docsgen-cli * chore: api: the Net API/CLI now remains only on daemon The only part of this repository that does lp2p is now lotus-daemon Remove the CommonNet type, used exclusively bu the CLI stack Adjust the rest of struct-memebership to match what went where End result best seen in diff of `documentation/en/api-v0-methods-miner.md` * Update changelog Update changelog * fix: events: sqlite db improvements (#12090) * fix: events: sqlite db improvements * fix unclosed multi-row query * tune options to limit wal growth Ref: #12089 * fix: events: use correct context for CollectEvents transaction * fix: events: close prepared read statement * fix: events: close initial query; handle lint failures * Update CHANGELOG.md --------- Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: Peter Rabbitson <[email protected]> Co-authored-by: Aarsh Shah <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Peter Rabbitson <[email protected]> * small fix in changelog * fix: release: update goreleaser config file Fixes: #12120 * fix go releaser and test with rc3 * Update CHANGELOG.md * lotus v1.27.1 prep * address review - resolve one more conflicts - revert 2 new line added * doc: events: note events db migration impact --------- Co-authored-by: Phi-rjan <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: smagdali <[email protected]> Co-authored-by: Aarsh Shah <[email protected]> Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: Peter Rabbitson <[email protected]> Co-authored-by: Peter Rabbitson <[email protected]>
bytes.Buffer
will often over-allocate. Here, we copy to an exact-sized buffer to avoid this. We'll pay some extra copying cost, but we can at least avoid the allocation overhead with a buffer pool in most cases.