Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bring herald/db/prefixes.go up to date with new tables #46

Closed
moodyjon opened this issue Aug 3, 2022 · 3 comments · Fixed by #49 or #51
Closed

Bring herald/db/prefixes.go up to date with new tables #46

moodyjon opened this issue Aug 3, 2022 · 3 comments · Fixed by #49 or #51
Assignees

Comments

@moodyjon
Copy link
Contributor

moodyjon commented Aug 3, 2022

Latest tables in (python) hub:

https://github.com/lbryio/hub/blob/4845b9224842e6800808aa1860330b34929b06ae/hub/db/common.py#L50

    channel_count = b'Z'
    support_amount = b'a'
    block_tx = b'b'
    trending_notifications = b'c'
    mempool_tx = b'd'
    touched_hashX = b'e'
    hashX_status = b'f'
    hashX_mempool_status = b'g'

Latest tables in herald.go:

BlockTXs = 'b'

	ChannelCount  = 'Z'
	SupportAmount = 'a'
	BlockTXs      = 'b'
@moodyjon moodyjon self-assigned this Aug 3, 2022
@moodyjon
Copy link
Contributor Author

moodyjon commented Aug 4, 2022

After writing some Pack/Unpack() code for one of the new tables, I'm searching for something that can automate this. Using reflect to make the Pack/Unpack() generic is not a new idea. Found these packages that do it:

https://pkg.go.dev/github.com/lunixbochs/struc
https://pkg.go.dev/github.com/go-restruct/restruct

Performance benchmark from 7 years ago indicated it was 10x slower than manually written code.
https://github.com/lunixbochs/struc#benchmark

@jeffreypicard
Copy link
Collaborator

After writing some Pack/Unpack() code for one of the new tables, I'm searching for something that can automate this. Using reflect to make the Pack/Unpack() generic is not a new idea. Found these packages that do it:

https://pkg.go.dev/github.com/lunixbochs/struc https://pkg.go.dev/github.com/go-restruct/restruct

Performance benchmark from 7 years ago indicated it was 10x slower than manually written code. https://github.com/lunixbochs/struc#benchmark

The plan is to use generics from go 1.18 for this https://go.dev/blog/why-generics. If we can get performance improvements out of this tangentially to that, it sounds good. We should talk about this in the call.

@moodyjon
Copy link
Contributor Author

moodyjon commented Aug 5, 2022

I have successfully tagged 2 existing structs (EffectiveAmountKey/Value) structs and ran Pack()/Unpack() based on https://pkg.go.dev/github.com/go-restruct/restruct and reflection.

I also set up a testing framework where the different Pack/Unpack() implementations can be tested against each other. For example, manual Pack() with "restruct" Unpack(), and vice versa.

TODO: Need to gather some test data for the new tables and finish the manually-implemented Pack/Unpack() for them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants