Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Structured logs of unit payloads can have sequence types collapsed to 'long' #351

Open
kthielen opened this issue May 6, 2020 · 0 comments

Comments

@kthielen
Copy link
Contributor

kthielen commented May 6, 2020

I was just looking at a pathological structured log that has a ton of unit payloads, and realized that we don't need to store these as a bunch of linked counts (essentially all that a batch of unit values means).

Instead we can just collapse the whole unit sequence into a number (how many of those unit values were stored) and where we correlate this data in other sequences (e.g. the transactions or log sequences) then it doesn't matter either, because we don't really store references to unit values (they encode in 0 bits, and it's always possible to make one trivially).

In some cases, this can be a significant space savings, and also a significant time savings for queries over these values (where we'd otherwise run all over memory finding all of the ints to add together).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant