Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consistent-hash: support reading by single vnode for specific states #3159

Closed
xx01cyx opened this issue Jun 13, 2022 · 5 comments · Fixed by #3628
Closed

consistent-hash: support reading by single vnode for specific states #3159

xx01cyx opened this issue Jun 13, 2022 · 5 comments · Fixed by #3628
Assignees

Comments

@xx01cyx
Copy link
Contributor

xx01cyx commented Jun 13, 2022

Currently, all states will maintain a set of vnodes, which are stored in the state's keyspace. However, every state of hash agg and hash join will only query data that belongs to one vnode. Therefore, we could narrow the range of vnodes when reading data from storage, i.e., only pass in one vnode instead of a set of vnodes.

The vnode that belongs to a state could be computed when the state is created.

@xx01cyx
Copy link
Contributor Author

xx01cyx commented Jun 13, 2022

cc. @fuyufjh @hzxa21

@skyzh
Copy link
Contributor

skyzh commented Jun 13, 2022

So we'll need to compute hash before every get and prefix scan in storage?

@xx01cyx
Copy link
Contributor Author

xx01cyx commented Jun 13, 2022

So we'll need to compute hash before every get and prefix scan in storage?

For MV, maybe yes. For other states, we could directly use the pre-computed vnode.

@hzxa21
Copy link
Collaborator

hzxa21 commented Jun 13, 2022

So we'll need to compute hash before every get and prefix scan in storage?

For MV, maybe yes. For other states, we could directly use the pre-computed vnode.

How to pre-compute the VNode of a get/prefix_scan key beforehand? We don't know the key until we need to read it.

For internal state (i.e. non-MV state), I think compure hash before every get/prefix_scan is okay since we will only hit one VNode.

@fuyufjh
Copy link
Member

fuyufjh commented Jun 13, 2022

I think this issue mainly focuses on the internal state of HashAgg & HashJoin. They both hold a bunch of ManagedStates for each join/agg key, where each of them only touches one vnode, determined by its join/agg key.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants