perf(query): Update CompressedBin IntersectionAlgo #9000
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
harshil-goel
requested review from
mangalaman93,
meghalims,
all-seeing-code,
billprovince and
joshua-goldstein
as code owners
dgraph-bot
added
area/testing
jairad26
reviewed
Sep 29, 2023
algo/uidlist.go
Outdated
| @@ -60,7 +60,7 @@ func IntersectCompressedWith(pack *pb.UidPack, afterUID uint64, v, o *pb.List) { | |||
|
|
|||
| // Select appropriate function based on heuristics. | |||
| ratio := float64(m) / float64(n) | |||
| if ratio < 500 { | |||
| if ratio < 10 { | |||
There was a problem hiding this comment.
set hardcoded variables to a const
algo/uidlist.go
Outdated
| return q[idx] >= lastUid | ||
| }) | ||
| if qidx >= len(q) { | ||
| if ld*10 < len(q) { |
There was a problem hiding this comment.
why 10? if it’s based on heuristic then you can just invoke the const
algo/uidlist.go
Outdated
| if len(uids) == 0 || u > uids[len(uids)-1] { | ||
| uids = dec.Seek(u, codec.SeekStart) | ||
| if lq*10 < ld { |
mangalaman93
requested changes
Oct 11, 2023
codec/codec.go
Outdated
| prevBlockIdx = 0 | ||
| } | ||
|
|
||
| pack := d.Pack |
codec/codec.go
Outdated
| @@ -223,6 +223,64 @@ func (d *Decoder) ApproxLen() int { | |||
|
|
|||
| type searchFunc func(int) bool | |||
|
|
|||
| // SeekToBlock will find the nearest block, and unpack it. Unlike Seek, it doesn't | |||
There was a problem hiding this comment.
we can improve this explanation
| if len(uids) == 0 || u > uids[len(uids)-1] { | ||
| uids = dec.Seek(u, codec.SeekStart) | ||
| if lq*linVsBinRatio < ld { |
There was a problem hiding this comment.
I tried updating the condition, but it made performance much worse. I am guessing its because that the decision to do binary or linear really depends on the total size of the arrays. If the array ratio is too high, then the numbers would be far apart.
mangalaman93
approved these changes
Oct 13, 2023
shivaji-kharse
pushed a commit
that referenced
this pull request
Mar 12, 2024
Updated algo to intersect. Getting upto 175% improvment overall
```
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/dgraph/algo
cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
│ in1 │ main_in │
│ sec/op │ sec/op vs base │
ListIntersectCompressBin/compressed:IntersectWith:ratio=0.01:size=100:overlap=0.01:-8 91.21n ± ∞ ¹ 387.70n ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=0.1:size=100:overlap=0.01:-8 336.0n ± ∞ ¹ 1733.0n ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=1:size=100:overlap=0.01:-8 1.093µ ± ∞ ¹ 3.481µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=10:size=100:overlap=0.01:-8 4.719µ ± ∞ ¹ 8.600µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=100:size=100:overlap=0.01:-8 19.76µ ± ∞ ¹ 44.32µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=0.01:size=1000:overlap=0.01:-8 481.7n ± ∞ ¹ 784.2n ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=0.1:size=1000:overlap=0.01:-8 1.502µ ± ∞ ¹ 3.667µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=1:size=1000:overlap=0.01:-8 5.454µ ± ∞ ¹ 36.977µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=10:size=1000:overlap=0.01:-8 33.14µ ± ∞ ¹ 70.73µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=100:size=1000:overlap=0.01:-8 179.8µ ± ∞ ¹ 485.1µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=0.01:size=10000:overlap=0.01:-8 3.214µ ± ∞ ¹ 4.459µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=0.1:size=10000:overlap=0.01:-8 12.00µ ± ∞ ¹ 32.78µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=1:size=10000:overlap=0.01:-8 76.90µ ± ∞ ¹ 356.05µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=10:size=10000:overlap=0.01:-8 371.6µ ± ∞ ¹ 923.1µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=100:size=10000:overlap=0.01:-8 1.894m ± ∞ ¹ 5.141m ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=0.01:size=100000:overlap=0.01:-8 48.13µ ± ∞ ¹ 60.59µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=0.1:size=100000:overlap=0.01:-8 146.7µ ± ∞ ¹ 611.9µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=1:size=100000:overlap=0.01:-8 943.7µ ± ∞ ¹ 3359.6µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=10:size=100000:overlap=0.01:-8 3.926m ± ∞ ¹ 8.849m ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=100:size=100000:overlap=0.01:-8 20.78m ± ∞ ¹
ListIntersectCompressBin/compressed:IntersectWith:ratio=0.01:size=1000000:overlap=0.01:-8 862.4µ ± ∞ ¹ 1142.5µ ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=0.1:size=1000000:overlap=0.01:-8 1.599m ± ∞ ¹ 7.878m ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=1:size=1000000:overlap=0.01:-8 9.791m ± ∞ ¹ 33.871m ± ∞ ¹ ~ (p=1.000 n=1) ²
ListIntersectCompressBin/compressed:IntersectWith:ratio=10:size=1000000:overlap=0.01:-8 44.40m ± ∞ ¹
geomean 66.19µ 104.6µ +175.93% ³
```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updated algo to intersect. Getting upto 175% improvment overall