Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dedup utils #256

Closed
2 tasks done
ehsandeep opened this issue Sep 19, 2023 · 0 comments · Fixed by #261
Closed
2 tasks done

dedup utils #256

ehsandeep opened this issue Sep 19, 2023 · 0 comments · Fixed by #261
Assignees
@dogancanbakir
Labels
Status: Completed Nothing further to be done with this issue. Awaiting to be closed. Type: Enhancement Most issues will probably ask for additions or changes.

Comments

@ehsandeep
Copy link
Member

ehsandeep commented Sep 19, 2023 •
edited by dogancanbakir
Loading

@dogancanbakir , we initially implemented dedupe for projectdiscovery/alterx at https://github.com/projectdiscovery/alterx/blob/main/dedupe.go

in that dedupe implementation , initially all dedupe happens using map datatype upto 100MB and after this threshold is reached it switches to goleveldb ( we should also use bloom filters if not already using ) thus it's faster and also has cleanup to remove residual data

we can move that implementation to utils and use it here as well as in alterx . in followup tickets . what do you think ?

Originally posted by @tarunKoyalwar in projectdiscovery/tlsx#350 (comment)

Tasks
Beta Give feedback
  1. Reuse dedupe utils alterx#27
    Type: Enhancement
    dogancanbakir
  2. Reuse dedupe utils tlsx#362
    Type: Enhancement
    dogancanbakir
@dogancanbakir dogancanbakir self-assigned this Sep 19, 2023
@tarunKoyalwar tarunKoyalwar linked a pull request Sep 25, 2023 that will close this issue
add dedupe #261
Merged
This was referenced Sep 25, 2023
Closed
Closed
@dogancanbakir dogancanbakir added the Type: Enhancement Most issues will probably ask for additions or changes. label Sep 26, 2023
@dogancanbakir dogancanbakir mentioned this issue Sep 28, 2023
Merged
@Mzack9999 Mzack9999 added the Status: Completed Nothing further to be done with this issue. Awaiting to be closed. label Oct 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dogancanbakir dogancanbakir

Labels
Status: Completed Nothing further to be done with this issue. Awaiting to be closed. Type: Enhancement Most issues will probably ask for additions or changes.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add dedupe projectdiscovery/utils
3 participants
@ehsandeep @Mzack9999 @dogancanbakir