Code for Deduplication

[qtli]

Hi, thanks so much for your promising work!
I was hoping to inquire if it's possible for you to provide me with the code for the "Deduplication" section. Thank you in advance for your help!

[zachschillaci27]

Hi, thanks so much for your promising work!

I was hoping to inquire if it's possible for you to provide me with the code for the "Deduplication" section. Thank you in advance for your help!

Plus one! I'm thinking of implementing something similar myself. The similarity based deduplication with embeddings seems straightforward enough, but I'd like to see their minhash implementation.

[zhangzx-uiuc]

Also, there is another concern. The embedding-based similarity approach seems to be (n^2) complexity. Does it take a lot of time in practice when the number of personas scales to 1 billion?