This repository has been archived by the owner on Oct 31, 2023. It is now read-only.
Question about Wikipedia Corpus Preprocessing #222
Comments
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Hi,
In this thread: #42, it was mentioned that the pages were split into 100 word passages using spaCy en-web tokenizer. I tried to reproduce this myself, counting if a token was a word using is_alpha from spaCy, but my passages on average were slightly longer than the Wikipedia DPR 100 word passages. Could you elaborate on how the tokenizer was used to count 100 words please?
The text was updated successfully, but these errors were encountered: