-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
remove all spécial caracters from the sentence #97
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You aren't considering all languages. For example you are removing valid characters in Greek.
i will move the filter to the "fr" file |
import re | ||
|
||
# All special characters | ||
FILTER_SYMBOLES_REG = re.compile( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While FILTER_SYMBOLES_REG
seems useful what might be better is a white list instead of a black list.
For example, the characters 男性 are not in your black list and would pass through un-altered.
No description provided.