add allow list to tokenizer

[yenwel]

Pull Request

Related issue

Fixes #132

What does this PR do?

• add allow list configuration to detection, segment and tokenizer
• add doc and test

PR checklist

Please check if your PR fulfills the following requirements:

• Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
• Have you read the contributing guidelines?
• Have you made sure that the title is accurate and descriptive of the changes?

[meili-bot]

This message is sent automatically

Hello @yenwel,
Thank you very much for contributing to Meilisearch ❤️.
However, the team is not available on the weekend, but they will be back on Monday 😊

[ManyTheFish]

Hello @yenwel!
I requested some changes, and they are not always easy because there are some lifetimes issues. Don't hesitate to ask if you need help! 😄

[ManyTheFish]

Ok, I'll investigate this move tomorrow! But I think we can merge your PR keeping it. 🤔

[ManyTheFish]

Looks good to me!
I let bors run the tests, then if everything passes, your PR will be merged automatically.
Thank you for your contribution despite its difficulty!

bors merge

[bors]

Build succeeded:

• tests

[meili-bot]

This message is sent automatically

Thank you for contributing to Meilisearch. If you are participating in Hacktoberfest, and you would like to receive some gift from Meilisearch too, please complete this form.