Code

add allow list to tokenizer#148

Code

Merged

bors[bot] merged 22 commits intomeilisearch:mainmeilisearch/charabia:mainfrom yenwel:allowlisttokenizeryenwel/charabia:allowlisttokenizerCopy head branch name to clipboardOct 12, 2022

+103-32

Commits

Commits on Oct 5, 2022

• add allowlist parameter to detection and implement it

  yenwel

  committed
  37f6c5b

  Copy full SHA for 37f6c5b

Commits on Oct 6, 2022

• implement segment methods with allowlist

  yenwel

  committed
  64b606c

  Copy full SHA for 64b606c

• documentation

  yenwel

  committed
  fbcd762

  Copy full SHA for fbcd762

• implement allow list in tokenizer

  yenwel

  committed
  9aea8c6

  Copy full SHA for 9aea8c6

Commits on Oct 7, 2022

• doc and test for allowlist

  yenwel

  committed
  0f7b71e

  Copy full SHA for 0f7b71e

• add allowlist parameter to detection and implement it

  yenwel

  committed
  69dc5ee

  Copy full SHA for 69dc5ee

• implement segment methods with allowlist

  yenwel

  committed
  8fc3255

  Copy full SHA for 8fc3255

• documentation

  yenwel

  committed
  dd31dff

  Copy full SHA for dd31dff

• implement allow list in tokenizer

  yenwel

  committed
  6b6d020

  Copy full SHA for 6b6d020

• doc and test for allowlist

  yenwel

  committed
  e78c834

  Copy full SHA for e78c834

• :qMerge branch 'allowlisttokenizer' of https://github.com/yenwel/charabia into allowlisttokenizer

  yenwel

  committed
  148e963

  Copy full SHA for 148e963

Commits on Oct 11, 2022

• Update src/tokenizer.rs

  Show description for edc373c
  

  yenwel

  and

  ManyTheFish

  authored
  edc373c

  Copy full SHA for edc373c

• Update src/tokenizer.rs

  Show description for 2e569ed
  

  yenwel

  and

  ManyTheFish

  authored
  2e569ed

  Copy full SHA for 2e569ed

• Update src/detection/mod.rs

  Show description for 61c450f
  

  yenwel

  and

  ManyTheFish

  authored
  61c450f

  Copy full SHA for 61c450f

• Update src/detection/mod.rs

  Show description for 542be81
  

  yenwel

  and

  ManyTheFish

  authored
  542be81

  Copy full SHA for 542be81

• Update src/detection/mod.rs

  Show description for 5a9e674
  

  yenwel

  and

  ManyTheFish

  authored
  5a9e674

  Copy full SHA for 5a9e674

• fix lifetimes and unwrapping allowlist for detection

  yenwel

  committed
  eedadf8

  Copy full SHA for eedadf8

• fix lifetime subtyping

  yenwel

  committed
  32731ec

  Copy full SHA for 32731ec

• Update src/detection/mod.rs

  Show description for a85dfed
  

  yenwel

  and

  ManyTheFish

  authored
  a85dfed

  Copy full SHA for a85dfed

• Update src/detection/mod.rs

  Show description for 24460cc
  

  yenwel

  and

  ManyTheFish

  authored
  24460cc

  Copy full SHA for 24460cc

• clean up lifetimes detection

  yenwel

  committed
  0ae9735

  Copy full SHA for 0ae9735

• clean up lifetime detection part deux

  yenwel

  committed
  1b929e4

  Copy full SHA for 1b929e4