is_oov and prob does not work for en_core_web_sm model

[lkluo]

How to reproduce the behaviour

lex = nlp.vocab[u"dog"] print(lex.is_oov, lex.prob)
producing
True -20.0

This bug had been reported #1204.

Your Environment

• Operating System: masOS
• Python Version Used: 3.6
• spaCy Version Used: 2.1.4
• Environment Information:

[BreakBB]

"dog" is really not included in "en_core_web_sm" as you can see if you iterate the StringStore:

import spacy
nlp = spacy.load("en_core_web_sm")

has_dog = False
for s in nlp.vocab.strings:
  if s == "dog":
    has_dog = True
print("has_dog:", str(has_dog))  # has_dog: False

or simply use __contains__:

import spacy
nlp = spacy.load("en_core_web_sm")

if u"dog" in nlp.vocab.strings:
  print("'dog' is in the StringStore")
else:
  print("'dog' is not in the StringStore")

Therefore is_oov has to be True.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.