Cleaning transcript files invalid label when building trie

[JRMeyer]

>>> DJ-Hay
[January 20, 2018, 1:41am]

Hello,

Similar to issue [ slash #1083] on Github, I encountered

Invalid label
Aborted (core dumped)

when running

path/to/folder/generate_trie alphabet.txt transcript.binary transcript.txt trie

I fixed the above error message for many symbols and characters I didn't
think were in the transcript by simply adding them to my alphabet.txt
file. This is a temporary solution before I go back and figure out where
the symbols are.

However, now I've encountered the following:

Invalid label
Aborted (core dumped)

when running

path/to/folder/generate_trie alphabet.txt transcript.binary transcript.txt trie

In other words, it seems like the invalid label is empty. I added an
'empty' to alphabet.txt by just creating a line with two spaces and
nothing else, but that didn't seem to help.

It seems like the problem is my transcript has some unknown character
messing up the alphabet.h file (seen here at
https://github.com/mozilla/DeepSpeech/blob/master/native_client/alphabet.h)
at line 50.

Any suggestions as to how I can debug this on my end?

[This is an archived TTS discussion thread from discourse.mozilla.org/t/cleaning-transcript-files-invalid-label-when-building-trie]

[JRMeyer]

>>> DJ-Hay
[January 20, 2018, 6:49pm]

I debugged it. Turns out an easier solution to trying to get print
statements to work is to just delete half of the transcript in an
iterative fashion until you find the offending line of code.

The odd symbol turned out to be some weird encoded period that didn't
work (so it showed up as a square box with some letters/numbers in it).

[Archived Post]

[JRMeyer]

>>> shanehmi
[January 20, 2018, 6:50pm]

dear i have some issue when i using my mozilla, some time different
sites not working properly, like facebook and other social sites, even
when i try to open other different sites like
Techriation mozilla could not open these
sites in first attempt. kindly tel me the solution? slash
should i re-install my mozilla or not???

[Archived Post]

[JRMeyer]

>>> DJ-Hay
[January 20, 2018, 6:54pm]

Hello Dear,

This forum is particularly for DeepSpeech, a Mozilla-made deep neural
network for speech recognition.

However, reinstalling Firefox should be simple enough. Perhaps you could
also explain what was wrong with the sites, your computer's OS, and how
you are installing Firefox?

[Archived Post]

[JRMeyer]

>>> phanthanhlong7695
[January 26, 2018, 10:06am]

what should i do to create trie -.- slash
help me

[Archived Post]

[JRMeyer]

>>> p.holetzky
[January 26, 2018, 11:30am]

This
tutorial
has a section that describes the TRIE creation. What part are you stuck
at?

Short version: you use the generate_trie for trie generation in the
native_client folder of the DeepSpeech project, using alphabet.txt,
lm.binary, vocabulary.txt as input.

You can download Mozillas
Model
to get an alphabet.txt file for reference. slash
lm.binary is built with Kenlm tools. slash
Vocabulary.txt is a file with your wave file transcripts.

(According to my understanding)

[Archived Post]

[JRMeyer]

>>> phanthanhlong7695
[January 27, 2018, 4:24am]

i know. but wen i run : slash
./generate_trie data/alphabet.txt data/lm.binary data/vocabulary.txt
data/trie

= slash > error:

Invalid label K

Aborted (core dumped)

help me

[Archived Post]

[JRMeyer]

>>> DJ-Hay
[January 27, 2018, 5:04am]

Hello,

I believe your problem is the capital letter 'K'. The DeepSpeech data
only recognizes the lowercase letters ( a to z ) , an apostrophe ( ' ),
and spaces ( ). Thus, you need to remove the capital K from your
vocabulary.txt file. There are likely other problems as well, so make
sure nothing in your vocabulary.txt file is not a-z, slash ', or a space.

I actually used the slash
Invalid label: <thing> slash
error report to help me figure out which unaccepted symbols were in my
vocabulary.txt. From there, I went into a scripting language and changed
all of the unaccepted symbols to acceptable symbols (changed K to k,
change + to plus, etc).

[Archived Post]

[JRMeyer]

>>> phanthanhlong7695
[January 27, 2018, 8:05am]

Thank for your help .
![:smiley:](

Vào 27 Th1, 2018 12:07 PM, 'DJ-Hay' <discoursemozilla-community.org> đã
viết:

[Archived Post]

[JRMeyer]

>>> phanthanhlong7695
[January 29, 2018, 3:38am]

can i use that for Vietnamese ? slash
it still error. slash
Invalid label

Aborted (core dumped)

[Archived Post]

[JRMeyer]

>>> DJ-Hay
[January 29, 2018, 4:24am]

Hello, slash
So whenever it says

Invalid label

it means there is still some non a-z, slash ', or space symbol in your
transcript. You indicate Vietnamese, so I'm assuming that the code is
trying to tell you the Vietnamese symbol is invalid, but the computer
isn't easily able to display that Vietnamese symbol using the standard
print function.

[Archived Post]

[JRMeyer]

>>> p.holetzky
[January 29, 2018, 10:46pm]

I found a cleanup script somewhere in the DeepSpeech issues section and
modified a little for german alphabet. It basically goes through every
line/word/character, converts it to lower case and removes lines that
contain invalid characters. Maybe this helps you aswell, if you modify
it for your needs?

just copy the characters from your alphabet.txt into the labels string
and change the infile path to your vocabulary.txt

# -- coding: utf-8 --

infile = './vocabulary.txt'
outfile = './vocabulary_cleaned.txt'

labels = u'abcdefghijklmnopqrstuvwxyz-äöüß'

with open(infile, 'r') as fi:
with open(outfile, 'w') as fo:
for line in fi:
line_flag = True
line_str = ''
wdlist = line.strip().split()
for wd in wdlist:
wd = wd.decode('utf-8')
WD = wd.lower()
word_flag = True
for c in WD:
if not c in labels:
print(wd, WD)
word_flag = False
line_flag = False
continue
if word_flag:
line_str += (WD + ' ')
if line_flag:
fo.write(line_str.encode('utf-8') + ' slash n')

[
[[jageshmaharjan
[May 9, 2018, 7:42am]

You dont have label 'K' in your alphabets.txt file

[Archived Post]

[JRMeyer]

>>> noor_e slash _emaan11
[January 11, 2019, 9:56am]

Hi, Thank you for responding in this thread. slash
I am facing the same issue. I am working on Urdu Language. But the
default coding is for English Language. slash
I edited the vocabulary.txt and alphabet.txt as per my
dataset/Language. slash
but the error is same: Invalid label. slash
I want to know where you edited these labels, I mean in which file of
Deepspeech? slash
Thank you!

Waiting for your help.

[Archived Post]

[JRMeyer]

>>> noor_e slash _emaan11
[January 11, 2019, 11:10am]

Can you please the explain the steps of doing it? slash
Thank you!!!

[Archived Post]