Integration of deepspeech polyglots new networks

[JRMeyer]

>>> dan.bmh
[January 25, 2021, 2:00pm]

and
,

as I already mentioned some time ago, I'm working on implementing new
STT networks using tensorflow2. slash
In the last days I did make a lot of progress and wanted to share it,
along with a request to you and a suggestion for future development
procedure.

My current network, which implements Quartznet15x5
(paper), reaches a WER of 3.7%
on LibriSpeech (test-clean) using your official English scorer. slash
The network also has much fewer parameters (19M vs 48M) and thus should
be faster in inference than the current DeepSpeech1 network. There is
also the option to use the even smaller model of Quartznet5x5 (6.7M
params), if required. I reached a WER of 4.5% with it.

You can find the pretrained networks and the source code here: slash
(Please note that the training code is still highly experimental and a
lot of features DeepSpeech had are still missing, but I hope to add most
of them in the next time). slash

GitLab

### Files · dspol · Jaco-Assistant / DeepSpeech-Polyglot

Training code and checkpoints for DeepSpeech in multiple languages

Now I would like to ask, if we could make those models usable with the
deepspeech bindings. The problem is that this will require some changes
in the native client codes, because apart from the network architecture
I also had to change the input pipeline. slash
It would be great if you could look into this and update the client
bindings accordingly. We would also need to think about a new procedure
for streaming inference, but some parts of the reference implementation
from Nvidia
(link)
should be usable for that.

Besides the request for updating the bindings, I would like to make
another suggestion, just as an idea to think about, which I think could
improve the development in future: Splitting DeepSpeech into the three
parts of usage, training and datasets. slash
We keep the github repo as main repository and entrypoint, but split out
the training part into DeepSpeech-Polyglot. This should save you a lot
of time compared to updating DeepSpeech and hopefully gives me some more
developing support. I would also give you access to the repository
then. slash
Splitting the parts of downloading and preparing the datasets into it's
own tool would make it usable for other STT projects, too, and therefore
new datasets might be added faster. I would suggest to use
corcua for that, which I did
create with the focus of making the addition of new datasets as easy as
possible (I first tried audiomate, but I found their architecture to
complicated).

What do you think about the two ideas?

Greetings slash
Daniel

slash
slash

(Notes on the above checkpoint: I did transfer the pretrained models
from Nvidia, who used pytorch, to tensorflow. While this does work well
for the network itself, I had some problems with the input pipeline. The
spectrogram+filterbanks calculation has a slightly different output in
tensorflow, which increases the WER of the transferred networks by about
1%. The problem could be reduced somewhat by training some additional
epochs on LibriSpeech, but I think we still could improve this by about
0.6% if we either solve the pipeline difference or run a longer training
over the transferred checkpoint.)

[This is an archived TTS discussion thread from discourse.mozilla.org/t/integration-of-deepspeech-polyglots-new-networks]

[JRMeyer]

>>> reuben
[January 25, 2021, 2:41pm]

Fantastic work!

> It would be great if you could look into this and update the client
> bindings accordingly.

As you may have noticed from the posts here in the past few months the
maintainers currently have limited time to work on the project.

will have time to do such a change, let alone having to understand your
codebase first. If you're invested in updating the native client
bindings and can write up a plan in an issue, with the relevant
differences highlighted and how they could be solved, that'd be a better
start to this conversation. If we can find a workable integration path
I'm happy to mentor and review code but I won't have bandwidth to work
on this directly.

> Notes on the above checkpoint: I did transfer the pretrained models
> from Nvidia

Note that due to the complete lack of any licensing terms on NVIDIA's
pre-trained checkpoints, we wouldn't be able to ever release models
transferred from them as part of official DeepSpeech releases.

> Splitting DeepSpeech into the three parts of usage, training and
> datasets.

I repeat myself, but again, not enough developer time to take on major
refactors like this that have implications throughout the entire
codebase right now.

[Archived Post]

[JRMeyer]

>>> dan.bmh
[January 25, 2021, 3:18pm]

> will have time to do such a change

Hmm, I will be quite busy over the next time with improving the training
itself. But maybe some of the other more experienced developers/users
here are interested in taking over some parts... We also could skip the
streaming feature first.

> relevant differences highlighted

Not an issue yet, but just to mention the most important pipeline
updates:

add those)

Question from me: Can we use tensorflow functions / tf datapipeline
directly in the native client or did you reimplement all those parts in
native C?

slash

> complete lack of any licensing terms on NVIDIA's pre-trained
> checkpoints

The NeMo project is under Apache2, this might be the case for the
checkpoints, too. But I will investigate this further.

[Archived Post]

[JRMeyer]

>>> lissyx
[January 25, 2021, 7:55pm]

> We also could skip the streaming feature first.

I don't think we want, realistically.

> Audiosignal: Added normalization and preemphasis (should be easy to
> add those)

This is something with a lot of opinions, I'm not sure about that.

> Added per feature (channel) normalization

Same.

> Question from me: Can we use tensorflow functions / tf datapipeline
> directly in the native client or did you reimplement all those parts
> in native C?

We tend to leverage as much as we can from TensorFlow, that's why we
have some patches to expose some internals to the builds.

[Archived Post]

[JRMeyer]

>>> dan.bmh
[January 26, 2021, 8:30am]

> This is something with a lot of opinions, I'm not sure about that. slash
> slash [... slash ] slash
> Same.

This might be opinionated but Nvidia did already make the decision here.

slash

I will try to package the preprocessing steps into the .pb and .tflite
checkpoints, that we have only the raw signal as input. Would this be
easier to support in the native client then? The model outputs the ctc
probabilities, with only a small difference to the deepspeech1 net, the
output timesteps are half the length of the input timesteps.

[Archived Post]

[JRMeyer]

>>> reuben
[January 26, 2021, 10:44am]

> Not an issue yet, but just to mention the most important pipeline
> updates:
>
> - Audiosignal: Added normalization and preemphasis (should be easy
> to add those)

We already apply preemphasis. I don't know what you mean with
'normalization'. Streaming support makes it hard to do normalizations
that operate over time.

> Question from me: Can we use tensorflow functions / tf datapipeline
> directly in the native client or did you reimplement all those parts
> in native C?

Yes, you can, that's already how the native client operates. The
inference graph has a few sub-graphs, one of which is for feature
computation and takes a single window's worth of samples and returns the
features. You should be able to change that at will as long as the input
and output node names and shapes match and it'll be compatible with
existing native client binaries.

> Added per feature (channel) normalization

Same comment as above about streaming and normalization.

[Archived Post]

[JRMeyer]

>>> dan.bmh
[January 26, 2021, 5:11pm]

> what you mean with 'normalization'

just a simple normalization to fit the audio into range -1 to 1 as
loudest signals (dividing with highest signal value)

> Streaming support makes it hard to do normalizations that operate over
> time.

I can leave out the signal normalization, it has almost no impact, but
dropping the feature normalization increases WER to 29.7%

slash

> The inference graph has a few sub-graphs, one of which is for feature
> computation and takes a single window's worth of samples and returns
> the features.

From the comments I've found in deepspeech.cc file it seems I need 3
input nodes. Could you please tell me the input names and shape or
provide me with a link to the source files?

Did you benchmark the difference of the inference speed using all three
buffers compared to a single buffer for audio? I'm not sure if I can
replicate the mfcc and batch buffers regarding the input shape.

[Archived Post]

[JRMeyer]

>>> lissyx
[January 26, 2021, 6:47pm]

> From the comments I've found in deepspeech.cc file it seems I need 3
> input nodes. Could you please tell me the input names and shape or
> provide me with a link to the source files?

github.com

#### mozilla/DeepSpeech/blob/master/native_client/deepspeech.cc

#include
#ifdef MSC_VER
#define USE_MATH_DEFINES
#endif
#include
#include
#include
#include
#include
#include

#include 'deepspeech.h'
#include 'alphabet.h'
#include 'modelstate.h'

#include 'workspace_status.h'

#ifndef USE_TFLITE
#include 'tfmodelstate.h'
#else

This file has been truncated. show
original

slash

github.com

#### mozilla/DeepSpeech/blob/master/native_client/modelstate.cc

#include

#include 'ctcdecode/ctc_beam_search_decoder.h'

#include 'modelstate.h'

using std::vector;

ModelState::ModelState()
: beam_width(-1)
, n_steps(-1)
, n_context_(-1)
, n_features_(-1)
, mfcc_feats_per_timestep_(-1)
, sample_rate_(-1)
, audio_win_len_(-1)
, audio_win_step_(-1)
, state_size_(-1)
{
}

This file has been truncated. show
original

slash

github.com

#### mozilla/DeepSpeech/blob/master/native_client/tflitemodelstate.cc

#include 'tflitemodelstate.h'
#include 'tensorflow/lite/string_util.h'
#include 'workspace_status.h'

#ifdef ANDROID
#include <android/log.h>
#define LOG_TAG 'libdeepspeech'
#define LOGD(...) android_log_print(ANDROID_LOG_DEBUG, LOG_TAG, VA_ARGS)
#define LOGE(...) android_log_print(ANDROID_LOG_ERROR, LOG_TAG, VA_ARGS)
#else
#define LOGD(...)
#define LOGE(...)
#endif // ANDROID

using namespace tflite;
using std::vector;

int
TFLiteModelState::get_tensor_by_name(const vector& list,
const char* name)

This file has been truncated. show
original

slash

github.com

#### mozilla/DeepSpeech/blob/master/native_client/tfmodelstate.cc

#include 'tfmodelstate.h'

#include 'workspace_status.h'

using namespace tensorflow;
using std::vector;

TFModelState::TFModelState()
: ModelState()
, mmap_env(nullptr)
, session(nullptr)
{
}

TFModelState::~TFModelState()
{
if (session) {
Status status = session->Close();
if (!status.ok()) {
std::cerr << 'Error closing TensorFlow session: ' << status << std::endl;

This file has been truncated. show
original

You might have to take a look at headers as well.

[Archived Post]

[JRMeyer]

>>> reuben
[January 26, 2021, 7:10pm]

This is where the inference graph is created:
https://github.com/mozilla/DeepSpeech/blob/f27908e7e3781b4ebed228a27439d9988b13a5c7/training/deepspeech_training/train.py#L687-L774

[Archived Post]

[JRMeyer]

>>> dan.bmh
[January 27, 2021, 4:36pm]

I have been able to package the input pipeline into the .pb model but
I'm currently struggling with the .tflite conversion. For now this does
only work with a fixed signal length. Not sure if this can be fixed or
the input has to be splitted into chunks of fixed sizes.

I also did take a closer look into Nvidia's streaming approach, which
uses fixed values for the per-feature-norm augmentation. In a test run
without file based normalizations, this did slightly reduce WER from
3.7% to 4.4%, but I think we could work with that. Compared to
DeepSpeech, the input buffer size is much larger. They use a frame size
of 2s and an additional window of 2.5 seconds before and after the
current frame.

[Archived Post]