Ability to get character/word level confidence scores

[remyzerems]

When using *WithMetadata functions, it would be helpful to get access to the confidence score of each token of a given candidate transcript.

It could be made available in the TokenMetadata class as a public member confidence.

[reuben]

The main decoder loop is here: https://github.com/coqui-ai/STT/blob/main/native_client/ctcdecode/ctc_beam_search_decoder.cpp

The trie data structure used to keep individual tokens is here: https://github.com/coqui-ai/STT/blob/main/native_client/ctcdecode/path_trie.h

The log_prob_c member contains log-probability for the current character. At decode time, only the accumulated score from the beginning of the transcript until the current node (the score member) is copied into the Output structure:

STT/native_client/ctcdecode/ctc_beam_search_decoder.cpp

Line 266 in 11c2edb

output.confidence = scores[prefixes_copy[i]];

This Output structure is then converted into the public facing Metadata/CandidateTranscript/TokenMetadata here:

STT/native_client/modelstate.cc

Line 40 in 11c2edb

ModelState::decode_metadata(const DecoderState& state,

Basically to do this one would have to write a bunch of boring code shuffling this data through the layers of the implementation, so it can be used at the API level.

[reuben]

@juliandarley ^