Building Coqui with TFLite delegate support

[jeremiahrose]

Hi there!

I gather from the docs and some old Deepspeech discussions that building and using Coqui STT with the GPU, NNAPI and Hexagon delegates was still in the experimental phase around 1 year ago (early 2020).

I'm wondering where that is at now, and what I would need to do to get it working?

Specifically I want to know the proper way to:

• Build libstt.so, stt native client, with delegates enabled. What Bazel flags are required? (I'm cross-compiling to ARMv7 Linux)
• Build the delegates themselves. What Bazel targets are required, and how do I make sure they're statically linked into libstt.so?
• Run stt native client with a delegate (is e.g STT_TFLITE_DELEGATE=gpu all that's required?)
• Make the Coqui 0.9.3 .tflite model work with a delegate (GPU or Hexagon, probably)

Happy to do whatever debugging is required to get this to work. Also hoping that this thread might become a starting point for other people who are trying to get this working.

[reuben]

All you need to do is set the environment variable and verify the model is behaving as intended. No need to change the build (or even build yourself).

Run stt native client with a delegate (is e.g STT_TFLITE_DELEGATE=gpu all that's required?)

Yep. Code here:

STT/native_client/tflitemodelstate.cc

Lines 102 to 156 in df26eca

	std::map<std::string, tflite::Interpreter::TfLiteDelegatePtr>
	getTfliteDelegates()
	{
	std::map<std::string, tflite::Interpreter::TfLiteDelegatePtr> delegates;
	const char* env_delegate_c = std::getenv("STT_TFLITE_DELEGATE");
	std::string env_delegate = (env_delegate_c != nullptr) ? env_delegate_c : "";
	#ifdef __ANDROID__
	if (env_delegate == std::string("gpu")) {
	LOGD("Trying to get GPU delegate ...");
	// Try to get GPU delegate
	{
	tflite::Interpreter::TfLiteDelegatePtr delegate = evaluation::CreateGPUDelegate();
	if (!delegate) {
	LOGD("GPU delegation not supported");
	} else {
	LOGD("GPU delegation supported");
	delegates.emplace("GPU", std::move(delegate));
	}
	}
	}
	if (env_delegate == std::string("nnapi")) {
	LOGD("Trying to get NNAPI delegate ...");
	// Try to get Android NNAPI delegate
	{
	tflite::Interpreter::TfLiteDelegatePtr delegate = evaluation::CreateNNAPIDelegate();
	if (!delegate) {
	LOGD("NNAPI delegation not supported");
	} else {
	LOGD("NNAPI delegation supported");
	delegates.emplace("NNAPI", std::move(delegate));
	}
	}
	}
	if (env_delegate == std::string("hexagon")) {
	LOGD("Trying to get Hexagon delegate ...");
	// Try to get Android Hexagon delegate
	{
	const std::string libhexagon_path("/data/local/tmp");
	tflite::Interpreter::TfLiteDelegatePtr delegate = evaluation::CreateHexagonDelegate(libhexagon_path, /* profiler */ false);
	if (!delegate) {
	LOGD("Hexagon delegation not supported");
	} else {
	LOGD("Hexagon delegation supported");
	delegates.emplace("Hexagon", std::move(delegate));
	}
	}
	}
	#endif // __ANDROID__
	return delegates;
	}

[jeremiahrose]

Thanks for the reply Reuben.
We are cross-compiling Coqui within a Yocto build system so fetching pre-built binaries is not currently an option.