Support Atom Based Processor

[koh-osug]

I'm migrating from deepspeech 0.9.3 to coqui. I had issues with deepspeech 0.9.3 because tensorflow was not working on atom processors because of a missing CPU instruction. AVX or a different feature is missing in comparison to other Intel processors. Do I have to face the same issue with coqui and have to rebuilt coqui from scratch?

[FrontierDK]

Apparently so. I have managed, on my Atom x6425e, to build the coqui stt ctcdecoder wheel and the stt wheel, and now finally the training part (CPU only, not GPU). The https://stt.readthedocs.io/en/latest/BUILDING.html "manual" is lacking a lot.

I have managed to create it using Ubuntu 20.04 in a VM, having 16GB RAM for the VM.

PS - do notice, it also takes a new tensorflow 1.15.4 recompile.

[fquirin]

Some of my SEPIA STT-Server users experience the same issue when running Coqui on a VM or as Docker container in a Synology DS920+ (CPU Intel Celeron J4125).

@FrontierDK Do you have wheel files for testing maybe? 🙂

[FrontierDK]

I sure do have the files needed

http://mit-ip.dk/linux.guf/coqui/atom/coqui_stt_ctcdecoder-1.4.0-cp38-cp38-linux_x86_64.whl
http://mit-ip.dk/linux.guf/coqui/atom/coqui_stt_training-1.4.0-py3-none-any.whl
http://mit-ip.dk/linux.guf/coqui/atom/generate_scorer_package.zip
http://mit-ip.dk/linux.guf/coqui/atom/STT-1.4.0-cp38-none-linux_x86_64.whl

I am training and running OK on Intel Atom x6425e / Ubuntu 20.04.05 in Windows HyperV virtual machines.

[FrontierDK]

Do notice that these files are CPU-only (no GPU)

[FrontierDK]

Word of advice: since these files are CPU-only, first try installing a fresh Ubuntu 20.04.05 virtual machine. Take a snapshot/backup after installing the updates (DON'T upgrade it to 21.xx/22.xx), and then create a list of commands which you enter.

To make it really easy for your self, install SSH in the Ubuntu VMs - this allows for you to use Putty (the SSH client for Windows) and easy/copy pasting. This allows for you to try out various directions, without using tons of hours of trying to fill out the many gaps in the Coqui documentation. And once you get it al rolling, make a backup of that/those VMs - since Ubuntu mirrors / program versions have a nasty habbit of disappearing. Also have a fresh backup, just prior to training.

Those backups can easily be used later on, and thus ensure a more robust enviroment. CUDA can even be used with virtual machines, using PCI-E passthrough - again allowing for a robust enviroment.

[fquirin]

Thanks for the files! :-) Just one issue ... 😅 do you have them for Python 3.9 as well? 🙈
I usually build Docker containers and since I use Debian Bullseye I already have a lot of Python 3.9 dependencies :-|. Maybe even better: Do you have a build script for the files, so I can rebuild everything when switching to Python 3.10 :-)?
About CUDA: atm I don't really care about it since I usually target machines at the lower end of the hardware spectrum (SBCs, simple home servers etc.).

[FrontierDK]

Sorry, no. Compiling both Tensorflow and Coqui binaries was a true headache, and Python 3.8 was the middle ground I got working, both with Cuda (Xeon) and without Cuda (both Atom and Xeon). My next headache was to get PCI-E passthrough working, and I kinda almost got it working - enough for training, but not in a nice way. But it's at least enough for creating models - again and again, even if the physical PC breaks.

[kaoh]

I have build TensorFlow wheels including instructions how to do this in this repo: https://github.com/HandsFreeGadgets/python-wheels
Obviously, a upstream support from TensorFlow or Coqui would be nice to have.

[fquirin]

Oh that repo looks interesting 👍 , I'll check out the wheels as soon as I find time and a test machine/user with Atom CPU ^^

[fquirin]

@kaoh Did you build the stt ctcdecoder as well? 🙂

[kaoh]

@kaoh Did you build the stt ctcdecoder as well? slightly_smiling_face

No.