Addition of VideoToolbox hardware acceleration codecs in ffmpeg

[LPFchan]

Hi,

I've noticed when I was encoding a video and audio file together in a-Shell, it takes a bit of patience for it to finish and heats the device up considerably, which lead me into researching a bit more. Turns out ffmpeg does support hardware acceleration on iOS and it's called VideoToolbox. Here's more info from ffmpeg:

https://trac.ffmpeg.org/wiki/HWAccelIntro#VideoToolbox

As far as I can tell, a build of ffmpeg embedded within a-Shell has no ability of hardware acceleration. But, you can add --enable-videotoolbox flag in FFmpeg configure script to enable it.

$ ffmpeg -h encoder=h264_videotoolbox
configuration: --cc=/Users/holzschu/src/Xcode_iPad/wasi-sdk/opt/bin/clang --ar=/Users/holzschu/src/Xcode _iPad/wasi-sdk/opt/bin/ar --ranlib=/Users/holzschu/src/Xcode_iPad/wasi-sdk/opt/bin/ranlib --disable-asm -- enable-cross-compile --disable-avfoundation --disable-appkit --disable-network --disable-pthreads --disabl e-debug --enable-libx264 --enable-gpl
Codec 'h264_videotoolbox' is not recognized by FFmpeg.

Apparently you can check it in ./configure after running the build script:
kewlbear/FFmpeg-iOS-build-script#58

Here's a build configuration example shared by devs of some app (just in case it helps):
https://medium.com/liveop-x-team/accelerating-h264-decoding-on-ios-with-ffmpeg-and-videotoolbox-1f000cb6c549

[holzschu]

Thanks for the detailed report and the links.
That would obviously require compiling ffmpeg for iOS instead of WebAssembly, but that seems like a good idea anyway.

[holzschu]

Update: ffmpeg compiles for iOS with --enable-videotoolbox --disable-audiotoolbox.

Enabling Arm64 assembly, even using the tips described at DeviLeo/DLGPlayer#19 (comment) did not work.

There are still many steps to make it compatible with the rest of a-Shell, but that's a beginning.

[holzschu]

Update: everything works except videotoolbox acceleration. I get some weird "DTS is invalid" errors, and the encoding stops.

[holzschu]

Update: ffmpeg compiled natively to arm64, with videotoolbox acceleration, is available for TestFlight testing. To make it work, you need to delete or rename the WebAssembly version.

[lxgr]

This is has been shipped, right?

[holzschu]

Yes. If you already installed the WebAssembly version of ffmpeg, you need to uninstall it.

[m-ruhl]

@holzschu The quality mode encoding is currently not working

Error: -q:v qscale not available for encoder. Use -b:v bitrate instead.

It should be supported for Apple Silicon (https://trac.ffmpeg.org/wiki/HWAccelIntro#VideoToolbox)

I think you just need to enable neon during compilation as Homebrew:
https://github.com/homebrew-ffmpeg/homebrew-ffmpeg/blob/88ca387a1dd19923a957965cedd6724a30a29dcc/Formula/ffmpeg.rb#L141

[holzschu]

If I remember correctly, --enable-neon uses assembly code, and crashes immediately on all the Arm iPads and iPhones I've tried. It seems there are subtle differences between assembly for Arm iPads and assembly for Arm Macs.

[m-ruhl]

If I remember correctly, --enable-neon uses assembly code, and crashes immediately on all the Arm iPads and iPhones I've tried. It seems there are subtle differences between assembly for Arm iPads and assembly for Arm Macs.

Seems to me, that ffmpeg-kit has neon enabled:

https://github.com/arthenica/ffmpeg-kit/blob/47f85fa9ea3f8c34f3c817b87d8667b61b87d0bc/scripts/apple/ffmpeg.sh#L58

Maybe this helps