FFmpegFrameGrabber autorotate frame based on video metadata "rotate"

[Xiaohan-Shen]

I am trying to cut and store certain part of a video, keeping the resolution and frame size the same. However, in some videos (mostly recorded by phones), the metadata "rotate" is not zero. For example, if the video is recorded with the phone held horizontally, the resulting video could have metadata "rotate" 270 or 90.

When editing videos like this, FFmpegFrameGrabber automatically rotates the video when grabbing. Is there a way to undo this? i.e. I want the video the way it is no matter what the metadata "rotate" says.

I have read other issues where FFmpegFrameFilter is recommended to be used to rotate the image back to normal. I have tried this, but the resulting video is never the same (either stretched or compressed, looking weird). For example, one of my original video has resolution 1280 * 592, metadata "rotate" is 270. The grabber has imageWidth automatically set as 592 and imageHeight 1280 (it should be the other way around). In the filter and recorder, I manually set the imageWidth to 1280 and imageHeight to 592, and give filters "transpose=cclock. " The resulting video is indeed 1280 * 592, but dramatically stretched and looks very different from the original one. Below is my code.

    // both start and end are in seconds, the test case cut from 0 to 60 seconds.
    private static File cutVideo(File video, long start, long end) {
        FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(video);
        grabber.setOption("rw_timeout", String.valueOf(getVideoTime(video) * 1000 * 1000 * 2));
        grabber.setOption("rtsp_transport", "tcp");
        File outputFile = new File("test.mp4");
        try {
            grabber.start();
            log.info("ImageWidth:" + grabber.getImageWidth());
            log.info("ImageHeight:" + grabber.getImageHeight());
            log.info("AudioChannels:" + grabber.getAudioChannels());
            log.info("Format:" + grabber.getFormat());
            log.info("Rotation angle: " + grabber.getVideoMetadata("rotate"));
            FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(outputFile, grabber.getImageWidth(),
                    grabber.getImageHeight(), grabber.getAudioChannels());
            recorder.setFrameRate(grabber.getFrameRate());
            recorder.setPixelFormat(AV_PIX_FMT_YUV420P);
            recorder.setAudioCodecName("aac");
            recorder.setVideoCodec(grabber.getVideoCodec());
            recorder.setFormat("mp4");
            grabAndRecord(recorder, grabber, start, end);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return outputFile;
    }

    private static void grabAndRecord(FFmpegFrameRecorder recorder, FFmpegFrameGrabber grabber, long start, long end) throws Exception {
        grabber.setTimestamp(start * 1000000);
        Frame nextFrame = null;
        String rotate = grabber.getVideoMetadata("rotate");
        if (rotate != null && rotate.length() > 1) {
            FFmpegFrameFilter filter = null;
            switch(rotate){
                case "90":
                    filter = new FFmpegFrameFilter("transpose=clock", grabber.getImageHeight(),
                            grabber.getImageWidth());
                    recorder.setImageHeight(filter.getImageHeight());
                    recorder.setImageWidth(filter.getImageWidth());
                    break;
                case "180":
                    filter = new FFmpegFrameFilter("rotate=PI", grabber.getImageWidth(),
                            grabber.getImageHeight());
                    break;
                case "270":
                    filter = new FFmpegFrameFilter("transpose=cclock", grabber.getImageHeight(),
                            grabber.getImageWidth());
                    recorder.setImageHeight(filter.getImageHeight());
                    recorder.setImageWidth(filter.getImageWidth());
                    break;
            }
            filter.setPixelFormat(grabber.getPixelFormat());
            filter.setFrameRate(grabber.getFrameRate());
            filter.start();
            nextFrame = grabber.grabImage();
            filter.push(nextFrame);
            Frame filteredFrame = null;
            recorder.start();
            while (grabber.getTimestamp() <= end * 1000000 && (filteredFrame = filter.pull()) != null) {
                recorder.record(filteredFrame);
                nextFrame = grabber.grabImage();
                filter.push(nextFrame);
            }
            filter.stop();
            filter.release();
        } else {
            recorder.start();
            while (grabber.getTimestamp() <= end * 1000000 && (nextFrame = grabber.grabFrame()) != null) {
                recorder.record(nextFrame);
            }
        }
        recorder.stop();
        recorder.release();
        grabber.stop();
        grabber.release();
    }
    

[saudet]

FFmpegFrameGrabber doesn't apply any rotation, regardless of the metadata. You'll need to use FFmpegFrameFilter for this, yes.

[saudet]

You have the width and height parameters reversed. You'll need to do it as shown in this unit test:
https://github.com/bytedeco/javacv/blob/master/platform/src/test/java/org/bytedeco/javacv/FrameFilterTest.java

[Xiaohan-Shen]

Thanks a lot for the reply! It just comes to me that it is not that FFmpegFrameGrabber apply rotation, but that it doesn't. The metadata "rotate" tells any video player to rotate it before playing, but FFmpegFrameGrabber does not do the autorotation. It only grabs the original frames.

The image now looks ok, but for some reason the video quality is reduced greatly (video size shrinked from 176MB to 3MB). How can I make the recording the same quality as it was?

More importantly, since I need to filter only the image but not the audio. How can I somehow pre-record the audio before recording the filtered images. The audio channels are completely gone because I used grabImage(). I have looked at the example you gave, where it used filter that filters both the image and audio parts, but I only need to filter the image, would you please show me a simple example how you could record the audio and video separately?

[saudet]

You'll need to use the same codec, same bitrate and/or quality setting, etc as the original video.
BTW, if you're only doing this to transcode files, it's a lot easier with the ffmpeg program:
http://bytedeco.org/javacpp-presets/ffmpeg/apidocs/org/bytedeco/ffmpeg/ffmpeg.html

[saudet]

If you don't specify an audio filter, exactly like you did, the audio frames won't get filtered. You don't need to do anything special.

[Xiaohan-Shen]

Thanks for the advice! I will take a look at the FFmpeg class.

For the filter solution, I am not sure what you mean by not doing anything special. The way I did it rotates the frames correctly, but since I only filtered the images but not the audio, the generated video does not have any sound. In short, I want to record the original audio without filtering it, but I am not sure how you could do that (grabber can only grab images or image + audio, not only audio, right?).

The example you gave shows a way to filter both images and audio, but I don't need to filter the audio.

[saudet]

Actually there was a small bug that I've just fixed in commit 50a597d.
If you replace grabImage() with grab() in your code,
it should work with JavaCV 1.5.4-SNAPSHOT: http://bytedeco.org/builds/

[Xiaohan-Shen]

Thanks a lot! It should work smoothly now, but I have taken your advice of using the ffmpeg program, which suits my need much better!

[haiahaiah]

Thanks a lot for the reply! It just comes to me that it is not that FFmpegFrameGrabber apply rotation, but that it doesn't. The metadata "rotate" tells any video player to rotate it before playing, but FFmpegFrameGrabber does not do the autorotation. It only grabs the original frames.

The image now looks ok, but for some reason the video quality is reduced greatly (video size shrinked from 176MB to 3MB). How can I make the recording the same quality as it was?

More importantly, since I need to filter only the image but not the audio. How can I somehow pre-record the audio before recording the filtered images. The audio channels are completely gone because I used grabImage(). I have looked at the example you gave, where it used filter that filters both the image and audio parts, but I only need to filter the image, would you please show me a simple example how you could record the audio and video separately?

@saudet hi~ thanks for your excellent work. I want to know the meaning of the result returned by this getDisplayRotation() For example, 90 means should I rotate the frame returned by FFmpegFrameGrabber ff = new FFmpegFrameGrabber(videoPath); Frame framge=ff.grabImage() clockwise or counterclockwise 90 degree to get the correct image? Thanks a lot!