Record video get stuck after orientation changed

[vanhungoz]

1. • I have read the SUPPORT document
2. • I have checked the Troubleshooting Guide
3. • I have tested with the latest version of Kurento

i am using kurento version 6.13 to record a streaming video.
whenever i rotate the screen, video always get stuck. it always displays the latest image of the previous orientation.
below is a video record sample.

https://corona-s3fs.s3-ap-northeast-1.amazonaws.com/96cb8a3c-9faa-48b5-8bf0-3a75025901bc.MP4

[vanhungoz]

the client is using react-native-webrtc

[j1elo]

This maybe is:
a) A limitation of the RecorderEndpoint: once the recording has started, changes in video resolution size are not accepted. UPDATE: Some testing proved that this is not true, and changes in resolution are correctly handled by the RecorderEndpoint.
b) A limitation of the container format (MP4, MKV, WEBM)
c) An issue introduced by Safari/iOS.

Related issues:

• twilio/twilio-video.js#637 iOS 12.2 Safari on Video Rotation (rotating right) Remote Video Freezes
• #359 Recorder Endpoint MP4 video freeze with Chrome
• #372 Chrome recording downscaled to 640x360px

Related topics:

• iOS Safari RTP Stream no video-orientation in SDP Answer
• Video freeze on camera orientation change
• Errors in recording video file, output file is of just 1sec

This is a way the issue has manifested before, with video coming from an iOS Safari, when the user rotated the device:

First format detected for GStreamer capabilities is like this:

video/x-h264, stream-format=(string)avc, [...], width=(int)480, height=(int)640, framerate=(fraction)0/1

then it suddenly changes to this...

video/x-h264, stream-format=(string)avc, [...], width=(int)640, height=(int)480, framerate=(fraction)0/1

...and then it is when the error message appears: pad video_0 refused renegotiation.

Note the chage of width and height. Other samples of this issue included a change in total video size upon rotating the device, so it is not only a matter of swapping width with height.

[j1elo]

I had forgotten, but from this comment maybe the problem is not RecorderEndpoint, but the MP4 container format that is not compatible with dynamic changes in video resolution.

Users affected by this issue should try recording with MP4, WEBM, and MKV formats, and let us know if the issue is solved with any of those formats, so we can update the RecorderEndpoint documentation to reflect that fact.

[j1elo]

I have been able to test with an iPhone 6 with iOS 12.4. The result is that rotation causes a stuck video in all 3 recording formats (WEBM, MKV, MP4).

I also tested with Chrome and Firefox on Android. This is a summary:

• On Android + Firefox and Chrome + WEBM and MKV, the result is a video with changing resolution when the phone is rotated. I.e. the player window (used ffplay) changes dynamically between 460x680 (vertical) and 680x460 (horizontal).

• On Android + Firefox and Chrome + MP4, the result is a video with static resolution. When the phone is rotated, the video contains black borders that make up for the new proportions of the video. The player window doesn't change.

• On iOS + Safari + all of WEBM, MKV, MP4, the result is a video that hangs whenever the phone is rotated. When the phone is rotated again, the video resumes. So if the phone started horizontal, a 90º rotation will hang the video, and another 90º rotation will make the video move again. Meanwhile, audio continues playing, and A/V synchronization is not lost.

What is even more important: ffplay says that there is no video data whatsoever during the stuck periods. During the time the phone was rotated and the recorded video hangs, there is actually no video data in the recorded file. So that's why it looks like it's stuck.

Capture of the ffplay command. Note how the timestamps come to a halt at 00:05:23, then it resumes at 00:11:00. That's from the time the iPhone was rotated 90 degrees.

I've been thinking about that and this might be caused by video frames not flowing between the WebRtcEndpoint and the RecorderEndpoint. But why? Is the iPhone actually sending video while the phone is rotated? (because if not, that's clearly an iOS issue).

[j1elo]

Adding to

Is the iPhone actually sending video while the phone is rotated?

I've set up a simple test with the OneToOne tutorial, between Safari/iOS and Chrome desktop.

The result is that the video from Safari also got stuck in the Chrome page. Chrome is not receiving input video when the iPhone is rotated!! This takes the RecorderEndpoint out of the issue.

Now the question is: why is Chrome not receiving video? Is it because Kurento has a bug and it stops sending such video? Or is it because iOS is the one with a bug and it stops sending?

[micaelgallego]

The conclusion is that webrtc video from iOS Safari can not be managed by Kurento when orientation is changed or iOS itself can not manage it?

A test with a other WebRtc service like whereby should clear the doubts.

[j1elo]

I guess the conclusion is that, when rotating, Safari does something with the way it encodes videothat is very different to what Chrome & Firefox do. Kurento was optimized to work with these two, and it doesn't like whatever Safari does.

Maybe it is how the H.264 stream is reconfigured, or how the keyframes are emitted. H.264 is not only video, but also metadata inbetween, so there is a lot that could be going on, and to which the decoder in Kurento can react better or worse. Now that I think of it, maybe upgrading the decoder (GStreamer) would have an impact on this issue.

[micaelgallego]

I understand... please add a note somewhere in RecorderEnpoint until we can spend time to investigate the problem further and try to fix it.

[j1elo]

I found another source that makes me think this is, after all, a bug in Safari for iOS ≤ 13.0:

• This Twilio issue:
  twilio/twilio-video.js#637 iOS 12.2 Safari on Video Rotation (rotating right) Remote Video Freezes

@vanhungoz what version of iOS have you tested? Can you compare between 12.x and 13.x?

[vanhungoz]

@j1elo sorry for my late. i am using ios 12.4.6

[vanhungoz]

client is app that is developed by react-native

[vanhungoz]

now, i need to update xcode to test ios 13.5. will be notify you when i done

[vanhungoz]

@j1elo this is some of my test:
MP4 + ios 12.4.6 and 13.3.1: the result is a video that hangs whenever the phone is rotated.
WEBM + ios 12.4.6 and 13.3.1 and 13.5.1: it work

[RaistGH]

Hi,

Just as a note, maybe it's not just a Safari issue. Remember this issue where Windows+Chrome is changing resolution after few seconds. The Safari version we tested was 12.1. Maybe new Safari versions are doing the same.
Whenever we get some budget I'll try to do new checks with WEBM and MKV containers to see if they solve also the problem for Windows+Chrome since now iOS can support containers other than MP4.

Regards

[AlexandrDiakov]

Are there any news or updates regarding this issue or whats causing it?
We are experiencing similar problem with video playback of incoming remote stream freezing after rotating iOS device (tested with 15.2.1) but it is inconsistent, sometimes it happens and sometimes not. Also, video freezes not upon first rotation but when user flips device back to its original orientation which makes me think it might be completely another problem.
Any thoughts would be much appreciated.

[j1elo]

Are there any news or updates regarding this issue or whats causing it? We are experiencing similar problem with video playback of incoming remote stream freezing after rotating iOS device (tested with 15.2.1) but it is inconsistent, sometimes it happens and sometimes not. Also, video freezes not upon first rotation but when user flips device back to its original orientation which makes me think it might be completely another problem. Any thoughts would be much appreciated.

Please open a separate bug report if you have observed these issues and are not related with the recording. This bug report is for the RecorderEndpoint.

[neilyoung]

The problem is in qtmux or mp4mux, which refuses renegotiation of caps if changed downstream (e.g. by resolution change).

It is even present, if not using Recorderendpoint, but RtpEndpoint and use a GST pipeline for the file save.

0:00:11.043992344 22062 0x7ff1100eb280 WARN                   qtmux gstqtmux.c:5346:gst_qt_mux_can_renegotiate:<qtmux0> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)0142c020ffe1000f6742c0208c8d407808bf700f08846a01000468ce3c80, level=(string)3.2, profile=(string)constrained-baseline, width=(int)960, height=(int)540, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true