Transformer sometimes generates video with initial black frame

[amezcua]

Version

Media3 1.2.0

More version details

No response

Devices that reproduce the issue

Android tests on S21 Ultra, emulator

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Yes

Reproduction steps

Open the demo app (keep the default video selected), tap Trimming, select a valid trimming interval for the video starting from a non zero value. Export the video, download the video to the PC and play it with QuickTime, the first frame(s) are sometimes black, depending on the interval.

Reproducible as well with VLC in OSX Sonoma but visible only when opening the file in advanced mode preview (opening it directly does not show the initial black frames). To reproduce in VLC:

File/Advanced Open File/Browse... In the dialog simple click on the file, the preview shows the first frame black

Reproducible with the Android tests as well with some of the videos.

• Test used clippedMedia_completesWithClippedDuration
• Video in the test MP4_ASSET_WITH_INCREASING_TIMESTAMPS_320W_240H_15S_URI_STRING does not reproduce the issue
• Change the video to MP4_ASSET_SEF_H265_URI_STRING
• Change the clipping interval values to:

    long clippingStartMs = 1_000;
    long clippingEndMs = 2_000;

The exported video has initial black frame(s)

• Change the clipping interval values to:

    long clippingStartMs = 1_500;
    long clippingEndMs = 5_000;

The exported video does NOT have the initial black frame(s)

These results are reproducible by running the test both in the emulator and in the Galaxy S21.

Expected result

The exported video does not have any black initial frame(s).

Actual result

Export succeeds with inconsistent results, some times including black frames at the beginning.

Media

Files with the issues. The issue is clearly visible when opening the following URLs in Safari. Chrome does not show the initial black frames. Otherwise download the files and test with QuickTime or VLC as mentioned in the repro steps.

Black initial frames. Test clippedMedia_completesWithClippedDuration. Export interval 1_000 to 2_000
https://github.com/androidx/media/assets/781581/71392df8-681d-4f0c-8abf-e66782d00824

No black initial frames. Test clippedMedia_completesWithClippedDuration. Export interval 1_500 to 5_000
https://github.com/androidx/media/assets/781581/864f25d6-c9ad-4912-bd10-1d0fb03cd1a8

Output of the sample app with the black frames
https://github.com/androidx/media/assets/781581/0d7cda03-d60a-4743-b5fc-fa9ab4434900

Bug Report

• You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

[dway123]

(I think either Tof or Sheena recently saw a bug regarding trimming and black initial frames on Samsung)

[tof-tof]

Wasn't me, maybe sheena, trying to assign her

[dway123]

Interesting, I see lots of tests for clipping in MediaItemExportTest, but none for clipping with an SEF asset. I also found an internal bug discussing SEF flattening + clipping specifically (http://b/233986762), alongside a TODO in our codebase corresponding to this.

We definitely don't support clipping for SEF flattening, but I wonder if we also don't support clipping for SEF in general.

Otherwise, I suspect that the issue here may have to do with:

1. Trimming near or away from a keyframe. I suspect changing the clipping interval might be necessary for the repro because it helps us hit a keyframe boundary that causes the bug.
2. Frames not being in ordered by timestamps (b-frames). This is because the repro case discusses swapping between MP4_ASSET_WITH_INCREASING_TIMESTAMPS_320W_240H_15S_URI_STRING and MP4_ASSET_SEF_H265_URI_STRING, where MP4_ASSET_WITH_INCREASING_TIMESTAMPS_320W_240H_15S_URI_STRING is nice in that there's a guarantee that all frames are in order of timestamps (no b-framse iiuc).

@tof-tof, assigning back to you since you understand trimming the best. (Also, @SheenaChhabra mentioned she hadn't seen a bug like this, and I couldn't find any internal/external bug discussing trimming and black initial frames, so it might have just been me misremembering.)

[dway123]

I also find this surprising that the black frame is visible on only some players.

I didn't see the black frame when playing the video on ExoPlayer (Samsung S10 and Pixel 7 Pro), using the command adb shell am start -a androidx.media3.demo.main.action.VIEW https://github.com/androidx/media/assets/781581/0d7cda03-d60a-4743-b5fc-fa9ab4434900. I could confirm seeing this on Safari but not Chrome though.

Safari and QuickTime, both use the same player infrastructure, as far as I understand, and I suspect (not sure) that the VLC preview is actually using QuickTime's infra to extract the frame under the hood, if it's using Finder.

@amezcua , did you see any other video player display a black frame when using the player (not Finder preview)? If not, I wonder if this is actually a bug in Safari / QuickTime video playing/extracting infrastructure. It's definitely still better for us not to produce videos that display poorly in other players though, but just not sure whether our videos are strictly incorrect.

[amezcua]

@dway123 The first report we received for this was when we shared a video to the iOS app counterpart. I also remember seeing the issue as well in an Android device but I can not reproduce it now (tried creating a simple POC) and don't have the original report so I may be misremembering as well.

What's odd to me though is that the exported videos only show the initial black frame depending on the clipping interval. Not all exported videos show this behavior when played in the same players.

If I can find any other way to reproduce it I'll report it here.

[amezcua]

I have done a couple more test this time using Windows Media Player in Windows 11 and the player behaves slightly different with the 2 videos, the one that shows the black frames in iOS and the one that does not.

If we play the video that shows the black frames, when the video ends, the player control track bar will reset to a position which is not the start of the bar. It can be seen in this screen capture.

BlackFramesClipped_WindowsMP.mp4

By contrast, if we play the video that did not show the black frames, the bar resets to the very beginning like here:

NoBlackFramesClipped_WindowsMP.mp4

Hopefully this can serve as a hint to you.

[dway123]

Thank you, @amezcua, it does seem like something's wrong with our first frame. I suspect this has to do with some interaction between trimming and keyframes.

Perhaps Chrome's / ExoPlayer's players are more tolerant of bad formats like this, whereas Windows 11's catches and adjusts for the bad format by moving the start position over by a frame, and Apple's simply disallows playback of the bad format.

I'll defer to @tof-tof who knows more about trimming for this.

[tof-tof]

The video seems to start on a blank frame rather than the first frame. I can't see the error on VLC but I definitely see it on safari and quicktime players. There seems to be other reports of safari not showing the poster frame.

The safari player shows something interesting where the seek bar jumps back in time before going forward. I also noticed the file has a (non-key) frame, then a key frame just after the 1 second mark, but this does not happen at the 1.5 second mark. This file has no B frames though and the first frame is timed just after 0 like the original clip. I've played it through a few frame by frame editors can can't see anything, so i think this may be an IOS issue, but i'll do some more investigation on next week

[tof-tof]

Looked into the frames of the files with ffprobe and one key difference in the two files.

For the video with no black frames, the comparison of the audio and video times is as follows

for the video with black frame

The key thing here is that in the first screen shot, the video starts before the audio, while in the second screenshot, the video starts after the audio, so safari is probably choosing to play the audio with a black frame, then play video when it starts, while most players will render the first video frame on screen before playback

when we trim, we currently take all the samples after the clip start time. In this faulty case, the first audio sample comes before the first video sample, leading to black frame, choice by safari.

Audio and video are processed separately in our pipeline, so this is not an easy problem to fix. I have a vague idea, but will take a while to fix. will update with decision on this at later date

[tof-tof]

I've done some more investigation into what it will take to fix this problem, and considering the fact that

• any fix will tamper with another feature we have to support optimising clipping operations, leading to hacky workarounds here.
• the problem isn't very widespread (only reproduces in certain conditions and oinly in safari and quicktime on desktop) and it's relatively easily fixable from the client side (can inspect the file and choose a slightly delayed starttime to miss the first audio buffer)

We won't fix this issue at this time. Thanks for letting us know. We'll be sure to keep this in mind in any future refactors of trimming that we plan.

[amezcua]

Thanks, @tof-tof

When you say:

and it's relatively easily fixable from the client side (can inspect the file and choose a slightly delayed starttime to miss the first audio buffer)

Do you have any pointers/sample on how we could do that in android?

(Also, the problem is not visible only in Safari and desktop, it is also noticeable in iOS, which is when we found it, when sharing an edited video from our Android app to our iOS app)

[tof-tof]

I'm not an expert, but i think there should be a few API to get timestamps out of audio and video tracks that you can use. for example AudioTrack.getTimestamp() on the audio side.

I'm not aware of API's on the video side, but iIrecently wrote code to extract the timestamp of the first key frame after a given time utilising media3 mp4Extractor. The class is on the main branch and creates an Mp4MetadataInfo
data class with this value as a field (firstSyncSampleTimestampUsAfterTimeUs). Hopefully that will give you the timestamp of the first video frame if you specify 0 as the given time. You could possibly run the output video through transformer again, with value from this data class as the clip start position and setStartsAtKeyFrame(true), which will run pretty fast since we won't need to decode and re-encode frames. One caveat is that you may need to be able to set the start position in microseconds, and that API is only on the main branch right now

[andrewlewis]

@tof-tof Would it make sense to provide an option to snap the start point requested by the app to the preceding video frame timestamp, and ensure that the first audio sample processed is the one at/after the first video frame timestamp? I'm not 100% sure if this actually would solve the problem, but it sounds like it might be a useful workaround for apps that need to ensure the media is playable on the affected player implementations. (Aside: I'm not sure AudioTrack.getTimestamp helps with this -- it gives you the current position of playout of raw audio to an audio output device, rather than providing information about a file?)

[tof-tof]

Talked to @andrewlewis offline and found a way to correct the issue without disturbing other trimming options. We'll add an api to Transformer.Builder to have an option snap to the first video frame, making sure the video is the first thing you see in playback. Currently working on and testing the change on the different browsers (example of new output with problematic case), I'll update this thread when it the change hits the main branch

[tof-tof]

As shown by the copybara bot, the API has been pushed to the main branch, with name Transformer.Builder.setEnsureFileStartsOnVideoFrameEnabled(). When enabled, this issue should be fixed

[amezcua]

Awesome! Thanks for that!