[question] Start an event in sync with playing a local video file

[jackjohnsonn]

Issue description

Hi, I am developing an app to help children who suffer from autism with an experimental technique. I play a video file and at the same time in sync, I do a specific random timely event. I want the event that I am doing to be synced with the video by a very small margin. Another issue in here had the following code which I used:

isPreparing = true;
player.prepare(...);
player.setPlayWhenReady(true);

@Override
public void onPlayerStateChanged(boolean playWhenReady, int playbackState) {
  if(isPreparing && playbackState == Player.STATE_READY){
     myrandomevent(); // My random instant event.
     isPreparing = false;
  }
}

I have tested it on a local mp4 file. On the first run, there is a slight delay after the player playing and before onPlayerStateChanged Player.STATE_READY is reached (around 300-400ms) which results in myrandomevent() being delayed. On (most) further runs, there is not much delay and I get my sync with the random event, properly. I need to find a way to solve or bypass this.

Any information is highly appreciated. I asked in stackoverflow but they labeled my question "too broad" and removed it.

Version of ExoPlayer being used

Tested on 2.5.2 and 2.7.0. I would not mind using any other version.

Device(s) and version(s) of Android being used

Motorola Moto G (Lollipop) (At the moment).

Edit: Edited out the old 2.5.2 comment because 2.7.0 is better.

[tonihei]

The callbacks in onPlayerStateChanged are delivered asynchronously on your app thread after the corresponding event happened on the playback thread. So, if you see a delay between the video playing (which definitely means that the state is STATE_READY) and the callback, the most likely reason is that your app thread is blocked doing something else and the message is therefore only delivered with a delay. Can you check if your app thread is busy doing UI layout or reacting to other user input?
Alternatively, the player sometimes holds back the state change if other commands are still unhandled (for example a seek). Not sure if that applies here.

To do something at a specific playback time, we recently added another option. This allows you to create a message and choose when it is delivered and on which thread. If you deliver the message on the playback thread, you are perfectly in sync with the reported playback position. For example:

player.createMessage(
    new PlayerMessage.Target() {
      @Override
      void handleMessage(int messageType, Object payload) throws ExoPlaybackException {
        myRandomEvent();
      }
    })
   .setPosition(/* positionMs= */ 0)
   .send();

Be aware that executing the message on the playback thread also means that myRandomEvent() needs to be fast to prevent blocking the playback thread too long. You can also specify another thread with .setHandler(...) if needed.

[jackjohnsonn]

Thank you so much for taking the time to help me out, tonihei. I have used some sample code without anything extra (based on the project from https://github.com/yusufcakmak/ExoPlayerSample):

https://dpaste.de/iTmY

It is an activity that gets called from a main activity when I tap a button. That is all. The activity loads and runs the video. Could the loading of the video into the view delay the callback? I have tried both the PlayerView and SurfaceView. The delay for the first run is persistent.

What I cannot understand is. If the called activity is terminated with the player being released when I tap back. Why is it when I tap the button to run a new video activity, I do not get the same delay on most further tries?

I even tried removing player.setPlayWhenReady(true) from the initialization function of the player. Then, I have added a delay within onPlayerStateChanged before starting the player and my random event together. This helped a little bit. Though, it did not solve the problem.

The player.createMessage idea is really awesome. Though, I have an issue with it. The app lands within handleMessage() between 100-200ms before the playing of the video which results in my random event being executed 100-200ms early. I have checked this using logging.

I have tried using player.createMessage and it complained about "void handleMessage" so I have changed it to "public void handleMessage". I hope this is alright. Anyway, I have tried using it before the player prepare statement. It did not work. So, I assumed it must be used after the prepare statement. I did it after and it worked like mentioned above.

I also tried using setPosition at a further position at 4000ms. My event was summoned earlier, too. I have added a logging statement inside handleMessage() with player.getCurrentPosition(). The time I have gotten was around 3860. On further runs the time was almost the same with a negligible difference between 1-4ms.

I have also tried using player.setPlayWhenReady(false) on initialization then added player.setPlayWhenReady(true) on handleMessage() with my random event. I have received an error.

What is a bit annoying is that I know this can be done. I saw this player app on the store that does something similar and it does it with almost 100% sync on all tries. Though, I highly doubt the author will share how it is done.

[tonihei]

As you are calling the message on the playback thread, player.getCurrentPosition() may indeed report a position which seems to be less than the intended one. This is because getCurrentPosition() is intended to be used from the app thread (and not from the playback thread) and may not have been updated correctly. Nevertheless, the actual player position has already passed the time you specified.

In addition to that, there are some slight non-linearities at the beginning of playback where the audio renderer reports slightly unstable timestamps (see #3841). The time when the player starts at position zero is also not exactly the time when the first frame is rendered. We have a VideoRendererEventListener which has a callback onRenderedFirstFrame if you want to get notified of that.

[jackjohnsonn]

tonihei, I really cannot thank you enough for all the useful tips and information. I have just implemented the VideoListener on my class and added and used onRenderedFirstFrame(). It is working, but I have the same issue as the original issue.

On first run, there is a slight delay between the video is played and my random event is reached. After that, further runs perform as expected. If I can get all runs to run in the same manner either with or without delay, I could solve my problem. I just need consistency on runs.

I am curious. Why is there a delay only on the first run? Does something stay from the player, on further runs that makes the further runs perform as expected/differently? Is there something to completely reset the player that I am missing other than doing player.release().

P.S. I have changed my onStop() to force the release on all APIs like this:

    @Override
    public void onStop() {
        super.onStop();
        releasePlayer();
    }

Still no difference as opposed to the original approach.

[tonihei]

I don't know for sure what exactly causes the delay on the first run, but I assume it's something that is loaded or initialized once and then is readily available on further runs. The Android system may keep stuff around if you use it again within a short period of time. To know for sure what your app is doing in this time I'd suggest to use tools like systrace.

[jackjohnsonn]

Here is my systrace on both first and second launches.

First launch:
https://i.imgur.com/Mnz1J91.png

Second launch:
https://i.imgur.com/Z6spYHv.png

I have cropped and used only the two rows that I have found interesting. I am guessing the app stores the inflated view even after I return to the main activity. Though, I doubt it is the view inflation that is doing the delay. I have tried a snippet that waits for setContentView to finish and it did not help.

One workaround that I am considering is to launch the video activity, close it, then relaunch it on the first run of the app.

[tonihei]

Alternatively, you may set playWhenReady to false, wait for the onPlaybackStateChanged to STATE_READY and then set playWhenReady to true again. This should move the start of the playback behind the initial activity setup. When doing this, you can then use the messages (maybe with message time of 1 to ensure it's not triggered before you start playing) to execute your event.

[jackjohnsonn]

Thanks for taking all this time to help, tonihei. I have tried both of the suggestions three days ago. The first run delay is still there. I think this issue might be related to old devices. I am not sure.

[tonihei]

Closing the issue as the question was answered. Feel free to reopen if you have new information.