-
Notifications
You must be signed in to change notification settings - Fork 559
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Media player support #196
Comments
Would really like this for WebChat. |
Also, there may be Audio player support. |
It is useful to include this, but give it some careful thought. As a user, I do not want video notifications or live tiles appearing on my PC or Phone. Consider implementation rules for auto-play, volumes etc |
From Action Center developer platform side (uses UWP renderer)...
We would need someone from the UX side to chime in on this, but I'd imagine we initially would disable the inline playback. For hosts that don't support hit targets and interactivity (like Timeline), they would need a way to configure it so that it just displays the thumbnail (without any play button), since it's only the entire card that is clickable, rather than specific content on the card that's clickable. Seems like that might already be possible with what's described here but writing that down just to make sure. |
The spec lacks size handling guidance for video. I think it should work like this:
What's more problematic IMO though (and I'm talking about the HTML implementation but it would probably apply to all platforms) is that each browser has its own controls implementation; they all look very different and they can't be styled. They will look quite bad inside a card, especially the audio player which can't even be sized. Therefore I believe we should own the controls, but that's a significantly higher investment unless we keep it simple, e.g. play/pause, maybe mute, but no seek nor volume sliders. |
Doing the play overlay button as an image in HostConfig is actually not that great a solution. Allowing the styling of the button to be done in CSS is a much better option and satisfies the customization needs. Besides, the design, at least for the HTML media player, calls for more than just the play button to be done as an overlay, and therefore additional properties would have to be added to HostConfig to handle the other button(s). I suggest we either:
|
I'm just discovering that since we last discussed and agreed on the spec, new requirements have been added, namely OnMediaClicked, OnMediaStarting, OnMediaPaused, OnMediaStopped (what's the difference between puase and stop?) and maxMediaSize (not sure about that last one though). These were never reviewed. What gives? |
The original spec was ambiguous and missing elements. For example, it didn't distinguish whether OnMediaStopped included paused (which would be needed by hosts), and it was completely missing scenario requirements. In fact, we still don't have full scenario requirements. I'm trying to talk to hosts today to determine the actual scenario requirements so that we can be sure we have the correct set of events (hosts might even need to be able to dynamically pause the video, for example) |
There was no ambiguity: these events were not part of the original spec, and we never reviewed them. Plus, these events don't need to be spec'ed here because they have nothing to do with the Adaptive Card format, they are features of the renderer that can technically be exposed in different ways depending on the platform. The spec above says "potentially, hosts need to be able to..." which doesn't seem like a strong requirement nor that there is a concrete scenario that needs the feature. |
As a matter of fact, you added all these to the spec 2 days ago @andrewleader. |
Yes there was ambiguity, that is an absolute fact witnessed by the devs who were attempting to develop this and had questions, since it was ambiguous (for example, the spec said that first frame of video is displayed unless author provides a poster, but the spec also said that the poster is displayed (or host default poster is displayed), so it's ambiguous what's supposed to be displayed. The "Potentially" one is an incomplete requirement, that's why it says potentially. We're trying to gather scenario requirements from hosts to determine whether those are needed. |
New open issue: What should the size of the play icon be? The spec currently just says to overlay and center the play icon image (which hosts can provide). That seems to imply that the play icon should be rendered however big the source image is. The problem with that is, what if a host wants to provide a high res play icon image? Say they provide a high-res 400x400 play icon… and the media element itself is only 500x500… 90% of the content would be covered by the play icon! Seems like it’d make more sense if the play icon is always a fixed size. Say 64x64 or something. Then, regardless of the resolution of the incoming image, the results are always consistent. Additionally, seems useful to let hosts override that fixed size. They might need more than 64x64, or they might want something smaller. Thoughts? Should we make those changes? What default size should we go with? Should we allow hosts to pick their own size? |
My opinion, as I shared here is that there is no need to allow for Play button customization in HostConfig. This is going too far for very little value add. I think every host should simply be responsible for choosing their own defaults, including size etc. Besides, the desired UI (for the HTML renderer at least) is this one: So it would be one thing to allow for the Play button customization, but what about the Rewind button? I think we need to change the spec. We can always add support for button customization later if we actually realize it's necessary, but it really doesn't strike me as necessary (nor practical) at the moment. |
The play button customization is only for the initial play button... Once clicked, the actual play/pause/rewind/volume controls appear (which cannot be customized). You can certainly imagine that Teams would want a purple-themed initial play button, etc. I'll check with some of the hosts to see if they actually require this functionality though, I'm not sure if they explicitly asked Matt for that functionality. |
The design for HTML calls for the same button to be used. The big play button changes into a big Pause button when clicked, then back to Play when clicked again. When the video is playing, the button automatically disappears if the mouse cursor moves outside the boundaries of the video or stays idle for a moment. It appears again as the mouse moves over the video. I think it should be possible for a renderer, be it the HTML one or anything other, to achieve the above design without being tied to using two different buttons. Said another way, I don't think there should be a requirement as to the UI of the video player for as long as it exposes the functional features described in this spec. |
The functional features in the spec require a initial play button to be displayed along with the poster, and when clicked, the poster disappears, video starts playing, and full media controls appear. If you disagree with the spec, then we have to have a conversation about that. |
What I do not agree with is that the initial Play button over the poster and the Play button once the video has started need to be different. I was hoping that my screenshot, explanation and further information here made that clear. And if they do not need to be different, I do not see it as necessary to make the Play button customizable in HostConfig - especially given that the design for HTML also calls for a Rewind button that looks and feels just like the Play button. So the bottom line IMO is:
So yes, as I have already said, I think we should have a conversation around reworking the spec a bit. |
Here's an example of why the initial play button and the media-loaded play button need to be different... in bots: Bots use media controls at the bottom, with a time scrub bar. But they want a large initial play button. Facebook Messenger does the same... In your case, where there's no media bar at the bottom, it certainly would make sense to re-use the same icon, but that's not the only case. |
So just to be clear - Bot clients that use the HTML renderer (WebChat for instance) will not get the experience you are showing above; they will get the experience I outlined and provided a screenshot of. In other words:
Just the two big play and pause buttons. That is, as I mentioned before, unless:
I do understand it's not the only case, but I am also saying that handling the richer case (like in the shots you included above) still doesn't require Play button customization in HostConfig, even if the buttons are different. The only value add of having it in HostConfig (with the complexities it leads to) is to, say, have the button as pictured in the bottom card of your first screen shot used in your second screenshot. It is not the case in your screenshot and the value add of supporting that is very, very low IMO. |
Thanks, we'll have to check with WebChat if that's acceptable then. I'll start a mail. |
Requirements from WebChat:
Can't believe we missed closed captions/accessibility! D'oh! |
Thanks @andrewleader. Now we need to have a conversation around it and decide together which of these requirements we decide to actually make part of the spec. To be upfront, though, if we decide that any of these is indeed required, then the HTML implementation will simply use the built-in player on all platforms, enabling every feature that can possibly be enabled. The implementation will be super cheap, but there will be no way to re-style the controls. However, even if HTML uses the built-in player, I still believe we should reconsider the spec for the Play button and even the Poster. Here is what I suggest and why:
On the caption front, this is how it's done in HTML: <video width="320" height="240" controls>
<source src="forrest_gump.mp4" type="video/mp4">
<source src="forrest_gump.ogg" type="video/ogg">
<track src="subtitles_en.vtt" kind="subtitles" srclang="en" label="English">
<track src="subtitles_no.vtt" kind="subtitles" srclang="no" label="Norwegian">
</video> It is likely we will have to add another property to the Media element to support text tracks, although I do not know how other platforms do it. While I am at it, I think we should also re-consider displaying the poster for Audio. Why can't card authors do a poster if they want to using an Image, and why is there a need for a default poster for an Audio stream? So let's please spend some time soon and re-discuss this spec. |
Reasons for the "initial state" (thumbnail + play button)
Although for HTML it looks like the HTML video player supports a poster property already? So is that what you meant by it's "handled automatically"? You meant we should just use that property?
They can? Isn't that what we wrote in the spec?
The default poster is currently shared between video and audio... definitely might make sense to have those separated, so that a host could have none or a different default poster for audio. |
I'll schedule a meeting with everyone soon! |
Many (most?) video files have an embedded cover image; retrieving it is hardly more expensive than downloading an image separately, and happens very fast. I think we need to be realistic when it comes to the developer experience - do we really want card authors to generate custom, separate poster images for all the videos they put in their cards, even though the video itself likely already has an embedded cover picture? Do we expect that the Media element will be most used to display videos owned by the card author or videos owned by a third party? If the latter, do we really think it's a realistic/acceptable workfow for card authors to go extract poster from a video they don't own, and store it separately on their side? You mention YouTube - our solution will not handle YouTube videos, since AFAIK one can only embed a YouTube player for that. Are we sure we do not want to support YouTube videos? I meant Image with a capital I, as in an Adaptive Image element. |
More thoughts. Say we do require the poster as per the spec:
So we have a choice to make here:
|
Good thoughts on the aspect ratio... Expanding on those some and giving them labels
Seems like the best option is Option A. We would tell authors that they should use the same aspect ratio on the poster as the video, but worst case (should be uncommon), the height changes when it starts playing... but at least there's never any black bars. The second best option seems like Option C. But I think the height changing would be preferred over black bars. If we were allowed to pre-download some of the video, Option B would seem ideal. But even if we did that, the poster would get downloaded first, and then the video size info would come in later... so we still would have to make a decision on how to size the content before the video gets loaded (and we need to make a decision on how to size the content before the poster gets loaded too...). Basically, there's three stages there...
So the size could change twice within 2 seconds... that could be wonky. Then again, authors can fix this by just providing the correct aspect ratio. So I say Option A is the best... it should be a minority that authors don't provide the correct aspect ratio, and even if they don't, a small jump in height won't be the end of the world. |
Looking at Edge and Chrome's default Given that it is also true we want to minimize to amount of data loaded (e.g. not load the video until the user decides to play it) and that my assumption that most videos have an embedded thumbnail image is apparently false (and even if it were true, browsers apparently all fetch the full video immediately as the page is loaded), I do agree that the poster is required. I am still not sold on the HostConfig-configurable Play button though, but we can discuss that at the meeting Andrew will setup. |
Documentation is done and in the mahiding/site1.1 branch. |
cc: @compulim |
Completed for the 1.1 release |
this version of adaptive card( playing video) is supported in which which channel? can i use it in webchat? |
Hey @amity5020, please ask that question on the Bot Framework WebChat issues. I believe the latest version of WebChat is running Adaptive 1.1, but they would know! |
Hi @dclaux , @matthidinger! I have a question about Adaptive Cards 1.1 in Microsoft Teams. Recently when I tried to post a card in 1.1 version Teams did not display it, instead showing the "fallbacktext". I wanted to display media. |
Hey @tposzytek, Teams hasn't updated to 1.1 yet as far as I'm aware. There's nothing you can do, as the Teams app itself needs to update to the newest version of Adaptive Cards. We've shared your feedback with them. However, what you can do is **leave the |
Hi! Thanks for the answer. Unfortunately when using version 1.0 the Media card is absolutely not being displayed. So this doesn’t work for me since I wanted to use that particular feature 😊
From: Andrew Leader <notifications@github.com>
Sent: poniedziałek, 27 maja 2019 18:42
To: microsoft/AdaptiveCards <AdaptiveCards@noreply.github.com>
Cc: Tomasz Poszytek <tomasz@poszytek.eu>; Mention <mention@noreply.github.com>
Subject: Re: [microsoft/AdaptiveCards] Media player support (#196)
Hey @tposzytek<https://github.com/tposzytek>, Teams hasn't updated to 1.1 yet as far as I'm aware. There's nothing you can do, as the Teams app itself needs to update to the newest version of Adaptive Cards. We've shared your feedback with them.
However, what you can do is **leave the "version" property set to "1.0". It's actually a minVersion property, where you're specifying the minimum version you require to be rendered. So, if you want your card to still be rendered (and any new 1.1 features will be ignored, simply leave it as "version": "1.0"! Does that work for you?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#196?email_source=notifications&email_token=AMCPVG2ZPTAT7A7R2GRBPIDPXQFOTA5CNFSM4DGJOMK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWKGCXY#issuecomment-496263519>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMCPVG5QNWKLAMSBK626T5TPXQFOTANCNFSM4DGJOMKQ>.
|
Implementation status
Problem
It is not possible to embed media elements in an Adaptive Card today. Media would initially support audio and video.
Scenario Requirements
Authors need to be able to provide both video and audio. Some bots only send audio snippets.
Media does NOT auto play.
NEW: Media is NOT downloaded until user clicks play
preload="none"
Hosts can choose to allow inline playback (or disable interactivity completely as usual).
By default, without any host changes...
For video...
For audio...
If hosts disabled inline playback, hosts need to know when the media element was tapped, so that they can handle playing the media themselves
Hosts need to be able to limit the max download size of the http media, so that a 1 GB file doesn't get downloaded, for example.
Authors can provide AltText for accessibility purposes
Authors can provide closed captions file
Asks
What Facebook does...
Cortana requirements
WebChat requirements
Comps
Default state
The state when the card first appears. This is the state that appears regardless of inline playback allowed. If interactivity isn't allowed, the play icon shouldn't be rendered.
Inline playing/paused state
If inline playback allowed, clicking from the default state results in what's seen below: Play button disappears, poster disappears, and native OS media player and controls should be used, no specific requirements on buttons available.
Open Questions
Example Comps
Spec
Schema Changes
Media
true
MediaSource[]
true
string
false
string
false
MediaSource
string
true
string
true
Example
Host Config
The following options allow hosts to override the
defaultPoster
if the card does not include aposter
. It also allows customization of theplay
overlay button before the media has played.Media states
Each platform represents media states in different ways. Listing them out here to determine the best union for what we want to use.
Renderer Requirements
sources
and use the first recognizablemimeType
to determine the type of media (audio or video)OnMediaClicked
event (using whatever local naming schema makes sense) ONLY if host disabled inline playback, so that host can display the mediamaxMediaSize
from host config to prevent accidental download of excessively large files.sources
contains an invalid mix of MIME types (audio and video) the element MUST be dropped and a warning SHOULD be logged.Media features intentionally left out
The following options are not provided to card authors.
Reference
For reference, HTML supports these:
Downlevel Impact
High
Preview image will display, but any actionable media will not play. It is important to not display play button when on a 1.0 renderer to limit confusion.
1.2 features
OnMediaStarting
,OnMediaPaused
andOnMediaStopped
events to allow hosts to know the state of the mediaThe text was updated successfully, but these errors were encountered: