Determine events that an application needs to handle

[johnataylor]

Categorized Activities by Channel

The following tables show what events (Activities on the wire) can come from what Channels.

This is the key for the tables:

Symbol	Meaning
✅	The Bot should expect to receive this Activity
❌	The Bot should never expect to receive this Activity
⬜	Currenly undetermined whether the Bot can receive this

Activities can meaningfully be split into separate categories. For each category we have a table of possible Activities.

Conversational

\	Cortana	Direct Line	Web Chat	Email	Facebook	GroupMe	Kik	Teams	Slack	Skype	Skype Business	Telegram	Twilio
Message	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
MessageReaction	❌	❌	❌	❌	❌	❌	❌	✅	❌	❌	❌	❌	❌

• All Channels send Message Activities.
• When using a Dialog, Message Activities should generally always be passed onto the Dialog.
• This is probably not true of the MessageReaction although they are very much part of the conversation.
• There are logically two types of MessageReaction: Added and Removed

("Message Reactions" are things like a "thumbs up" on a previous comment. They can happen out of order so can be thought of as similar to buttons. This Activity is currently sent by the Teams Channel.)

Welcome

\	Cortana	Direct Line	Web Chat	Email	Facebook	GroupMe	Kik	Teams	Slack	Skype	Skype Business	Telegram	Twilio
ConversationUpdate	✅	✅	✅	❌	⬜	✅	✅	✅	✅	❌	❌	✅	❌
ContactRelationUpdate	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	❌	❌

• It is common for Channels to send ConversationUpdate Activities.
• There are logically two types of MessageReaction: Added and Removed
• It is very tempting to assume bot "Welcome" behavior can be simply implemented by wiring up ConversationUpdate.Added and this sometimes works.
• However, this is a simplification, in order to produce a reliable "Welcome" behavior the bot implementation may also need to use state.

Application Extensibility

\	Cortana	Direct Line	Web Chat	Email	Facebook	GroupMe	Kik	Teams	Slack	Skype	Skype Business	Telegram	Twilio
Event.*	⬜	✅	✅	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜
Event.CreateConversation	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜
Event.ContinueConversation	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜

• Event Activities are an extensibility mechanism in Direct Line (aka Direct Line).
• An application that owns both the client and server may chose to tunnel their own events through the service using this Event Activity.

Microsoft Teams

\	Cortana	Direct Line	Web Chat	Email	Facebook	GroupMe	Kik	Teams	Slack	Skype	Skype Business	Telegram	Twilio
Invoke.TeamsVerification	❌	❌	❌	❌	❌	❌	❌	✅	❌	❌	❌	❌	❌
Invoke.ComposeResponse	❌	❌	❌	❌	❌	❌	❌	✅	❌	❌	❌	❌	❌

• Along with a number of the other typed Activities, Microsoft Teams defines a few Teams specific Invoke Activities.
• Invoke Activities are specific to an application and not something a client would define.
• There is no general notion of Invoke only specific subtypes of the Activity.
• Invoke is currently the only Activity that triggers a request-reply behavior on the bot.

This is very important: if using Dialogs for the OAuth Prompt to work the Invoke.TeamsVerification Activity must be forwarded to the Dialog.

Message Update

\	Cortana	Direct Line	Web Chat	Email	Facebook	GroupMe	Kik	Teams	Slack	Skype	Skype Business	Telegram	Twilio
MessageUpdate	❌	❌	❌	❌	❌	❌	❌	✅	⬜	❌	❌	❌	❌
MessageDelete	❌	❌	❌	❌	❌	❌	❌	✅	⬜	❌	❌	❌	❌

• Message Update is currently supported by Teams.

OAuth

\	Cortana	Direct Line	Web Chat	Email	Facebook	GroupMe	Kik	Teams	Slack	Skype	Skype Business	Telegram	Twilio
Event.TokenResponse	⬜	✅	✅	❌	⬜	⬜	⬜	❌	⬜	⬜	⬜	⬜	⬜

This is very important: if using Dialogs for the OAuth Prompt to work the Event.TokenResponse Activity must be forwarded to the Dialog.

Uncategorized

\	Cortana	Direct Line	Web Chat	Email	Facebook	GroupMe	Kik	Teams	Slack	Skype	Skype Business	Telegram	Twilio
EndOfConversation	❌	✅	✅	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌
InstallationUpdate	❌	✅	✅	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌
Typing*	❌	✅	✅	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌
Handoff	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌

Out of Use (includes Payment specific Invoke)

• DeleteUserData
• Invoke.PaymentRequest
• Invoke.Address
• Ping

---

Summary of Activities supported per Channel

Cortana

• Message
• ConversationUpdate
• Event.TokenResponse
• EndOfConversation (when the window closes?)

Direct Line

• Message
• ConversationUpdate
• Event.TokenResponse
• Event.*
• Event.CreateConversation
• Event.ContinueConversation

Email

• Message

Facebook

• Message
• Event.TokenResponse

GroupMe

• Message
• ConversationUpdate
• Event.TokenResponse

Kik

• Message
• ConversationUpdate
• Event.TokenResponse

Teams

• Message
• ConversationUpdate
• MessageReaction
• MessageUpdate
• MessageDelete
• Invoke.TeamsVerification
• Invoke.ComposeResponse

Slack

• Message
• ConversationUpdate
• Event.TokenResponse

Skype

• Message
• ContactRelationUpdate
• Event.TokenResponse

Skype Business

• Message
• ContactRelationUpdate
• Event.TokenResponse

Telegram

• Message
• ConversationUpdate
• Event.TokenResponse

Twilio

• Message

Summary Table All Activities to All Channels

\	Cortana	Direct Line	Web Chat	Email	Facebook	GroupMe	Kik	Teams	Slack	Skype	Skype Business	Telegram	Twilio
Message	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
MessageReaction	❌	❌	❌	❌	❌	❌	❌	✅	❌	❌	❌	❌	❌
ConversationUpdate	✅	✅	✅	❌	⬜	✅	✅	✅	✅	❌	❌	✅	❌
ContactRelationUpdate	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	❌	❌
Event.*	⬜	✅	✅	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜
Event.CreateConversation	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜
Event.ContinueConversation	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜	⬜
Invoke.TeamsVerification	❌	❌	❌	❌	❌	❌	❌	✅	❌	❌	❌	❌	❌
Invoke.ComposeResponse	❌	❌	❌	❌	❌	❌	❌	✅	❌	❌	❌	❌	❌
MessageUpdate	❌	❌	❌	❌	❌	❌	❌	✅	⬜	❌	❌	❌	❌
MessageDelete	❌	❌	❌	❌	❌	❌	❌	✅	⬜	❌	❌	❌	❌
Event.TokenResponse	⬜	✅	✅	❌	⬜	⬜	⬜	❌	⬜	⬜	⬜	⬜	⬜
EndOfConversation	❌	✅	✅	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌
InstallationUpdate	❌	✅	✅	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌
Typing*	❌	✅	✅	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌
Handoff	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌

Please note that WebSockets vs REST is another layer of complexity not yet accounted for in these charts. Typing activities only work on WebSockets and do not work on REST.

[edited by @corinagum]

[Stevenic]

I think Skype sends ContactRelationUpdate

[EricDahlvang]

• Bing is no longer a channel
• Kik is a channel
• Slack is a channel

Skype does NOT send ConversationUpdate (I don’t think Facebook does either).

[johnataylor]

We have updated both C# and JavaScript in master. I'm not closing this, just relabeling, because we would like to keep this open as a doc issue and something to base samples on.

[johnataylor]

@EricDahlvang can you take another look.

Anything I wasn't sure about I made italics.

[EricDahlvang]

@johnataylor this is one my favorite documents.

[johnataylor]

Web Chat will send:
• "message": with "text" and/or "attachments"
• "event": with "name" and "value" (as JSON/string)
• "typing": if the user set an option, namely "sendTypingIndicator"
Web Chat will not send "contactRelationUpdate". And Web Chat do not support "messageReaction", no one explicitly ask us to support this feature.

By default, Web Chat will render:
• "message": will render as either carousel or stacked, depends on the option in the activity
• "typing": will render for 5s and hide it, or until next activity come in
• "conversationUpdate": will hide
• "event": will hide
• Others: will show a warning box (we never see it in production)
Web devs can modify this render pipeline to add/remove/replace any custom render.

Devs can use Web Chat to send any activity type and payload, we don't doc this feature, we don't recommend this feature. We always recommend custom things goes to "event" activity.

[corinagum]

Does this need to be updated with LINE events?

[corinagum]

I'm not sure if the ✅ is the correct here - the end of conversation docs say that Direct Line passes the activity forward to the client but does not do anything with this information. For Direct Line and Web Chat, should this be ⬜️ instead?

Also please note that WebSocket vs REST is a whole other layer of complexity to this conversation that is not accounted for above. For example, Typing only works on WebSocket and not REST, so Direct Line expecting to receive Typing is only true when using WebSockets.

[Micro-Muncher]

From the cortana side...

Hi; regarding how close conversation is used in Cortana

Desktop

1. User talks to “client”. Client calls speech to text. Client sends text to agent.
2. Agent figures out how to route the text. For example, open a third party skill.
3. Agent lets client know a skill is opening. Client opens a “canvas” in which to have a conversation. We have a session. A “Microsoft.Launch” intent is added and the invocation text is sent to the bot.
4. Client, agent, and bot now talk with request/response. Responses can include speak in which case text to speech is called by client.
5. a. When the bot determines its purpose is fulfilled, it asks session to end the conversation. This should end the session and close the canvas. b. when the client closes, right now nothing happens on the bot, but client and agent tear down the conversation. (log fill with errors if any response is sent from the bot. ideally the canvas close should cause the event to flow through agent to the bot to do whatever clean up it needs to do.)

Mobile
Same but different.
3. There is no canvas to open. However the app now has a context and messages flow from app to agent to bot.
5. Bot lets agent and client know that conversation/session is done.

So….
App< Hi I am Cortana. What can I do for you?
User> Open SlowChess
App< I’m on it!
[Launch.Intent]
SlowChess<Welcome to ShowChess. I see you have a game in progress. What is your move? Or do you want to see the board?
User>move Q C6 to C7
SlowChess<Black Queen C6 takes White Pawn on C7. Want to see the board or leave?
User>leave
SlowChess<Thanks for playing!
[endConversation]
App< Cortana here. What can I do for you?

Hopefully one day we supported nested (stacked conversations with warm hand offs. In that case, endConversation pops the stack.

I have other use cases I’d love to see 😊

For example
As a skill developer, I can set the duration up to a reasonable amount for the user to respond such that I can change the course of a conversation or provide feedback to the user.
As a skill developer, the skill is notified upon the end of rendering such that I can I can reprompt the user or change the flow of a conversation. (For example, if I send an audio card, I am sent an event when it finishes playing.)(Parity with Google Actions.)

[Micro-Muncher]

Cortana ignores tokenResponse because it handles its own oauth.
I [tried to] put a diagram together here.
https://github.com/bw-kforce-ms/CortanaSkillsWIP/blob/master/Support/OAuth.md#supporting-multiple-channel-oauth

[Micro-Muncher]

Does cortana get conversation updates? We implement welcome by watching for a Microsoft.Launch intent.