Skip to content

Commit

Permalink
Implement proper WebRTC call management
Browse files Browse the repository at this point in the history
This is a big CR that implements the whole proper call logic.

-- Other Contributors --

This CR was created in pair programming with:

-  Charlie Duquette <charlie_duquette@hotmail.fr>

-- New files --

- `ConversationProvider`: Provides the conversation object to its children. Also contains the function to begin a call (first step in the flow) which sends the `BeginCall` event.
- `CallProvider`: Contains the call logic that was previously in WebRTCProvider. From now on, WebRTCProvider contains only the WebRTC logic, while CallProvider
  contains everything related to the call (get the media devices, start/accept calls...)
- `NotificationManager`: Wrapper component to bind the WebSocket CallBegin event listener. That listener will fire when a `BeginCall` event is received and will then redirect the user to the call receiving page.

-- New routes --

When a `conversationId` is included in the URL, all pages are inside a `<ConversationProvider />`.

When starting a call, the caller is redirected to:

> http://localhost:3000/conversation/:conversationId/call?role=caller

When receiving a call, the receiver is redirected to:

> http://localhost:3000/conversation/:conversationId/call?role=receiver&hostId={HOST_ID}

When the user is in a `.../call` route, the `WebRTCContext` and
`CallContext` are provided

The `hostId` is the account URI of the caller. It's used when the receiver wants to send their answer back to the caller.

```
/
|-- login: <Welcome />
|-- settings: <AccountSettings />
|-- ...
`-- conversation: <Messenger />
    |-- add-contact/:contactId
    `-- :conversationId: <ConversationProvider />
        |-- /: <ConversationView />
        `-- /call: <WebRTCProvider>
                     <CallProvider>
                       <CallInterface />
                     </CallProvider>
                    </WebRTCProvider>
```

-- Call flow --

1. Caller:

- Clicks "Call" button
- Sends `BeginCall` event
- Redirects to call page `/call?role=caller`
- Sets `callStatus` to "Ringing"

2. Receiver:

- Receieves `BeginCall` event
- The callback in `NotificationManager` is called
- Redirects to the call receiving page `/conversation/{CONVERSATION_ID}/call?role=receiver`

3. Receiver:

- Clicks the "Answer call" button
- Sends a `CallAccept` event
- Sets `callStatus` to "Connecting"

4. Caller:

- Receives `CallAccept` event
- The callback in `CallProvider` is called
- Sets `callStatus` to "Connecting"
- Creates WebRTC Offer
- Sends `WebRTCOffer` event containing the offer SDP

5. Receiver:

- Receives `WebRTCOffer` event
- The callback in `WebRTCProvider` is called
- Sets WebRTC remote description.
- WebRTC `icecandidate` event fired. Sends `IceCandidate` WebSocket event
- Creates WebRTC answer
- Sends `WebRTCAnswer` event
- Sets WebRTC local description
- Sets connected status to true. Call page now shows the call interface

6. Caller:

- Receives `WebRTCAnswer` event
- Sets WebRTC local description
- Sets WebRTC remote description
- WebRTC `icecandidate` event fired. Sends `IceCandidate` WebSocket event
- Sets connected status to true. Call page now shows the call interface

-- Misc Changes --

- Improve CallPending and CallInterface UI
- Move `useUrlParams` hook from the (now deleted) `client/src/utils/hooks.ts` file to `client/src/hooks/useUrlParams.ts`
- Disable StrictMode. This was causing issues, because some event would be sent twice. There is a TODO comment to fix the problem and reenable it.
- Improvements in server `webrtc-handler.ts`. This is still a WIP
- Rename dash-case client files to camelCase

GitLab: #70
Change-Id: I6c75f6b867e8acb9ccaaa118b0123bba30431f78
  • Loading branch information
tran-simon committed Nov 22, 2022
1 parent b05ad99 commit f929a36
Show file tree
Hide file tree
Showing 30 changed files with 928 additions and 449 deletions.
1 change: 1 addition & 0 deletions client/i18next-parser.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
* <https://www.gnu.org/licenses/>.
*/
export default {
keepRemoved: true,
locales: ['fr', 'en'],
output: 'src/locale/$LOCALE/$NAMESPACE.json',
input: ['src/**/*.{ts,tsx,js,jsx}'],
Expand Down
8 changes: 6 additions & 2 deletions client/src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,12 @@ import { getAccessToken } from './utils/auth';
import { apiUrl } from './utils/constants';

export async function checkSetupStatus(): Promise<boolean> {
const { data } = await axios.get('/setup/check', { baseURL: apiUrl });
return data.isSetupComplete;
try {
const { data } = await axios.get('/setup/check', { baseURL: apiUrl });
return data.isSetupComplete;
} catch (e) {
throw new Error('Cannot connect to server', { cause: e });
}
}

export async function appLoader({ request }: LoaderFunctionArgs) {
Expand Down
4 changes: 2 additions & 2 deletions client/src/components/Button.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,11 @@ import {
} from './SvgIcon';
import CustomTooltip from './Tooltip';

type ShapedButtonProps = IconButtonProps & {
export type ShapedButtonProps = IconButtonProps & {
Icon: ComponentType<SvgIconProps>;
};

const RoundButton = styled(({ Icon, ...props }: ShapedButtonProps) => (
export const RoundButton = styled(({ Icon, ...props }: ShapedButtonProps) => (
<IconButton {...props} disableRipple={true}>
<Icon fontSize="inherit" />
</IconButton>
Expand Down
162 changes: 121 additions & 41 deletions client/src/components/CallButtons.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,14 @@
* <https://www.gnu.org/licenses/>.
*/

import { styled } from '@mui/material/styles';
import { IconButton, IconButtonProps, PaletteColor } from '@mui/material';
import { styled, Theme } from '@mui/material/styles';
import React, { useContext, useMemo } from 'react';
import { useTranslation } from 'react-i18next';
import { useNavigate } from 'react-router-dom';

import { WebRTCContext } from '../contexts/WebRTCProvider';
import { ExpandableButton, ExpandableButtonProps, ToggleIconButton } from './Button';
import { CallContext } from '../contexts/CallProvider';
import { ExpandableButton, ExpandableButtonProps, ShapedButtonProps, ToggleIconButton } from './Button';
import {
CallEndIcon,
ChatBubbleIcon,
Expand All @@ -44,8 +46,6 @@ import {
WindowIcon,
} from './SvgIcon';

type ColoredCallButtonColor = 'red' | 'green';

const CallButton = styled((props: ExpandableButtonProps) => {
return <ExpandableButton {...props} />;
})({
Expand All @@ -57,19 +57,28 @@ const CallButton = styled((props: ExpandableButtonProps) => {

const ColoredCallButton = styled(
({
buttonColor,
paletteColor,
Icon,
...props
}: ExpandableButtonProps & {
buttonColor: ColoredCallButtonColor;
}: ShapedButtonProps & {
paletteColor?: PaletteColor | ((theme: Theme) => PaletteColor);
}) => {
return <ExpandableButton {...props} />;
return (
<IconButton {...props} disableRipple={true}>
<Icon fontSize="inherit" />
</IconButton>
);
}
)(({ theme, paletteColor = theme.palette.primary }) => {
if (typeof paletteColor === 'function') {
paletteColor = paletteColor(theme);
}
)(({ buttonColor }) => {

return {
color: 'white',
backgroundColor: buttonColor === 'green' ? '#183722' : '#5E070D',
color: paletteColor.contrastText,
backgroundColor: paletteColor.dark,
'&:hover': {
backgroundColor: buttonColor === 'green' ? '#0B8271' : '#CC0022',
backgroundColor: paletteColor.main,
},
};
});
Expand All @@ -79,7 +88,19 @@ export const CallingChatButton = (props: ExpandableButtonProps) => {
};

export const CallingEndButton = (props: ExpandableButtonProps) => {
return <ColoredCallButton buttonColor="red" aria-label="call end" Icon={CallEndIcon} {...props} />;
const navigate = useNavigate();
return (
<ColoredCallButton
paletteColor={(theme) => theme.palette.error}
onClick={() => {
// TODO: Send event to end call
navigate('/');
}}
aria-label="call end"
Icon={CallEndIcon}
{...props}
/>
);
};

export const CallingExtensionButton = (props: ExpandableButtonProps) => {
Expand Down Expand Up @@ -132,7 +153,7 @@ export const CallingScreenShareButton = (props: ExpandableButtonProps) => {
};

const useMediaDeviceExpandMenuOptions = (kind: MediaDeviceKind) => {
const { mediaDevices } = useContext(WebRTCContext);
const { mediaDevices } = useContext(CallContext);

return useMemo(
() =>
Expand Down Expand Up @@ -162,7 +183,6 @@ export const CallingVolumeButton = (props: ExpandableButtonProps) => {
};

export const CallingMicButton = (props: ExpandableButtonProps) => {
const { isAudioOn, setAudioStatus } = useContext(WebRTCContext);
const options = useMediaDeviceExpandMenuOptions('audioinput');

return (
Expand All @@ -173,22 +193,26 @@ export const CallingMicButton = (props: ExpandableButtonProps) => {
options,
},
]}
IconButtonComp={(props) => (
<ToggleIconButton
IconOn={MicroIcon}
IconOff={MicroOffIcon}
selected={isAudioOn}
toggle={() => setAudioStatus(!isAudioOn)}
{...props}
/>
)}
IconButtonComp={ToggleAudioCameraIconButton}
{...props}
/>
);
};

const ToggleAudioCameraIconButton = (props: IconButtonProps) => {
const { isAudioOn, setAudioStatus } = useContext(CallContext);
return (
<ToggleIconButton
IconOn={MicroIcon}
IconOff={MicroOffIcon}
selected={isAudioOn}
toggle={() => setAudioStatus(!isAudioOn)}
{...props}
/>
);
};

export const CallingVideoCameraButton = (props: ExpandableButtonProps) => {
const { isVideoOn, setVideoStatus } = useContext(WebRTCContext);
const options = useMediaDeviceExpandMenuOptions('videoinput');

return (
Expand All @@ -199,29 +223,85 @@ export const CallingVideoCameraButton = (props: ExpandableButtonProps) => {
options,
},
]}
IconButtonComp={(props) => (
<ToggleIconButton
IconOn={VideoCameraIcon}
IconOff={VideoCameraOffIcon}
selected={isVideoOn}
toggle={() => setVideoStatus(!isVideoOn)}
{...props}
/>
)}
IconButtonComp={ToggleVideoCameraIconButton}
{...props}
/>
);
};

const ToggleVideoCameraIconButton = (props: IconButtonProps) => {
const { isVideoOn, setVideoStatus } = useContext(CallContext);
return (
<ToggleIconButton
IconOn={VideoCameraIcon}
IconOff={VideoCameraOffIcon}
selected={isVideoOn}
toggle={() => setVideoStatus(!isVideoOn)}
{...props}
/>
);
};

// Calling pending/receiving interface
export const CallingAnswerAudioButton = (props: ExpandableButtonProps) => {
return <ColoredCallButton aria-label="answer audio" buttonColor="green" Icon={PlaceAudioCallIcon} {...props} />;
export const CallingCancelButton = (props: IconButtonProps) => {
const navigate = useNavigate();

return (
<ColoredCallButton
aria-label="cancel call"
onClick={() => {
navigate(-1);
}}
Icon={CallEndIcon}
paletteColor={(theme) => theme.palette.error}
{...props}
/>
);
};

export const CallingAnswerAudioButton = (props: IconButtonProps) => {
const { acceptCall } = useContext(CallContext);

return (
<ColoredCallButton
aria-label="answer call audio"
onClick={() => {
acceptCall();
}}
Icon={PlaceAudioCallIcon}
paletteColor={(theme) => theme.palette.success}
{...props}
/>
);
};

export const CallingAnswerVideoButton = (props: ExpandableButtonProps) => {
return <ColoredCallButton aria-label="answer video" buttonColor="green" Icon={VideoCameraIcon} {...props} />;
export const CallingAnswerVideoButton = (props: IconButtonProps) => {
const { acceptCall } = useContext(CallContext);

return (
<ColoredCallButton
aria-label="answer call video"
onClick={() => {
acceptCall();
}}
paletteColor={(theme) => theme.palette.success}
Icon={VideoCameraIcon}
{...props}
/>
);
};

export const CallingRefuseButton = (props: ExpandableButtonProps) => {
return <ColoredCallButton aria-label="reject" buttonColor="red" Icon={RoundCloseIcon} {...props} />;
export const CallingRefuseButton = (props: IconButtonProps) => {
const navigate = useNavigate();
return (
<ColoredCallButton
aria-label="refuse call"
onClick={() => {
navigate(-1);
}}
paletteColor={(theme) => theme.palette.error}
Icon={RoundCloseIcon}
{...props}
/>
);
};
2 changes: 1 addition & 1 deletion client/src/components/ConversationListItem.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ export default function ConversationListItem({ conversation }: ConversationListI
const isSelected = conversation.getDisplayUri() === pathId;
const navigate = useNavigate();
const userId = conversation?.getFirstMember()?.contact.getUri();
const uri = conversation.getId() ? `/conversation/${conversation.getId()}` : `/add-contact/${userId}`;
const uri = conversation.getId() ? `/conversation/${conversation.getId()}` : `/conversation/add-contact/${userId}`;
return (
<Box onContextMenu={contextMenuHandler.handleAnchorPosition}>
<ConversationMenu
Expand Down
68 changes: 9 additions & 59 deletions client/src/components/ConversationView.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -16,61 +16,19 @@
* <https://www.gnu.org/licenses/>.
*/
import { Divider, Stack, Typography } from '@mui/material';
import { Account, Conversation, ConversationMember, WebSocketMessageType } from 'jami-web-common';
import { useContext, useEffect, useMemo, useState } from 'react';
import { Account, ConversationMember } from 'jami-web-common';
import { useContext, useMemo } from 'react';
import { useTranslation } from 'react-i18next';
import { useNavigate } from 'react-router';

import { useAuthContext } from '../contexts/AuthProvider';
import { WebSocketContext } from '../contexts/WebSocketProvider';
import { ConversationContext } from '../contexts/ConversationProvider';
import ChatInterface from '../pages/ChatInterface';
import { useConversationQuery } from '../services/Conversation';
import { translateEnumeration, TranslateEnumerationOptions } from '../utils/translations';
import { AddParticipantButton, ShowOptionsMenuButton, StartAudioCallButton, StartVideoCallButton } from './Button';
import LoadingPage from './Loading';

type ConversationViewProps = {
conversationId: string;
};
const ConversationView = ({ conversationId }: ConversationViewProps) => {
const ConversationView = () => {
const { account } = useAuthContext();
const webSocket = useContext(WebSocketContext);
const [conversation, setConversation] = useState<Conversation | undefined>();
const [isLoading, setIsLoading] = useState(true);
const [error, setError] = useState(false);

const accountId = account.getId();

const conversationQuery = useConversationQuery(conversationId);

useEffect(() => {
if (conversationQuery.isSuccess) {
const conversation = Conversation.from(accountId, conversationQuery.data);
setConversation(conversation);
}
}, [accountId, conversationQuery.isSuccess, conversationQuery.data]);

useEffect(() => {
setIsLoading(conversationQuery.isLoading);
}, [conversationQuery.isLoading]);

useEffect(() => {
setError(conversationQuery.isError);
}, [conversationQuery.isError]);

useEffect(() => {
if (!conversation || !webSocket) {
return;
}
console.log(`set conversation ${conversationId} ` + webSocket);
webSocket.send(WebSocketMessageType.ConversationView, { accountId, conversationId });
}, [accountId, conversation, conversationId, webSocket]);

if (isLoading) {
return <LoadingPage />;
} else if (error || !account || !conversation) {
return <div>Error loading {conversationId}</div>;
}
const { conversationId, conversation } = useContext(ConversationContext);

return (
<Stack height="100%">
Expand All @@ -97,9 +55,9 @@ type ConversationHeaderProps = {
adminTitle: string | undefined;
};

const ConversationHeader = ({ account, members, adminTitle, conversationId }: ConversationHeaderProps) => {
const ConversationHeader = ({ account, members, adminTitle }: ConversationHeaderProps) => {
const { t } = useTranslation();
const navigate = useNavigate();
const { beginCall } = useContext(ConversationContext);

const title = useMemo(() => {
if (adminTitle) {
Expand All @@ -124,14 +82,6 @@ const ConversationHeader = ({ account, members, adminTitle, conversationId }: Co
return translateEnumeration<ConversationMember>(members, options);
}, [account, members, adminTitle, t]);

const startCall = (withVideo = false) => {
let url = `/call/${conversationId}`;
if (withVideo) {
url += '?video=true';
}
navigate(url);
};

return (
<Stack direction="row" padding="16px" overflow="hidden">
<Stack flex={1} justifyContent="center" whiteSpace="nowrap" overflow="hidden">
Expand All @@ -140,8 +90,8 @@ const ConversationHeader = ({ account, members, adminTitle, conversationId }: Co
</Typography>
</Stack>
<Stack direction="row" spacing="20px">
<StartAudioCallButton onClick={() => startCall(false)} />
<StartVideoCallButton onClick={() => startCall(true)} />
<StartAudioCallButton onClick={() => beginCall()} />
<StartVideoCallButton onClick={() => beginCall()} />
<AddParticipantButton />
<ShowOptionsMenuButton />
</Stack>
Expand Down
2 changes: 1 addition & 1 deletion client/src/components/JamiWelcomeLogo.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
import { Stack, StackProps, Typography } from '@mui/material';
import { useTranslation } from 'react-i18next';

import { ReactComponent as JamiLogo } from '../icons/jami-logo-icon.svg';
import { ReactComponent as JamiLogo } from '../icons/jamiLogoIcon.svg';
import { jamiLogoDefaultSize } from '../utils/constants';

interface WelcomeLogoProps extends StackProps {
Expand Down
Loading

0 comments on commit f929a36

Please sign in to comment.