-
Sign Language is a form of communication by people who are deaf, hard of hearing, and non-verbal. This mainly employs signs made by moving the hands. However, not only hands, but also palm orientation, movement, location, and expression/non-manual signals, ... lead to the good performance of the Sign Language Recognition System.
-
Over 466 million people(5%, might reach 900M in 2050) are speech or hearing impaired, and 80% of them are semi-illiterate or illiterate(WHO).
- In US, ASL is a native language for around 2,50,000-5,00,000 people.
- In China, Chinese Sign Language is being used in China by approximately 1M to 20M deaf people.
- In UK, Approximately 1,50,000 people use British Sign Language (BSL).
- In India, approximately 1.5 million signers use Indo-Pakistani Sign Language.
-
One approach supporting phone call: A relay service (text and video based)
- Takes extra time, complex settings and inconsistent call
- Only person should talk or type at a time
- Sensitive and private information or data & image might be leaked
-
SLR is really necessary but not easy to create a tool that meets daily conversation (including online) and there are gaps to fill in such as real-time and natural output decoding, sensor selection and placement optimization, and robustness with non-uniform background environments ...
-
Proposed pipeline for making phone call:
-
Wearable System for Recognizing American Sign Language in Real-Time Using IMU and Surface EMG Sensors (2016):
- Fusion IMU and sEMG signals and give suitable feature extraction and selection
- Adaptive auto-segmentation to extract periods during which signs are performed using sEMG
- Proved sEMG is useful to combine by experiments
-
DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation (2017):
- Uses Leap Motion – an infrared light-based sensing device extracting the skeleton joints information of fingers, palms and forearms
- Propose 2 branch models for corresponding hands and fusion
-
SignSpeaker: A Real-time, High-Precision SmartWatch-based Sign Language Translator (2019)
- Using portable and lightweight smartwatch and mobile phone
- Supporting both fingerspelling and sentence-level using bilstm given extracted spectrogram
- Additional TTS engine
-
SonicASL: An Acoustic-based Sign Language Gesture Recognizer Using Earphones (2021)
- First method for using Sonic wav(inaudible) + earphones
- Propose CNN-LSTM CTC
- Support TTS engine
-
SignFi: Sign Language Recognition Using WiFi (2018 - IMWUT)
- Collects CSI measurements to capture wireless signal characteristics of sign gestures.
- Raw CSI measurements are pre-processed to remove noises and recover CSI changes over sub-carriers and sampling time
- Only word-level recognition and relatively low accuracy in new-user
-
MyoSign: Enabling End-to-End Sign Language Recognition with Wearables (IUI 2019)
- Emphasize the application of EMG in SLR
- Use CNN for feature extraction and then BiLSTM + CTC
-
GASLA: Enhancing the Applicability of Sign Language Translation (Infocom 2022)
- Introduce a method for collecting suitable word and sentence data (synthesized by words)
- Also use BiLSTM and CTC and use spectrogram as input
- Focus on the translation task, which is a higher version of the recognition task when outputting spoken text instead of sign text.
-
WearSign: Pushing the Limit of Sign Language Translation Using Inertial and EMG Wearables (IMWUT 2022)
- Leverages a smartwatch and an armband of ElectroMyoGraphy (EMG) sensors to capture the sophisticated sign gestures to do translation task
- Introduce a training method with 2 CTC layers and encoder-decoder architecture together with an attention mechanism.
- Borrow the idea of back-translation and leverage the much more available spoken language data to synthesize the paired sign language data