Google DeepMind introduces sign language translation model for mobile devices

A new multilingual model translates sign language movements directly into text for mobile devices. The technology debuts on Pixel 11 devices, starting with American Sign Language.

2 min read

A person using a smartphone to translate sign language gestures into text.
A user demonstrates the sign language translation model on a mobile device.

Google DeepMind has introduced a sign-language-to-text translation model designed to bring sign language features to consumer devices. The technology powers new sign-to-text dictation capabilities in Gboard and Live Transcribe on Pixel 11 devices, starting with American Sign Language to English translation. Additional devices and languages are planned for future releases.

Key facts

Training data
100,000 hours
Sign languages supported
Over 50
American Sign Language data share
Roughly 25%
FLEURS-ASL score
70 BLEURT
Initial deployment devices
Pixel 11
Close-up of a mobile phone displaying real-time text transcription from sign language input.
Close-up view of the mobile interface handling real-time sign language dictation., Source ↗

Overcoming translation challenges

Translating sign languages presents distinct technical hurdles compared to transcribing spoken languages. Sign languages are independent systems with separate grammars and lexicons, requiring true machine translation rather than a simple sequential mapping from sounds to words. Additionally, translation systems must track physical movements across the hands, arms, torso, head, and face at high frame rates.

American Sign Language version | Putting sign language AI into users’ hands

Product demonstration

Video demonstration of American Sign Language translation capabilities.

Early sign language technology often relied on physical accessories like gloves, which failed to capture full-body movements. The new model processes video input by tracking the geometric coordinates of specific points on the signer's body using an on-device tracking tool called MediaPipe Holistic. By sending only these coordinate locations to the server instead of raw video feeds, the system preserves user privacy and discards the original video immediately.

Training and performance

The model is trained on more than 100,000 hours of data spanning over 50 sign languages, with roughly a quarter of the data dedicated to American Sign Language. Rather than using intermediate annotations known as glosses, the system translates coordinate sequences directly into streaming text outputs. This approach removes vocabulary limits and allows translation quality to scale alongside data volume.

On the FLEURS-ASL benchmark — an evaluation framework measuring American Sign Language to English translation quality — the model achieves a zero-shot score of 70 BLEURT. Development teams have also focused on practical usability metrics, including reducing streaming latency, filtering out non-signing inputs, accommodating left-handed signers, and supporting one-handed signing scenarios.

Community collaboration

To guide deployment, the development team established the AI Sign Language Advisory Committee to coordinate with global Deaf organizations and subject-matter experts. The project also relies on foundational research contributions from teams across Google DeepMind and Android, establishing a collaborative approach for future accessibility updates.

Source: Original announcement

Worth a read?

Report

Comments