Google DeepMind introduces sign language translation model for mobile devices
A new multilingual model translates sign language movements directly into text for mobile devices. The technology debuts on Pixel 11 devices, starting with American Sign Language.
2 min read, Thursday, 13 August

Google DeepMind has introduced a sign-language-to-text translation model designed to bring sign language features to consumer devices. The technology powers new sign-to-text dictation capabilities in Gboard and Live Transcribe on Pixel 11 devices, starting with American Sign Language to English translation. Additional devices and languages are planned for future releases.
Key facts
- Training data
- 100,000 hours
- Sign languages supported
- Over 50
- American Sign Language data share
- Roughly 25%
- FLEURS-ASL score
- 70 BLEURT
- Initial deployment devices
- Pixel 11

Overcoming translation challenges
Translating sign languages presents distinct technical hurdles compared to transcribing spoken languages. Sign languages are independent systems with separate grammars and lexicons, requiring true machine translation rather than a simple sequential mapping from sounds to words. Additionally, translation systems must track physical movements across the hands, arms, torso, head, and face at high frame rates.
American Sign Language version | Putting sign language AI into users’ hands
Video demonstration of American Sign Language translation capabilities.
Early sign language technology often relied on physical accessories like gloves, which failed to capture full-body movements. The new model processes video input by tracking the geometric coordinates of specific points on the signer's body using an on-device tracking tool called MediaPipe Holistic. By sending only these coordinate locations to the server instead of raw video feeds, the system preserves user privacy and discards the original video immediately.
Training and performance
The model is trained on more than 100,000 hours of data spanning over 50 sign languages, with roughly a quarter of the data dedicated to American Sign Language. Rather than using intermediate annotations known as glosses, the system translates coordinate sequences directly into streaming text outputs. This approach removes vocabulary limits and allows translation quality to scale alongside data volume.
On the FLEURS-ASL benchmark — an evaluation framework measuring American Sign Language to English translation quality — the model achieves a zero-shot score of 70 BLEURT. Development teams have also focused on practical usability metrics, including reducing streaming latency, filtering out non-signing inputs, accommodating left-handed signers, and supporting one-handed signing scenarios.
Community collaboration
To guide deployment, the development team established the AI Sign Language Advisory Committee to coordinate with global Deaf organizations and subject-matter experts. The project also relies on foundational research contributions from teams across Google DeepMind and Android, establishing a collaborative approach for future accessibility updates.
Source: Original announcement
Worth a read?



Comments