Google DeepMind introduces sign language translation model for mobile devices

Google DeepMind has introduced a sign-language-to-text translation model designed to bring sign language features to consumer devices. The technology powers new sign-to-text dictation capabilities in Gboard and Live Transcribe on Pixel 11 devices, starting with American Sign Language to English translation. Additional devices and languages are planned for future releases.

Overcoming translation challenges

Translating sign languages presents distinct technical hurdles compared to transcribing spoken languages. Sign languages are independent systems with separate grammars and lexicons, requiring true machine translation rather than a simple sequential mapping from sounds to words. Additionally, translation systems must track physical movements across the hands, arms, torso, head, and face at high frame rates.

Early sign language technology often relied on physical accessories like gloves, which failed to capture full-body movements. The new model processes video input by tracking the geometric coordinates of specific points on the signer's body using an on-device tracking tool called MediaPipe Holistic. By sending only these coordinate locations to the server instead of raw video feeds, the system preserves user privacy and discards the original video immediately.