AgentHands: generating interactive hand gestures for spatially grounded agent conversations in XR

As artificial intelligence assistants move from flat screens into immersive platforms like Android XR, developers face a new challenge. Flat interfaces rely on visual bounding boxes on a camera feed to highlight objects. In a 3D world, however, users need a more natural and embodied way to interact with digital companions.

To bridge this gap, researchers introduced AgentHands, a system that equips conversational agents with synchronized hand gestures. The research was published at CHI 2026.

The AgentHands taxonomy

To determine what makes a virtual hand legible in a three-dimensional environment, the team developed a multi-dimensional taxonomy. This framework defines how an agent should use its hands to ground a conversation in the user's physical space across six dimensions:

• Handedness and gesture, choosing between one or two hands and selecting forms like a palm or a cylindrical grip.
• Spatiality, using XR depth to place hands mid-air, object-anchored, or user-relative.
• Temporal dynamics and visual effects, utilizing animated motions and effects like a red glow for warnings.

System workflow

The core innovation maps the high-level reasoning of large language models into precise physical motions. First, a lightweight object registration module uses eye gaze and scene reconstruction to let users tag items into a 3D registry.

Next, the system draws from a hand gesture library containing three semantic categories: deictic for referencing, iconic for depicting actions, and expression for conveying social cues. When a user asks a question, the backend model generates a response that includes inline gesture events attached to specific trigger words. A local parser on the headset then coordinates text-to-speech playback with the animation engine, ensuring the hands perform gestures in perfect sync with the spoken words.

User study and results

A within-subjects user study with twelve participants compared the prototype to a speech-only baseline across two tasks: caring for an orchid and operating a 3D printer. The results showed that the combination of extended reality and co-speech gestures significantly improved spatial grounding and made complex actions easier to follow. Participants also reported that warnings were much more noticeable when paired with visual effects, and overall cognitive load was reduced.