InteractGesture

InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control

ECCV 2026Workshop · Interactive Social Avatars ↗

Ekkasit Pinyoanuntapong1,2,*, Ajinkya Deogade1, Paul Streli1, Wenjing Zhang1, Joanna Materzynska1, Pu Wang2, Vittorio Ferrari1, Jie Shen1

1 Meta logo
2 University of North Carolina at Charlotte logo

* Work conducted during an internship at Meta.

arXiv GitHub ↗
Four gesture-generation examples: speech-only motion, joint position control, dense trajectory control, and pointing direction control.

Supports multiple applications. Place a joint at a keyframe, draw a continuous trajectory, or direct an arm toward a target.

InteractGesture guides individual joints toward user-defined 3D targets while preserving natural, speech-aligned motion—without retraining the gesture model.

01 / Motivation

The missing control layer

Speech-driven models produce plausible gestures, but animators cannot tell a hand where to land or a finger where to point. Directly fixing poses after generation often looks unnatural, and independently generated chunks break motion around their boundaries.

InteractGesture solves both problems at inference time: spatial errors from the future flow through a differentiable motion decoder, while overlapping chunks remain editable as new audio arrives.

02 / Method

Guide the latent progressively.

The pretrained generator stays frozen. At selected sampling steps, its latent estimate is decoded into SMPL-X joints, compared with the controlled targets, and adjusted using the spatial loss.

InteractGesture architecture showing speech-conditioned sampling, differentiable motion decoding, spatial loss, and latent guidance.
Inference-time guidance Gradients update motion latents—not model weights—so the approach can be added to a pretrained generator.
1

Predict

The speech-conditioned sampler estimates the next motion latent.

2

Decode

A differentiable RVQ-VAE and SMPL-X recover the 3D joints.

3

Guide

Spatial loss gradients move selected joints toward their targets.

03 / Continuous Control

Progressively perturb multiple chunks.

Progressive Chunk Guidance introduces motion chunks with a staggered delay and keeps an active window editable. A target in the incoming chunk can therefore smooth the trajectory that leads into it—without waiting for the full speech sequence.

(a) Sequential: cannot see future (b) Synchronous: accurate, but offline (c) Progressive: accurate control + online
Comparison of sequential, synchronous, and progressive chunk-guidance schedules.
Three schedules Progressive guidance balances online generation with cross-boundary control.

04 / Results

Precise control, natural motion.

On BEAT2, progressive guidance reaches spatial targets far more accurately than sequential guidance, inverse kinematics, or chunk-wise ControlNet—while retaining the best generated-motion FGD.

Qualitative comparison of generated gestures under shared wrist and elbow position controls.
Qualitative comparison. Guided motion follows the requested wrist and elbow locations while preserving a coherent gesture trajectory.

In one sentence

InteractGesture makes pretrained co-speech generators spatially controllable for continuous streaming sequences.

Download paper