HyperAIHyperAI

Command Palette

Search for a command to run...

MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models

Zunnan Xu* Yukang Lin* Haonan Han* Sicheng Yang Ronghui Li Yachao Zhang† Xiu Li†

Abstract

Gesture synthesis is a vital realm of human-computer interaction, withwide-ranging applications across various fields like film, robotics, andvirtual reality. Recent advancements have utilized the diffusion model andattention mechanisms to improve gesture synthesis. However, due to the highcomputational complexity of these techniques, generating long and diversesequences with low latency remains a challenge. We explore the potential ofstate space models (SSMs) to address the challenge, implementing a two-stagemodeling strategy with discrete motion priors to enhance the quality ofgestures. Leveraging the foundational Mamba block, we introduce MambaTalk,enhancing gesture diversity and rhythm through multimodal integration.Extensive experiments demonstrate that our method matches or exceeds theperformance of state-of-the-art models.


Build AI with AI

From idea to launch — accelerate your AI development with free AI co-coding, out-of-the-box environment and best price of GPUs.

AI Co-coding
Ready-to-use GPUs
Best Pricing

HyperAI Newsletters

Subscribe to our latest updates
We will deliver the latest updates of the week to your inbox at nine o'clock every Monday morning
Powered by MailChimp