A Comprehensive Review of Data-Driven Co-Speech Gesture Generation

Simple item page
dc.contributor.authorNyatsanga, Simbarasheen_US
dc.contributor.authorKucherenko, Tarasen_US
dc.contributor.authorAhuja, Chaitanyaen_US
dc.contributor.authorHenter, Gustav Ejeen_US
dc.contributor.authorNeff, Michaelen_US
dc.contributor.editorBousseau, Adrienen_US
dc.contributor.editorTheobalt, Christianen_US
dc.date.accessioned2023-05-03T06:13:38Z
dc.date.available2023-05-03T06:13:38Z
dc.date.issued2023
dc.description.abstractGestures that accompany speech are an essential part of natural and efficient embodied human communication. The automatic generation of such co-speech gestures is a long-standing problem in computer animation and is considered an enabling technology for creating believable characters in film, games, and virtual social spaces, as well as for interaction with social robots. The problem is made challenging by the idiosyncratic and non-periodic nature of human co-speech gesture motion, and by the great diversity of communicative functions that gestures encompass. The field of gesture generation has seen surging interest in the last few years, owing to the emergence of more and larger datasets of human gesture motion, combined with strides in deep-learning-based generative models that benefit from the growing availability of data. This review article summarizes co-speech gesture generation research, with a particular focus on deep generative models. First, we articulate the theory describing human gesticulation and how it complements speech. Next, we briefly discuss rule-based and classical statistical gesture synthesis, before delving into deep learning approaches. We employ the choice of input modalities as an organizing principle, examining systems that generate gestures from audio, text and non-linguistic input. Concurrent with the exposition of deep learning approaches, we chronicle the evolution of the related training data sets in terms of size, diversity, motion quality, and collection method (e.g., optical motion capture or pose estimation from video). Finally, we identify key research challenges in gesture generation, including data availability and quality; producing human-like motion; grounding the gesture in the co-occurring speech in interaction with other speakers, and in the environment; performing gesture evaluation; and integration of gesture synthesis into applications. We highlight recent approaches to tackling the various key challenges, as well as the limitations of these approaches, and point toward areas of future development.en_US
dc.description.documenttypestar
dc.description.number2
dc.description.sectionheadersState of the Art Reports
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume42
dc.identifier.doi10.1111/cgf.14776
dc.identifier.issn1467-8659
dc.identifier.pages569-596
dc.identifier.pages28 pages
dc.identifier.urihttps://doi.org/10.1111/cgf.14776
dc.identifier.urihttps://diglib.eg.org/handle/10.1111/cgf14776
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.subjectco-speech gestures, gesture generation, deep learning, virtual agents, social robotics,CCS Concepts: Computing methodologies -> Animation; Machine learning; Human-centered computing -> Human computer interaction (HCI)
dc.subjectco-speech gestures
dc.subjectgesture generation
dc.subjectdeep learning
dc.subjectvirtual agents
dc.subjectsocial robotics
dc.subjectComputing methodologies
dc.subjectAnimation
dc.subjectMachine learning
dc.subjectHuman centered computing
dc.subjectHuman computer interaction (HCI)
dc.titleA Comprehensive Review of Data-Driven Co-Speech Gesture Generationen_US
Files
Original bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
v42i2pp569-596_cgf14776.pdf
Size:
877.47 KB
Format:
Adobe Portable Document Format
Download
Collections
EG 2023 - STARs (CGF 42-2)