Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model

Salil Deena; Shaobo Hou; Aphrodite Galata

doi:10.1145/1891903.1891942

Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model

Salil Deena, Shaobo Hou, Aphrodite Galata

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We present a novel approach to speech-driven facial animation using a non-parametric switching state space model based on Gaussian processes. The model is an extension of the shared Gaussian process dynamical model, augmented with switching states. Audio and visual data from a talking head corpus are jointly modelled using the proposed method. The switching states are found using variable length Markov models trained on labelled phonetic data. We also propose a synthesis technique that takes into account both previous and future phonetic context, thus accounting for coarticulatory effects in speech. © 2010 ACM.

Original language	English
Title of host publication	International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010\|Int. Conf. Multimodal Interfaces Workshop Mach. Learn. Multimodal Interact., ICMI-MLMI
Publisher	Association for Computing Machinery
ISBN (Print)	9781450304146
DOIs	https://doi.org/10.1145/1891903.1891942
Publication status	Published - 2010
Event	1st International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010 - Beijing Duration: 1 Jul 2010 → …

Conference

Conference	1st International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010
City	Beijing
Period	1/07/10 → …

Keywords

artificial talking head
speech-driven facial animation
visual speech synthesis

Access to Document

10.1145/1891903.1891942

Cite this

Deena, S., Hou, S., & Galata, A. (2010). Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model. In International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010|Int. Conf. Multimodal Interfaces Workshop Mach. Learn. Multimodal Interact., ICMI-MLMI Association for Computing Machinery. https://doi.org/10.1145/1891903.1891942

Deena, Salil ; Hou, Shaobo ; Galata, Aphrodite. / Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model. International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010|Int. Conf. Multimodal Interfaces Workshop Mach. Learn. Multimodal Interact., ICMI-MLMI. Association for Computing Machinery, 2010.

@inproceedings{44eb8d437240406c886c1cb544bf3ff3,

title = "Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model",

abstract = "We present a novel approach to speech-driven facial animation using a non-parametric switching state space model based on Gaussian processes. The model is an extension of the shared Gaussian process dynamical model, augmented with switching states. Audio and visual data from a talking head corpus are jointly modelled using the proposed method. The switching states are found using variable length Markov models trained on labelled phonetic data. We also propose a synthesis technique that takes into account both previous and future phonetic context, thus accounting for coarticulatory effects in speech. {\textcopyright} 2010 ACM.",

keywords = "artificial talking head, speech-driven facial animation, visual speech synthesis",

author = "Salil Deena and Shaobo Hou and Aphrodite Galata",

year = "2010",

doi = "10.1145/1891903.1891942",

language = "English",

isbn = "9781450304146",

booktitle = "International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010|Int. Conf. Multimodal Interfaces Workshop Mach. Learn. Multimodal Interact., ICMI-MLMI",

publisher = "Association for Computing Machinery",

address = "United States",

note = "1st International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010 ; Conference date: 01-07-2010",

}

Deena, S, Hou, S & Galata, A 2010, Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model. in International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010|Int. Conf. Multimodal Interfaces Workshop Mach. Learn. Multimodal Interact., ICMI-MLMI. Association for Computing Machinery, 1st International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010, Beijing, 1/07/10. https://doi.org/10.1145/1891903.1891942

Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model. / Deena, Salil; Hou, Shaobo; Galata, Aphrodite.
International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010|Int. Conf. Multimodal Interfaces Workshop Mach. Learn. Multimodal Interact., ICMI-MLMI. Association for Computing Machinery, 2010.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model

AU - Deena, Salil

AU - Hou, Shaobo

AU - Galata, Aphrodite

PY - 2010

Y1 - 2010

N2 - We present a novel approach to speech-driven facial animation using a non-parametric switching state space model based on Gaussian processes. The model is an extension of the shared Gaussian process dynamical model, augmented with switching states. Audio and visual data from a talking head corpus are jointly modelled using the proposed method. The switching states are found using variable length Markov models trained on labelled phonetic data. We also propose a synthesis technique that takes into account both previous and future phonetic context, thus accounting for coarticulatory effects in speech. © 2010 ACM.

AB - We present a novel approach to speech-driven facial animation using a non-parametric switching state space model based on Gaussian processes. The model is an extension of the shared Gaussian process dynamical model, augmented with switching states. Audio and visual data from a talking head corpus are jointly modelled using the proposed method. The switching states are found using variable length Markov models trained on labelled phonetic data. We also propose a synthesis technique that takes into account both previous and future phonetic context, thus accounting for coarticulatory effects in speech. © 2010 ACM.

KW - artificial talking head

KW - speech-driven facial animation

KW - visual speech synthesis

U2 - 10.1145/1891903.1891942

DO - 10.1145/1891903.1891942

M3 - Conference contribution

SN - 9781450304146

BT - International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010|Int. Conf. Multimodal Interfaces Workshop Mach. Learn. Multimodal Interact., ICMI-MLMI

PB - Association for Computing Machinery

T2 - 1st International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010

Y2 - 1 July 2010

ER -

Deena S, Hou S, Galata A. Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model. In International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 2010|Int. Conf. Multimodal Interfaces Workshop Mach. Learn. Multimodal Interact., ICMI-MLMI. Association for Computing Machinery. 2010 doi: 10.1145/1891903.1891942

Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this