Interdisciplinary Institute in Artificial Intelligence

DESCRIPTION

The "Bayesian Cognition and Machine Learning for Speech Communication" chair, part of the Grenoble MIAI institute, brings together researchers whose areas of expertise lie in the fields of speech communication, cognition, machine learning, and probabilistic modeling of sensorimotor systems. The team members are from Gipsa-lab (UMR CNRS 5216) and LPNC (UMR CNRS 5105), two laboratories at UGA and Grenoble INP-UGA. The goal of the chair is to develop a comprehensive computational model of speech production and speech perception—that is, a system capable of learning how to speak and perceive speech from examples provided by the environment. To this end, an original approach is proposed that combines the algorithmic and mathematical framework of data-driven deep learning with hypothesis-driven probabilistic modeling. This approach was developed to design more interpretable—and thus more explainable and transferable—models, with faster and more cost-effective implementations, as well as greater robustness and versatility. Our goal is to build models of speech communication that achieve the state-of-the-art performance of current deep learning-based systems while drastically reducing the amount of training data required.

SCIENTIFIC APPROACH

Figure 1- Processes and variables included in the joint model of speech and speech perception

Jointly modeling the speech production and perception processes involves designing models of the relationships between the various variables involved in those processes, namely motor/control variables, multisensory variables, and linguistic/phonological variables (Figure 1). Over the years, the members of the group have explored two complementary approaches to achieve this. First, a hypothesis-driven probabilistic programming approach (Tenenbaum et al., 2011; Bessière et al., 2013) has been used to explicitly design a set of multidimensional probabilistic functions that link the variables. These probabilistic models are derived from theoretical hypotheses regarding the physical mechanisms, neurocognitive processing, and representations of speech production and speech perception in humans. This has led to a number of significant insights concerning speech perception under adverse conditions (Moulin-Frier et al., 2012; Laurent et al., 2015), variability and robustness in speech production (Patri et al., 2015, 2018), speech representations in the brain (Barnaud et al., 2018), and the emergence of sound systems in a society of communicating agents (Moulin-Frier et al., 2015; Schwartz et al., 2016). Secondly, data-driven deep-learning algorithmic and mathematical frameworks—such as artificial deep neural networks—make it possible to establish a direct mapping, i.e., a deterministic regression, between subsets of variables. This approach formed the basis for the development of efficient systems for speech processing, voice conversion, noise reduction, feature extraction, acoustic-articulatory inversion, speech synthesis, and brain-computer interfaces (Hueber et al., 2015; Hueber & Bailly, 2016; Bocquelet et al., 2016; Fabre et al., 2017; Girin et al., 2017; Schultz et al., 2017). The computational challenge addressed in the chair’s research program is to draw on the strengths of these two complementary approaches to develop a computational system capable of learning how to produce and perceive speech from examples provided by the environment. These two approaches have recently begun to converge with the emergence of deep generative models, such as variational autoencoders (VAE) and generative adversarial networks (GAN). Essentially, parameters characterizing the probability distributions of data are encoded or mapped within deep neural networks. These models can serve as supervised probabilistic priors within a more general (probabilistic) model. They provide efficient methods for extracting, modeling, and manipulating the low-dimensional latent space that represents the structure of the high-dimensional data.

RESEARCH PROGRAM

Figure 2- Components of the research program

In this context, it is crucial to clarify how complex multidimensional probabilistic relationships between speech production and perception variables can be learned, and to make this learning more efficient. A systematic exploration of the numerous multidimensional spaces involved in these processes is not feasible. This is why a cognition-based approach to probabilistic programming is preferred, as it provides a framework for developing computational models and learning processes in relation to existing knowledge in physics (aeroacoustics, biomechanics) and neuroscience/psychology (sensory-motor control of speech, language representations in the brain, and the developmental timeline of speech and language acquisition in infancy and childhood). The overall program of the project consists of combining the formalization of the speech communication model—made possible by the probabilistic programming framework—with the learning capabilities of advanced machine learning methods to address these learning challenges, develop new learning models, and evaluate them (Figure 2).

More specifically, over the next four years, research will be organized around three main topics:
 

• Developing a computational agent capable of learning speech representations from raw speech data in a weakly supervised setting. This agent will include an articulatory model of the human vocal tract, an articulatory-to-acoustic synthesis system, and a learning architecture that combines deep learning algorithms with developmental principles inspired by cognitive science. The first step will involve designing, implementing, and testing a “deep” version of a Bayesian computational model of speech communication called COSMO (Moulin-Frier et al., 2012, 2015; Laurent et al., 2017; Barnaud et al., 2019), in which some of the probability distributions are implemented using generative neural models (e.g., VAE, GAN). The second stage will consist of reimplementing the speech communication agent entirely within an end-to-end neural architecture. PhD project by Marc-Antoine Georges.

 

• Developing a general methodology for incremental sequence-to-sequence mapping. This will require the development of end-to-end classification and regression neural networks capable of generating chunks of output data on the fly, based solely on a partial observation of the input data. Possible approaches include: (1) Predicting “the future” of the output sequence online based on “the past and present” of the input sequence, with an acceptable tolerance for possible errors, or (2) automatically learning from the data an optimal “waiting policy” that prevents the model from outputting data when uncertainty is too high. This will be applied to address two speech processing problems: Incremental Text-to-Speech synthesis, in which speech is synthesized as the user types the text, and Incremental Speech Enhancement, in which unintelligible portions of the speech signal are replaced on-the-fly with reconstructed portions. This work is carried out in close collaboration with the MIAI Chair in “Artificial Intelligence & Language” held by Laurent Besacier. PhD project by Brooke Stephenson.

 

• Developing a model of speech production that incorporates real-time processing of feedback information by extending our current Bayesian model for speech motor planning, known as Bayesian GEPPETO (Patri et al., 2015). This involves first learning a “dynamical internal model” of the speech production apparatus, which predicts articulatory movements and the spectro-temporal acoustic properties of speech production from time-varying motor commands, and predicts with minimal delay the probability of achieving the intended sensory goals of speech production with the appropriate timing. This learning relies on simulations of a biomechanical model of the vocal tract, which generates articulatory movements from motor commands and whose computational complexity will be reduced using machine-learning-based methods of model order reduction (PhD project by Maxime Calka). Then, assuming that the brain functions as an optimal controller/estimator, research will focus on how sensory feedback and internal prediction of the speech signal concurrently guide speech production, enable online error correction, and shape speech perception (upcoming PhD project).

This work is also linked to research being conducted as part of an international project carried out by members of the research group in collaboration with Anne-Lise Giraud and Itsaso Olasagasti of the “Auditory, Speech, and Language Neuroscience” group at UNIGE (University of Geneva, Switzerland), and funded by the IDEX “Université Grenoble Alpes Université de l’Innovation” (Bio-Bayes project – IDEX ISP19). In this project, Bayesian hierarchical and predictive models of speech communication will be developed that account for observations of neuronal oscillatory systems in the brain (Ph.D. project of Mamady Nabé).

The impact of this research program on the development of speech technologies will be evaluated in terms of the speed and completeness of learning, as well as the amount of data required to achieve a satisfactory level of learning. Furthermore, in the context of deep learning, analyzing the representations learned by deep artificial neural networks from raw articulatory, acoustic, and linguistic data could provide important insights into the sensory-motor representations potentially encoded in the human brain.

SELECTED LIST OF PUBLICATIONS

• Marc-Antoine Georges, Laurent Girin, Jean-Luc Schwartz, Thomas Hueber. Learning robust speech representation with an articulatory-regularized variational autoencoder. Interspeech 2021—22nd Annual Conference of the International Speech Communication Association, Aug. 2021, Brno, Czech Republic.
• Brooke Stephenson, Thomas Hueber, Laurent Girin, Laurent Besacier. “Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Input.” Interspeech 2021—22nd Annual Conference of the International Speech Communication Association, Aug. 2021, Brno, Czech Republic. pp. 3865–3869.
• Xiaoyu Bie, Laurent Girin, Simon Leglaive, Thomas Hueber, Xavier Alameda-Pineda. “A Benchmark of Dynamical Variational Autoencoders Applied to Speech Spectrogram Modeling.” Interspeech 2021—22nd Annual Conference of the International Speech Communication Association, Aug. 2021, Brno, Czech Republic. pp. 1–5.
• Pierre Baraduc, Tsiky Rakatomalala, Pascal Perrier. Tongue Motor Control: Deriving Articulator Trajectories and Muscle Activation Patterns from an Optimization Principle. Neural Control of Movement 2021 (Virtual Conference). Abstract. April 2021
• Pascal Perrier, Ny-Tsiky Rakotomalala, Pierre Baraduc. Some Thoughts on Trajectory Formation in Speech Production. Neural Bases of Speech Production, 2021, UCSF, San Francisco, Virtual Symposium, Invited Conference, May 2021.
• Mamady Nabé, Jean-Luc Schwartz, Julien Diard. COSMO-Onset: A Neurally-Inspired Computational Model of Spoken Word Recognition, Combining Top-Down Prediction and Bottom-Up Detection of Syllabic Onsets. Frontiers in Systems Neuroscience, Frontiers, 2021, 15, pp. 653975.
• Girin, L., Leglaive, S., Bie, X., Diard, J., Hueber, T., & Alameda-Pineda, X. (2021). Dynamical Variational Autoencoders: A Comprehensive Review. Foundations and Trends in Machine Learning. To be published in December.
• Georges, M.-A., Badin, P., Diard, J., Girin, L., Schwartz, J.-L., Hueber, T. (2020). “Toward an Articulatory-Driven Neural Vocoder for Speech Synthesis.” ISSP 2020—12th International Seminar on Speech Production, Dec. 2020, Providence (virtual), United States.
• Baraduc, P., Perrier, P. (2020). Tongue motor control stability: integrating feedback, dynamical internal representation, and optimal planning. ISSP 2020—12th International Seminar on Speech Production, Dec. 2020, Providence (virtual), United States.
• Calka, M., Perrier, P., Ohayon, J., Grivot Boichon, C., Rochette, Payan, Y. (2020). Real-time simulations of human tongue movements using a reduced-order model of a nonlinear dynamic biomechanical model. Computer Methods in Biomechanics and Biomedical Engineering, Taylor & Francis, 2020, 23 (sup1), pp. S55–S57.
• Stephensen, B., Besacier, L., Girin, L., Hueber, T. (2020). What the Future Brings: Investigating the Impact of Lookahead for Incremental Neural TTS. Proceedings of Interspeech 2020 (pp. 215–219). October 25–29, 2020, Shanghai, China.
• Girin, L., Leglaive, S., Bie, X., Diard, J., Hueber, T., & Alameda-Pineda, X. (2020). Dynamical Variational Autoencoders: A Comprehensive Review. ArXiv preprint arXiv:2008.12595.
• Calka, M., Perrier, P., Ohayon, J., Grivot-Boichon, C., Rochette, M., Payan, Y. (2021).
• Machine-learning-based model order reduction of a biomechanical model of the human tongue. Computer Methods and Programs in Biomedicine, Vol. 198, 105786.
• Hueber, T., Tatulli, E., Girin, L., & Schwartz, J-L. (2020). Evaluating the potential gain of auditory and audiovisual speech predictive coding using deep learning. Neural Computation, vol. 32(3), 596–625. https://doi.org/10.1162/neco_a_01264.
• Patri, J. F., Ostry, D. J., Diard, J., Schwartz, J. L., Trudeau-Fisette, P., Savariaux, C., & Perrier, P. (2020). Speakers are able to categorize vowels based on tongue somatosensation. Proceedings of the National Academy of Sciences, 117(11), 6255–6263.
• Patri, J.F., Diard, J., & Perrier, P. (2019). Modeling sensory preference in speech motor planning: a Bayesian modeling framework. Frontiers in Psychology, 10, Article 2339.

 

EXPECTED OUTCOMES

The research program, which combines cognitive plausibility and alignment with ground-truth data, should lead to significant improvements in three key areas:

• "More explainable and transferable": Linking data-driven machine learning approaches with cognitive and developmental assumptions should make emerging features and structures more explainable and interpretable. This should facilitate the assessment of their limitations in applicability, the prediction of their errors, and the identification of ways to improve their performance.
• "Faster and more economical": Implementing plausible developmental sequences and plausible hierarchies in the model structure should promote learning transfer, ensure faster learning, and lead to quicker convergence. It should enable the models to learn using smaller datasets and promote efficient hardware processing (by leveraging mechanisms such as predictive coding, attentional filtering, or multiplex coding).
• “More robust and versatile”: The generative nature of the implemented models and the appropriateness of the developmental schedules used should enable the models to process atypical or noisy speech through internally generated outputs, and conversely to adapt their own speech productions in response to perturbations. This should result in natural and coherent variability that is constrained and structured by the properties of the modeled system.

INDUSTRIAL PARTNERS

• ProBayes, 38330 Montbonnot
• ANSYS France, 69100 Villeurbanne

NATIONAL AND INTERNATIONAL COLLABORATIONS

• UNIGE, Geneva, Switzerland: Anne-Lise Giraud, auditory processing and neurophysiological modeling (Grant from IDEX “Université Grenoble Alpes Université de l’Innovation”)

• TIMC-IMAG, UGA: Yohan Payan, Order Reduction of Biomechanical Models of Speech Articulators (CIFRE doctoral grant from ANSYS France and ANRT)