DESCRIPTION
Motivation and Key Objectives:
At the core of AI lies a set of sophisticated nonlinear, data-driven, or implicitly defined machine learning methods and algorithms. The latter, however, rely largely on “low-dimensional intuitions” and heuristics that are mostly inappropriate and behave strikingly differently in high dimensions. This “curse” of dimensionality notably explains the failure of kernel methods and the difficulty in understanding powerful deep networks. At the same time, the strong pressure and demand for reliable AI tools from both academia and industry are driving an unprecedented need for novel mathematical tools and theoretical guarantees in machine learning algorithms.
Recent advances in high-dimensional statistics—and particularly in random matrices and statistical physics—have provided important insights and striking initial results regarding the understanding and future directions of improvements in machine learning methods in high dimensions: a new approach to kernel methods is emerging, a renewed methodology for semi-supervised learning has emerged, and initial advances in the challenging field of deep neural network theory have appeared. More surprisingly, by exploiting a notion of universality, statistical advances in random matrix theory are increasingly shown to be robust to real-world datasets and thus powerful predictors of practical performance. Many of these recent findings stem from the work of the members of this project.
Chair Description and Program:
The LargeDATA chair builds on these recent breakthroughs and aims to develop a consistent mathematical framework for the high-dimensional analysis, improvement, and redesign of data processing methods ranging from basic to advanced. LargeDATA is a theoretical chair, organized around (i) two methodological work packages focused on the development and application of random matrix theory and statistical physics to machine learning for large datasets, and (ii) a “transfer” work package that explores both more realistic data models and applications to real data (see details below).
The long-term goal is for the chair to hold a leading position in core AI within the MIAI institute, delivering world-class mathematical advances and enhancing the institute’s appeal in the automated processing of large, voluminous, and dynamic data.
ACTIVITIES
WP1. Random Matrix Theory (RMT) for AI. This work package analyzes nonlinear matrix models (kernel random matrices, random graphs, random neural network models, large random tensors) [Comon, Couillet, Tremblay] and their implications for related machine learning algorithms (LS-SVM, SSL, spectral clustering, neural networks, DPP sampling) [Amblard, Barthleme, Couillet]. It also studies the performance of implicit solutions to high-dimensional optimization problems for classification and regression (SVM, logistic regression, GLMM) [Chatelain, Couillet].
WP2. Statistical Physics and Graphs in AI. This work package specifically explores statistical physics (and related heuristics) in scenarios where mathematical developments and techniques are difficult or currently inaccessible [Barthelme, Couillet, Tremblay]. This specifically concerns statistical models of sparse random matrices and graphs, and complex nonlinear learning methods such as deep neural networks.
WP3. Universality Results: From Theory to Practice. This work package develops theoretical foundations for applying high-dimensional statistics to more practical applications (such as in brain signal processing [Barthelme]). The focus will be on concentration of measure theory for neural network analysis [Couillet], sparse matrix and network analyses for clustering and graph mining [Couillet, Tremblay], as well as on heuristic methods to better understand the most challenging real-world data and algorithm models (e.g., for deep learning methods) [Couillet, Tremblay].
CHAIR EVENTS
The chair organizes regular meetings with its main industry partners: HUAWEI Labs Paris, CEA Leti/List, and STMicroelectronics.
A joint GAIA (at GIPSA-lab) and LargeDATA chair seminar is held weekly at GIPSA-lab.
SELECTED LIST OF PUBLICATIONS
- R. Couillet, F. Chatelain, N. Le Bihan,"Two-way kernel matrix puncturing: toward resource-efficient PCA and spectral clustering," International Conference on Machine Learning (ICML’21), virtual conference, 2021. [article|notebook]
- Ch. Séjourné, R. Couillet, P. Comon,"A Large-Dimensional Analysis of Symmetric SNE," IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’21), Toronto, Canada, 2021. [article]
- M. Seddik, C. Louart, R. Couillet, M. Tamaazousti,"The Unexpected Deterministic and Universal Behavior of Large Softmax Classifiers," Artificial Intelligence and Statistics (AISTATS’21), virtual conference, 2021. [article]
- M. Tiomoko, H. Tiomoko, R. Couillet,"Deciphering and Optimizing Multi-Task and Transfer Learning: A Random Matrix Approach," International Conference on Learning Representations (ICLR’21), virtual conference, 2021.Spotlight article. [article]
- Z. Liao, R. Couillet, M. Mahoney,"Sparse Quantized Spectral Clustering," International Conference on Learning Representations (ICLR’21), virtual conference, 2021.Spotlight article. [article]
- M. Seddik, R. Couillet, M. Tamaazousti,"A Random Matrix Analysis of Learning with α-Dropout," International Conference on Machine Learning (ICML’20), Artemiss workshop, Graz, Austria, 2020. [article]
- Z. Liao, R. Couillet, M. Mahoney,"A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent," Conference on Neural Information Processing Systems (NeurIPS’20), Vancouver, Canada, 2020. [article]
- T. Zarrouk, R. Couillet, F. Chatelain, N. Le Bihan,"Performance-Complexity Trade-Off in High-Dimensional Statistics," International Workshop on Machine Learning for Signal Processing (MLSP’20), Espoo, Finland, 2020. [article]
- M. Tiomoko, C. Louart, R. Couillet,"Large-Dimensional Asymptotics of Multi-Task Learning," IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’20), Barcelona, Spain, 2020. [article]
- L. Dall’Amico, R. Couillet, N. Tremblay, “A unified framework for spectral clustering in sparse graphs,” Journal of Machine Learning Research, vol. 22, no. 187, pp. 1–56, 2022. [article]
- L. Dall’Amico, R. Couillet, N. Tremblay,"Nishimori meets Bethe: a spectral method for node classification in sparse weighted graphs,"(to appear in) Journal of Statistical Mechanics: Theory and Experiment, 2021. [article]
- L. Dall’Amico, R. Couillet, N. Tremblay, “Community detection in sparse time-evolving graphs with a dynamical Bethe-Hessian,” Conference on Neural Information Processing Systems (NeurIPS’20), Vancouver, Canada, 2020. [article]
- L. Dall’Amico, R. Couillet, N. Tremblay,"Optimal Laplacian Regularization for Sparse Spectral Community Detection," IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’20), Barcelona, Spain, 2020. [article|video]
- R. Couillet, Y. Cinar, E. Gaussier, M. Imran,"Word Representations Concentrate and This is Good News!", SIGNLL Conference on Computational Natural Language Learning (CoNLL’20), virtual conference, 2020. [article]
- M. Seddik, R. Couillet, M. Tamaazousti,"Random Matrix Theory Proves that Deep Learning Representations of GAN Data Behave as Gaussian Mixtures," International Conference on Machine Learning (ICML’20), Graz, Austria, 2020. [article]
