Control for neural network analysis

Postdoc position, 24 months

Université Côte d’Azur, J. A. Dieudonné CNRS Lab. (Nice)

Control has recently proved very instrumental for the analysis of neural networks, and for their use in classification. When dealing with very deep residual networks (ResNets), for instance, a relevant approximation is to assume that there is a continuum of layers, indexed by time. In this framework, the composition of a finite number of cells can be interpreted as discretising a continuous ODE, called the neural ODE. In turn, for a large number of layers, the continuous model is meaningful to understand the network properties. The weights, labeled by time, are the controls of the system, and there are several issues that can be efficiently addressed through this point of view. First, in the case of supervised learning: learning a classifier is just learning some input-output map, a standard task in control theory. And meeting the requirements tied to the known data is associated with controllability issues, namely ensemble controllability, ultimately related to controllability on the group of diffeomorphisms on the ambient manifold (Agrachev and Sarychev, 2022; Scagliotti, 2023). Lie bracket techniques are in order for these questions, as well as other more constructive approaches that exploit the structure of the nonlinear activation functions of the original network (Li, Lin and Chen, 2023; Zuazua, 2022; Ruiz-Balet and Zuazua, 2022). Another line of active search to contribute to is the analysis of the convergence of training of neural networks, modelled by neural ODEs, and its relation to the well known turnpike phenomenon in optimal control (Geshkovski and Zuazua, 2022).

Contact

Jean-Baptiste Caillau and Ludovic Rifford

References

Agrachev, A. A.; Sarychev, A. V. Control in the Spaces of Ensembles of Points. SIAM J. Control Optim.58 (2020), no. 3, 1579–1596.

Agrachev, A. A.; Sarychev, A. V. Control on the Manifolds of Mappings with a View to the Deep Learning. J. Dyn. Control Syst. 28 (2021), 989–1008.

Li, Q.; Lin, T; Shen, Z. Deep learning via dynamical systems: An approximation perspective. J. Eur. Math. Soc. 25 (2023), 1671–1709.

Ruiz-Balet, D.; Zuazua, E. Neural ode control for classification, approximation and transport. SIAM Review 65 (2022), no. 3, 735-773.

Scagliotti, A. Deep Learning approximation of diffeomorphisms via linear-control systems. MCRF 13(2023), no. 3, 1226-1257.