Controlled descent training

DOI10.1002/RNC.7194MaRDI QIDQ6197966zbMATH OpenOpenAlexFDO

Authors Balázs Varga, Rebecka Jörnsten, B. Kulcsár

Publication date 20 March 2024

Published in International Journal of Robust and Nonlinear Control (Search for Journal in Brave)

Copyright license Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International

Full work available at URL https://arxiv.org/abs/2303.09216

zbMATH Keywords

convergent learninggradient decent traininglabel augmentationlabel selectionneural tangent kerneloptimal labels

Mathematics Subject Classification ID

Computer science (68-XX)

Abstract: In this work, a novel and model-based artificial neural network (ANN) training method is developed supported by optimal control theory. The method augments training labels in order to robustly guarantee training loss convergence and improve training convergence rate. Dynamic label augmentation is proposed within the framework of gradient descent training where the convergence of training loss is controlled. First, we capture the training behavior with the help of empirical Neural Tangent Kernels (NTK) and borrow tools from systems and control theory to analyze both the local and global training dynamics (e.g. stability, reachability). Second, we propose to dynamically alter the gradient descent training mechanism via fictitious labels as control inputs and an optimal state feedback policy. In this way, we enforce locally $mathcal{H}_2$ optimal and convergent training behavior. The novel algorithm, extit{Controlled Descent Training} (CDT), guarantees local convergence. CDT unleashes new potentials in the analysis, interpretation, and design of ANN architectures. The applicability of the method is demonstrated on standard regression and classification problems.

Cites work

• Geometry of deep learning. A signal processing perspective
• scientific article; zbMATH DE number 3435336 (Why is no real title available?)
• Nonlinear control systems.
• Nonlinear systems.
• Reachability Analysis of Nonlinear Systems Using Matrix Measures
• The interpolation phase transition in neural networks: memorization and generalization under lazy training
• Time-optimal control of large-scale systems of systems using compositional optimization
• When and why PINNs fail to train: a neural tangent kernel perspective

This page was built for publication: Controlled descent training

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6197966)