Robust and resource-efficient identification of two hidden layer neural networks

DOI10.1007/S00365-021-09550-5zbMATH Open1504.65042arXiv1907.00485OpenAlexW3176108623MaRDI QIDQ2117339FDOQ2117339

Authors: Massimo Fornasier, Timo Klock, Michael Rauchensteiner 

Publication date: 21 March 2022

Published in: Constructive Approximation (Search for Journal in Brave)

Abstract: We address the structure identification and the uniform approximation of two fully nonlinear layer neural networks of the type $f(x)=1^{T}h(B^{T}g(A^{T}x))$ on $mathbb{R}^d$ from a small number of query samples. We approach the problem by sampling actively finite difference approximations to Hessians of the network. Gathering several approximate Hessians allows reliably to approximate the matrix subspace $mathcalW$ spanned by symmetric tensors $a_{1}otimes{a}_1,dots,a_{m_0}otimes{a}_{m_0}$ formed by weights of the first layer together with the entangled symmetric tensors $v_{1}otimes{v}_1,dots,v_{m_1}otimes{v}_{m_1}$, formed by suitable combinations of the weights of the first and second layer as $v_{e}ll=A{G}_0{b}_{e}ll/|A{G}_0{b}_{e}ll{|}_2$, $ellin[m_1]$, for a diagonal matrix $G_0$ depending on the activation functions of the first layer. The identification of the 1-rank symmetric tensors within $mathcalW$ is then performed by the solution of a robust nonlinear program. We provide guarantees of stable recovery under a posteriori verifiable conditions. We further address the correct attribution of approximate weights to the first or second layer. By using a suitably adapted gradient descent iteration, it is possible then to estimate, up to intrinsic symmetries, the shifts of the activations functions of the first layer and compute exactly the matrix $G_0$. Our method of identification of the weights of the network is fully constructive, with quantifiable sample complexity, and therefore contributes to dwindle the black-box nature of the network training phase. We corroborate our theoretical results by extensive numerical experiments.

Full work available at URL: https://arxiv.org/abs/1907.00485

Recommendations

• Reconstructing a neural net from its output
• Efficient estimation of neural weights by polynomial approximation
• Neural network identifiability for a family of sigmoidal nonlinearities
• Affine symmetries and neural network identifiability
• On the approximation by neural networks with bounded number of neurons in hidden layers

zbMATH Keywords

framesdeep neural networksactive samplingdeparametrizationnonconvex optimization on matrix spacesexact identifiability

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Nonconvex programming, global optimization (90C26) Algorithms for approximation of functions (65D15)

Cites Work

• Weak convergence and empirical processes. With applications to statistics
• Estimation of the mean of a multivariate normal distribution
• Semiparametric least squares (SLS) and weighted SLS estimation of single-index models
• Title not available (Why is that?)
• High-dimensional probability. An introduction with applications in data science
• Title not available (Why is that?)
• Understanding machine learning. From theory to algorithms
• Direct estimation of the index coefficient in a single-index model
• On Principal Hessian Directions for Data Visualization and Dimension Reduction: Another Application of Stein's Lemma
• Tensor Rank and the Ill-Posedness of the Best Low-Rank Approximation Problem
• Title not available (Why is that?)
• Perturbation bounds in connection with singular value decomposition
• A mathematical introduction to compressive sensing
• Greed is Good: Algorithmic Results for Sparse Approximation
• Most tensor problems are NP-hard
• Finite normalized tight frames
• Learning functions of few arbitrary linear parameters in high dimensions
• Classes of finite equal norm Parseval frames
• Title not available (Why is that?)
• Neural Network Learning
• Title not available (Why is that?)
• Tensor rank is NP-complete
• Entropy and sampling numbers of classes of ridge functions
• Title not available (Why is that?)
• Title not available (Why is that?)
• Approximation by Ridge Functions and Neural Networks
• Capturing ridge functions in high dimensions from point queries
• Active subspaces. Emerging ideas for dimension reduction in parameter studies
• Active subspace methods in theory and practice: applications to kriging surfaces
• Finding a low-rank basis in a matrix subspace
• Title not available (Why is that?)
• Interpolation by ridge polynomials and its application in neural networks
• Breaking the curse of dimensionality with convex neural networks
• Reconstructing a neural net from its output
• Size-independent sample complexity of neural networks
• DeepStack: expert-level artificial intelligence in heads-up no-limit poker
• Provable approximation properties for deep neural networks
• High-dimensional covariance decomposition into sparse Markov and independence models
• Deep Neural Network Approximation Theory
• Robust and resource efficient identification of shallow neural networks by fewest samples
• Energy Propagation in Deep Convolutional Neural Networks

Cited In (6)

• Stable recovery of entangled weights: towards robust identification of deep neural networks from minimal samples
• Affine symmetries and neural network identifiability
• Efficient Identification of Butterfly Sparse Matrix Factorizations
• Parameter identifiability of a deep feedforward ReLU neural network
• Information theory and recovery algorithms for data fusion in Earth observation
• Approximate real symmetric tensor rank

Uses Software

• ImageNet
• DeepStack
• AlexNet

This page was built for publication: Robust and resource-efficient identification of two hidden layer neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2117339)