Searching for minimal optimal neural networks

DOI10.1016/J.SPL.2021.109353MaRDI QIDQ2667597zbMATH OpenOpenAlexFDO

Authors Lam Si Tung Ho, Vu Dinh

Publication date 4 March 2022

Published in Statistics \& Probability Letters (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2109.13061

zbMATH Keywords

adaptive Lassoconsistencydestructive techniquemodel selectionneural networks

Mathematics Subject Classification ID

Statistical ranking and selection procedures (62F07) General nonlinear regression (62J02) Ridge regression; shrinkage estimators (Lasso) (62J07) Neural nets and related approaches to inference from stochastic processes (62M45)

Abstract: Large neural network models have high predictive power but may suffer from overfitting if the training set is not large enough. Therefore, it is desirable to select an appropriate size for neural networks. The destructive approach, which starts with a large architecture and then reduces the size using a Lasso-type penalty, has been used extensively for this task. Despite its popularity, there is no theoretical guarantee for this technique. Based on the notion of minimal neural networks, we posit a rigorous mathematical framework for studying the asymptotic theory of the destructive technique. We prove that Adaptive group Lasso is consistent and can reconstruct the correct number of hidden nodes of one-hidden-layer feedforward networks with high probability. To the best of our knowledge, this is the first theoretical result establishing for the destructive technique.

Recommendations

• Statistical guarantees for regularized neural networks
• scientific article; zbMATH DE number 1203361
• The hidden layer size in feed-forward neural networks: A statistical point of view
• Designing neural networks for modeling biological data: a statistical perspective
• Relation between weight size and degree of over-fitting in neural network regression

Cites work

• A note on adaptive group Lasso
• scientific article; zbMATH DE number 5957408 (Why is no real title available?)
• Lasso-type recovery of sparse representations for high-dimensional data
• The Adaptive Lasso and Its Oracle Properties

Cited in

(5)

• scientific article; zbMATH DE number 1086504 (Why is no real title available?)
• MinT: A Database for Optimal Net Parameters
• Adaptive group Lasso neural network models for functions of few variables and time-dependent data
• A generalization bound of deep neural networks for dependent data
• The construction of minimal multilayered perceptrons: A case study for sorting

This page was built for publication: Searching for minimal optimal neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2667597)