Nonconvex regularization for sparse neural networks

DOI10.1016/J.ACHA.2022.05.003MaRDI QIDQ6339337arXivFDO

Authors Konstantin Pieper, A. Petrosyan

Publication date 23 April 2020

Published in arXiv

Full work available at URL https://doi.org/10.1016/J.ACHA.2022.05.003

Mathematics Subject Classification ID

Computer science (68-XX) Biology and other natural sciences (92-XX)

arXiv classification

cs.LGmath.OCstat.ML

Abstract: Convex $el{l}_1$ regularization using an infinite dictionary of neurons has been suggested for constructing neural networks with desired approximation guarantees, but can be affected by an arbitrary amount of over-parametrization. This can lead to a loss of sparsity and result in networks with too many active neurons for the given data, in particular if the number of data samples is large. As a remedy, in this paper, a nonconvex regularization method is investigated in the context of shallow ReLU networks: We prove that in contrast to the convex approach, any resulting (locally optimal) network is finite even in the presence of infinite data (i.e., if the data distribution is known and the limiting case of infinite samples is considered). Moreover, we show that approximation guarantees and existing bounds on the network size for finite data are maintained.

This page was built for publication: Nonconvex regularization for sparse neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6339337)