Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape
From MaRDI portal
Abstract: We study the loss landscape of two-layer mildly overparameterized ReLU neural networks on a generic finite input dataset for the squared error loss. Our approach involves bounding the dimension of the sets of local and global minima using the rank of the Jacobian of the parameterization map. Using results on random binary matrices, we show most activation patterns correspond to parameter regions with no bad differentiable local minima. Furthermore, for one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima. We experimentally confirm these results by finding a phase transition from most regions having full rank to many regions having deficient rank depending on the amount of overparameterization.
This page was built for publication: Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6510356)
