Stop Memorizing: A Data-Dependent Regularization Framework for Intrinsic Pattern Learning

DOI10.1137/19M1236886MaRDI QIDQ5025786zbMATH OpenOpenAlexWikidataFDO

Authors Wei Zhu, Qiang Qiu, Bao Wang, Guillermo Sapiro, Ingrid Daubechies, Jianfeng Lu

Publication date 3 February 2022

Published in SIAM Journal on Mathematics of Data Science (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1805.07291

zbMATH Keywords

deep neural networksdata-dependent regularizationintrinsic pattern learningnetwork memorization

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07) Pattern recognition, speech recognition (68T10) Computer graphics; computational geometry (digital and algorithmic aspects) (68U05)

Abstract: Deep neural networks (DNNs) typically have enough capacity to fit random data by brute force even when conventional data-dependent regularizations focusing on the geometry of the features are imposed. We find out that the reason for this is the inconsistency between the enforced geometry and the standard softmax cross entropy loss. To resolve this, we propose a new framework for data-dependent DNN regularization, the Geometrically-Regularized-Self-Validating neural Networks (GRSVNet). During training, the geometry enforced on one batch of features is simultaneously validated on a separate batch using a validation loss consistent with the geometry. We study a particular case of GRSVNet, the Orthogonal-Low-rank Embedding (OLE)-GRSVNet, which is capable of producing highly discriminative features residing in orthogonal low-rank subspaces. Numerical experiments show that OLE-GRSVNet outperforms DNNs with conventional regularization when trained on real data. More importantly, unlike conventional DNNs, OLE-GRSVNet refuses to memorize random data or random labels, suggesting it only learns intrinsic patterns by reducing the memorizing capacity of the baseline DNN.

Recommendations

• An implicit memory-based method for supervised pattern recognition
• Learning in pattern recognition: Some thoughts
• scientific article; zbMATH DE number 7415108
• Learning fixed point patterns by recurrent networks
• Learning regularization parameters of inverse problems via deep neural networks
• scientific article; zbMATH DE number 4174340

Cites work

• scientific article; zbMATH DE number 1332320 (Why is no real title available?)
• scientific article; zbMATH DE number 1049347 (Why is no real title available?)
• scientific article; zbMATH DE number 961607 (Why is no real title available?)
• 10.1162/153244302760200704
• 10.1162/153244303321897690
• Characterization of the subdifferential of some matrix norms
• Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization
• Learning transformations for clustering and classification

Describes a project that uses

Uses Software

• FaceNet
• CIFAR

This page was built for publication: Stop Memorizing: A Data-Dependent Regularization Framework for Intrinsic Pattern Learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5025786)