High-dimensional penalty selection via minimum description length principle

DOI10.1007/S10994-018-5732-2MaRDI QIDQ1631787zbMATH OpenOpenAlexWikidataFDO

Authors Kohei Miyaguchi, Kenji Yamanishi

Publication date 7 December 2018

Published in Machine Learning (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1804.09904

zbMATH Keywords

concave-convex procedureluckiness normalized maximum likelihoodminimum description length principlepenalty selectionregularized empirical risk minimization

Mathematics Subject Classification ID

Point estimation (62F10) Estimation in multivariate analysis (62H12) Classification and discrimination; cluster analysis (statistical aspects) (62H30)

Abstract: We tackle the problem of penalty selection of regularization on the basis of the minimum description length (MDL) principle. In particular, we consider that the design space of the penalty function is high-dimensional. In this situation, the luckiness-normalized-maximum-likelihood(LNML)-minimization approach is favorable, because LNML quantifies the goodness of regularized models with any forms of penalty functions in view of the minimum description length principle, and guides us to a good penalty function through the high-dimensional space. However, the minimization of LNML entails two major challenges: 1) the computation of the normalizing factor of LNML and 2) its minimization in high-dimensional spaces. In this paper, we present a novel regularization selection method (MDL-RS), in which a tight upper bound of LNML (uLNML) is minimized with local convergence guarantee. Our main contribution is the derivation of uLNML, which is a uniform-gap upper bound of LNML in an analytic expression. This solves the above challenges in an approximate manner because it allows us to accurately approximate LNML and then efficiently minimize it. The experimental results show that MDL-RS improves the generalization performance of regularized estimates specifically when the model has redundant parameters.

Recommendations

• Normalized ML and the MDL principle for variable selection in linear regression
• Model selection based on minimum description length
• Model Selection and the Principle of Minimum Description Length
• Minimal penalties for Gaussian model selection
• Model Selection Using the Minimum Description Length Principle

Cites work

• 10.1162/15324430152748236
• A learning criterion for stochastic rules
• A new look at the statistical model identification
• A theory of the learnable
• Bayesian heuristic approach to discrete and global optimization. Algorithms, visualization, software, and applications. Incl. 2 disks
• Efficient Empirical Bayes Variable Selection and Estimation in Linear Models
• Estimating the dimension of a model
• Extended Bayesian information criteria for model selection with large model spaces
• Fisher information and stochastic complexity
• Gaussian processes for machine learning.
• scientific article; zbMATH DE number 4095371 (Why is no real title available?)
• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• MDL Denoising Revisited
• Minimum complexity density estimation
• Minimum description length penalization for group and multi-task sparse learning
• Modeling by shortest data description
• Sparse inverse covariance estimation with the graphical lasso
• Sparse signal recovery with exponential-family noise
• The Concave-Convex Procedure
• Understanding machine learning. From theory to algorithms

Cited in

(1)

• Minimum description length revisited

Describes a project that uses

Uses Software

• glasso

This page was built for publication: High-dimensional penalty selection via minimum description length principle

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1631787)