On the complexity of logistic regression models
From MaRDI portal
Publication:5214359
Abstract: We investigate the complexity of logistic regression models which is defined by counting the number of indistinguishable distributions that the model can represent (Balasubramanian, 1997). We find that the complexity of logistic models with binary inputs does not only depend on the number of parameters but also on the distribution of inputs in a non-trivial way which standard treatments of complexity do not address. In particular, we observe that correlations among inputs induce effective dependencies among parameters thus constraining the model and, consequently, reducing its complexity. We derive simple relations for the upper and lower bounds of the complexity. Furthermore, we show analytically that, defining the model parameters on a finite support rather than the entire axis, decreases the complexity in a manner that critically depends on the size of the domain. Based on our findings, we propose a novel model selection criterion which takes into account the entropy of the input distribution. We test our proposal on the problem of selecting the input variables of a logistic regression model in a Bayesian Model Selection framework. In our numerical tests, we find that, while the reconstruction errors of standard model selection approaches (AIC, BIC, regularization) strongly depend on the sparsity of the ground truth, the reconstruction error of our method is always close to the minimum in all conditions of sparsity, data size and strength of input correlations. Finally, we observe that, when considering categorical instead of binary inputs, in a simple and mathematically tractable case, the contribution of the alphabet size to the complexity is very small compared to that of parameter space dimension. We further explore the issue by analysing the dataset of the "13 keys to the White House" which is a method for forecasting the outcomes of US presidential elections.
Recommendations
Cites work
- A new look at the statistical model identification
- An interior-point method for large-scale l₁-regularized logistic regression
- An invariant form for the prior probability in estimation problems
- Belief propagation and replicas for inference and learning in a kinetic Ising model with hidden spins
- Counting probability distributions: Differential geometry and model selection
- Estimating the dimension of a model
- Fisher information and stochastic complexity
- High-dimensional Ising model selection using \(\ell _{1}\)-regularized logistic regression
- scientific article; zbMATH DE number 3423418 (Why is no real title available?)
- scientific article; zbMATH DE number 3426675 (Why is no real title available?)
- scientific article; zbMATH DE number 3144846 (Why is no real title available?)
- scientific article; zbMATH DE number 4068049 (Why is no real title available?)
- scientific article; zbMATH DE number 765034 (Why is no real title available?)
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- On Bayesian Analysis of Generalized Linear Models Using Jeffreys's Prior
- Properties and implementation of Jeffreys's prior in binomial regression models
- Sparse model selection in the highly under-sampled regime
- Statistical Inference, Occam's Razor, and Statistical Mechanics on the Space of Probability Distributions
- The elements of statistical learning. Data mining, inference, and prediction
- The Selection of Prior Distributions by Formal Rules
This page was built for publication: On the complexity of logistic regression models
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5214359)
