Estimation bias in maximum entropy models

DOI10.3390/E15083109MaRDI QIDQ280482zbMATH OpenOpenAlexFDO

Authors Jakob H. Macke, Iain Murray, Peter E. Latham

Publication date 10 May 2016

Published in Entropy (Search for Journal in Brave)

Full work available at URL https://doi.org/10.3390/e15083109

zbMATH Keywords

asymptotic biasdichotomized GaussianIsing modelmaximum entropymodel-misspecificationneural population codingneurophysiologysampling bias

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Applications of statistics to biology and medical sciences; meta analysis (62P10)

Review text

Summary: Maximum entropy models have become popular statistical models in neuroscience and other areas in biology and can be useful tools for obtaining estimates of mutual information in biological systems. However, maximum entropy models fit to small data sets can be subject to sampling bias; i.e., the true entropy of the data can be severely underestimated. Here, we study the sampling properties of estimates of the entropy obtained from maximum entropy models. We focus on pairwise binary models, which are used extensively to model neural population activity. We show that, if the data is well described by a pairwise model, the bias is equal to the number of parameters divided by twice the number of observations. If, however, the higher order correlations in the data deviate from those predicted by the model, the bias can be larger. Using a phenomenological model of neural population recordings, we find that this additional bias is highest for small firing probabilities, strong correlations and large population sizes -- for the parameters we tested, a factor of about four higher. We derive guidelines for how long a neurophysiological experiment needs to be in order to ensure that the bias is less than a specified criterion. Finally, we show how a modified plug-in estimate of the entropy can be used for bias correction.

Recommendations

• A simple method for estimating the entropy of neural activity
• Bias analysis in entropy estimation
• Estimation of Entropy and Mutual Information
• Tight Data-Robust Bounds to Mutual Information Combining Shuffling and Model Selection Techniques
• scientific article; zbMATH DE number 1164155

Cites work

• Entropy expressions and their estimators for multivariate distributions
• Estimation of Entropy and Mutual Information
• Estimation of the entropy of a multivariate normal distribution
• Generating Spike Trains with Specified Correlation Coefficients
• scientific article; zbMATH DE number 3062467 (Why is no real title available?)
• Large-sample asymptotic approximations for the sampling and posterior distributions of differential entropy for multivariate normal distributions
• Model-based decoding, information estimation, and change-point detection techniques for multineuron spike trains
• On some models for multivariate binary variables parallel in complexity with the multivariate Gaussian distribution
• Synchronous Firing and Higher-Order Interactions in Neuron Pool
• The impact of high-order interactions on the rate of synchronous discharge and information transmission in somatosensory cortex
• Tight Data-Robust Bounds to Mutual Information Combining Shuffling and Model Selection Techniques

Cited in

(6)

• Bias analysis in entropy estimation
• The simplest maximum entropy model for collective behavior in a neural network
• A simple method for estimating the entropy of neural activity
• scientific article; zbMATH DE number 1164155 (Why is no real title available?)
• How Biased Is Your Model? Concentration Inequalities, Information and Model Bias
• Tight Data-Robust Bounds to Mutual Information Combining Shuffling and Model Selection Techniques

This page was built for publication: Estimation bias in maximum entropy models

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q280482)