Maximum entropy fundamentals
Summary: In its modern formulation, the maximum entropy principle was promoted by E. T. Jaynes, starting in the mid-fifties. The principle dictates that one should look for a distribution, consistent with available information, which maximizes the entropy. However, this principle focuses only on distributions and it appears advantageous to bring information theoretical thinking more prominently into play by also focusing on the ``observer and on coding. This view was brought forward by the second named author in the late seventies and is the view we follow here. It leads to the consideration of a certain game, the code length game, and, via standard game theoretical thinking, to a principle of game theoretical equilibrium. This principle is more basic than the maximum entropy principle in the sense that the search for one type of optimal strategies in the code length game translates directly into the search for distributions with maximum entropy. In the present paper we offer a self-contained and comprehensive treatment of fundamentals of both principles mentioned based on a study of the code length game. Though new concepts and results are presented, the reading should be instructional and accessible to a rather wide audience, at least if certain mathematical details are left aside at a first reading. The most frequently studied instance of entropy maximization pertains to the mean energy model, which involves a moment constraint related to a given function, here taken to represent ``energy. This type of application is very well known from the literature with hundreds of applications pertaining to several different fields and will also serve here as an important illustration of the theory. But our approach reaches further, especially regarding the study of continuity properties of the entropy function, and this leads to new results which allow for a discussion of models with so-called entropy loss. These results have tempted us to speculate over the development of natural languages. In fact, we are able to relate our theoretical findings to the empirically found Zipf's law, which involves statistical aspects of words in a language. The apparent irregularity inherent in models with entropy loss turns out to imply desirable stability properties of languages.
- Maximum entropy principle revisited
- Common sense and maximum entropy
- Maximizing relative entropy at given energies
- Obtaining the thermodynamic relations for the Gibbs ensemble using the maximum entropy method
- Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
- On the origin of ambiguity in efficient communication
- Entropy-based algorithm for supply-chain complexity assessment
- On the entropy of couplings
- Estimates for discontinuity jumps of information characteristics of quantum systems and channels
- On the relation between the maximum entropy principle and the principle of least effort
- Probability distributions and the maximum entropy principle
- Geometric probability theory and Jaynes's methodology
- Maximum Rényi Entropy Rate
- scientific article; zbMATH DE number 4131424 (Why is no real title available?)
- Power-law and exponential rank distributions: a panoramic Gibbsian perspective
- scientific article; zbMATH DE number 1804111 (Why is no real title available?)
- Constant conditional entropy and related hypotheses
- Zipf's law, hyperbolic distributions and entropy loss
- Entropy concentration and the empirical coding game
- Exponential families and MaxEnt calculations for entropy measures of statistical physics
- scientific article; zbMATH DE number 3621626 (Why is no real title available?)
- Temperatures in linguistics as a model of thermodynamics
- Maximum entropy versus minimum risk and applications to some classical discrete distributions
- CONTINUITY OF A CLASS OF ENTROPIES AND RELATIVE ENTROPIES
- scientific article; zbMATH DE number 762940 (Why is no real title available?)
- The principle of stationary action in the calculus of variations
- Maximum Entropy Guide for BSS
- Equivocation axiom on first order languages
- Excess entropy in natural language: present state and perspectives
- Algorithmic Learning Theory
- COMPLEX DEPENDENCIES IN LARGE SOFTWARE SYSTEMS
- An information-theoretic approach to partially identified auction models
- Stability criteria for \(q\)-expectation values
- Entropy: the Markov ordering approach
- Variational formulation of hyperbolic conservation laws
- Maximum entropy on compact groups
- Blending Bayesian and frequentist methods according to the precision of prior information with applications to hypothesis testing
- Maximum entropy principle and power-law tailed distributions
This page was built for publication: Maximum entropy fundamentals
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1613055)
