Excess entropy in natural language: present state and perspectives
From MaRDI portal
(Redirected from Publication:5264343)
Abstract: We review recent progress in understanding the meaning of mutual information in natural language. Let us define words in a text as strings that occur sufficiently often. In a few previous papers, we have shown that a power-law distribution for so defined words (a.k.a. Herdan's law) is obeyed if there is a similar power-law growth of (algorithmic) mutual information between adjacent portions of texts of increasing length. Moreover, the power-law growth of information holds if texts describe a complicated infinite (algorithmically) random object in a highly repetitive way, according to an analogous power-law distribution. The described object may be immutable (like a mathematical or physical constant) or may evolve slowly in time (like cultural heritage). Here we reflect on the respective mathematical results in a less technical way. We also discuss feasibility of deciding to what extent these results apply to the actual human communication.
Recommendations
- Entropy analysis of word-length series of natural language texts: effects of text language and genre
- Entropy of natural languages: Theory and experiment
- ENTROPIC ANALYSIS OF THE ROLE OF WORDS IN LITERARY TEXTS
- Towards the quantification of the semantic information encoded in written language
- Information Theory Meets Power Laws
Cites work
- A convergent gambling estimate of the entropy of English
- A general definition of conditional information and its application to ergodic decomposition
- A Theory of Program Size Formally Identical to Information Theory
- An Example of Statistical Investigation of the Text Eugene Onegin Concerning the Connection of Samples in Chains
- Asymptotically mean stationary measures
- Complexity through nonextensivity
- Computational mechanics: pattern and prediction, structure and simplicity.
- Grammar-based codes: a new class of universal lossless source codes
- scientific article; zbMATH DE number 3664335 (Why is no real title available?)
- scientific article; zbMATH DE number 3738962 (Why is no real title available?)
- scientific article; zbMATH DE number 3086001 (Why is no real title available?)
- Least effort and the origins of scaling in human language
- Maximum entropy fundamentals
- ON A CLASS OF SKEW DISTRIBUTION FUNCTIONS
- On a definition of random sequences with respect to conditional probability
- On the Vocabulary of Grammar-Based Codes and the Logical Consistency of Texts
- Predictability, complexity, and learning
- Prediction and Entropy of Printed English
- Regularities unseen, randomness observed: Levels of entropy convergence
- String matching bounds via coding
- Strong, weak and false inverse power laws
- The complexity of songs
- The Smallest Grammar Problem
- Universal redundancy rates do not exist
- Variable-length coding of two-sided asymptotically mean stationary measures
- Word frequency and entropy of symbolic sequences: A dynamical perspective
Cited in
(9)- On hidden Markov processes with infinite excess entropy
- On the origin of ambiguity in efficient communication
- Predictive rate-distortion for infinite-order Markov processes
- Constant conditional entropy and related hypotheses
- Towards the quantification of the semantic information encoded in written language
- Two halves of a meaningful text are statistically different
- Natural complexity, computational complexity and depth
- A comparative dynamical analysis of Hebrew texts
- Divergent predictive states: the statistical complexity dimension of stationary, ergodic hidden Markov processes
This page was built for publication: Excess entropy in natural language: present state and perspectives
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5264343)
