On a generalized entropy and a coding theorem
Consider an experiment having \(n\) possible outcomes with probability distribution \(P=(p_1,\ldots,p_n)\), where \(p_i\geq 0\) \((i=1,\ldots,n)\) and \(\sum p_i=1\). The amount of uncertainty (i.e. the entropy of the distribution \(P\), as defined by C. E. Shannon) is given by \(H(P)=-\sum p_i\log p_i\). A. Rényi introduced the concept of incomplete distributions and defined the entropy of order \(\alpha\) by the expression \[ H_\alpha(P)=(1-\alpha)^{-1}\log\left(\sum p_i^{\alpha}\left/\sum p_i\right),\quad \alpha>0,\;\alpha\neq 1,\;\sum p_i\leq 1.\right. \] In this paper, two generalizations of the above entropies and others are given. They are: (a) The generalized entropy of order unity and type \(\{\beta_i\}\), \[ H_i^{p_i\brace \beta_i}=-\sum p_i^{\beta_i}\log p_i\left/\sum p_i^{\beta_i},\quad \beta_i\geq 1,\;\sum p_i\leq 1;\right. \] (b) The generalized entropy of order \(\alpha\) and type \(\beta_i\), \[ H_i^{p_i\brace \beta_i}=(1-\alpha)^{-1}\log\left(\sum p_i^{\alpha\beta_i-1}\left/\sum p_i^{\beta_i}\right),\quad \alpha>0,\;\alpha\neq 1,\;\beta_i\geq 1,\;\sum p_i\leq 1.\right. \] These generalized entropies are characterized by a set of four postulates and a few interesting properties are mentioned. The generalized entropy of order \(\alpha\) and type \(\beta_i\) is a monotonic decreasing function of \(\alpha\) for given \(\beta_i\) and \(p_i\) and satisfies the following partial differential equation: \[ \frac {\partial}{\partial\alpha} H_\alpha^{p_i\brace \beta_i}-\sum \frac {\partial}{\partial\beta_i} H_\alpha^{p_i\brace \beta_i}(1-\alpha)^{-1}H_\alpha^{p_i\brace \beta_i}=(1-\alpha)^{-1}H_1^{p_i\brace \beta_i}. \] Certain types of generalized average code lengths -- \(L^{p_i\brace \beta_i}(t)\) and \(L_M^{p_i\brace \beta_i}(t)\) of order \(t\) and type \(\{\beta_i\}\), resembling the mean lengths -- are defined. A lemma is proved for any code satisfying the inequality \(\sum p_i^{\beta_i-1}D^{-n_i}\leq p_i^{\beta_i}\) which, in some sense, is a generalization of Kraft's inequality for uniquely decipherable codes. The generalized coding theorem states that the average code length of order \(t\) and type \(\{\beta_i\}\) per input symbol, i.e. \(L_M^{p_i\brace \beta_i}(t)/M\), can be made as close to \(H_\alpha^{p_i\brace \beta_i}\) as desired by encoding sufficiently long sequences, but it is not possible to find a code whose average length of order \(t\) and type \(\{\beta_i\}\) is less than \(H_\alpha^{p_i\brace \beta_i}\), where \(\alpha=(1+t)^{-1}\) and \(-1\leq t\leq\infty\). The corresponding coding theorem for Rényi's entropy given recently by L. L. Campbell follows as a special case.
- Generalized entropies in coding theory
- A coding theorem on generalized R-norm entropy
- The entropy theorem via coding bounds
- scientific article; zbMATH DE number 4193602
- Some coding theorem on generalized Havrda-Charvat and Tsallis's entropy
- A coding theorem connected on R-norm entropy
- Coding theorems on a non-additive generalized entropy of Havrda-Charvat and Tsallis
- Some coding theorems for nonadditive generalized mean-value entropies
- scientific article; zbMATH DE number 4168384
- scientific article; zbMATH DE number 3920365
- Some properties of generalized exponential entropies with applications to data compression
- A note on coding theorems of information theory
- A unified characterization of generalized information and certainty measures
- An axiomatic derivation of the coding-theoretic possibilistic entropy
- A spectral representation for the entropy of topological dynamical systems
- Entropy as an integral operator: erratum and modification
- Generalized entropies in coding theory
- Some characterization theorems for generalized measures of uncertainty and information
- A coding theorem on generalized R-norm entropy
- Entropy and Source Coding for Integer-Dimensional Singular Random Variables
- A Comparative Assessment of Various Measures of Entropy
- Coding Theory: A General Framework and Two Inverse Problems
- Mean Entropies
- Entropy encoding, Hilbert space, and Karhunen-Loève transforms
- Coding Theorems on the Threshold Scheme for a General Source
- scientific article; zbMATH DE number 3920365 (Why is no real title available?)
- scientific article; zbMATH DE number 4039787 (Why is no real title available?)
- scientific article; zbMATH DE number 4091401 (Why is no real title available?)
- The analogy between coding theory and multifractals
- Heterogeneous questionnaires and Rényi's entropy
- scientific article; zbMATH DE number 64775 (Why is no real title available?)
- scientific article; zbMATH DE number 3601519 (Why is no real title available?)
- Some coding theorems for nonadditive generalized mean-value entropies
- Inequalities between entropy and index of coincidence derived from information diagrams
- Information and Coding Discrimination of Pseudo-Additive Entropies (PAE)
- Some coding theorem on generalized Havrda-Charvat and Tsallis's entropy
- scientific article; zbMATH DE number 6280500 (Why is no real title available?)
- Entropy lower bounds related to a problem of universal coding and prediction
- scientific article; zbMATH DE number 3326137 (Why is no real title available?)
- scientific article; zbMATH DE number 2246256 (Why is no real title available?)
- Some coding theorem connected on generalized Renyi's entropy for incomplete power probability distribution \(p^\beta\)
- Source coding with escort distributions and Rényi entropy bounds
- The finitary coding of two Bernoulli schemes with unequal entropies has finite expectation
- Assessing goodness-of-fit for sparse categories using Rényi divergence
- Entropies of the Poisson distribution as functions of intensity: ``normal and ``anomalous behavior
This page was built for publication: On a generalized entropy and a coding theorem
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5587626)