On a generalized entropy and a coding theorem

From MaRDI portal





Consider an experiment having \(n\) possible outcomes with probability distribution \(P=(p_1,\ldots,p_n)\), where \(p_i\geq 0\) \((i=1,\ldots,n)\) and \(\sum p_i=1\). The amount of uncertainty (i.e. the entropy of the distribution \(P\), as defined by C. E. Shannon) is given by \(H(P)=-\sum p_i\log p_i\). A. Rényi introduced the concept of incomplete distributions and defined the entropy of order \(\alpha\) by the expression \[ H_\alpha(P)=(1-\alpha)^{-1}\log\left(\sum p_i^{\alpha}\left/\sum p_i\right),\quad \alpha>0,\;\alpha\neq 1,\;\sum p_i\leq 1.\right. \] In this paper, two generalizations of the above entropies and others are given. They are: (a) The generalized entropy of order unity and type \(\{\beta_i\}\), \[ H_i^{p_i\brace \beta_i}=-\sum p_i^{\beta_i}\log p_i\left/\sum p_i^{\beta_i},\quad \beta_i\geq 1,\;\sum p_i\leq 1;\right. \] (b) The generalized entropy of order \(\alpha\) and type \(\beta_i\), \[ H_i^{p_i\brace \beta_i}=(1-\alpha)^{-1}\log\left(\sum p_i^{\alpha\beta_i-1}\left/\sum p_i^{\beta_i}\right),\quad \alpha>0,\;\alpha\neq 1,\;\beta_i\geq 1,\;\sum p_i\leq 1.\right. \] These generalized entropies are characterized by a set of four postulates and a few interesting properties are mentioned. The generalized entropy of order \(\alpha\) and type \(\beta_i\) is a monotonic decreasing function of \(\alpha\) for given \(\beta_i\) and \(p_i\) and satisfies the following partial differential equation: \[ \frac {\partial}{\partial\alpha} H_\alpha^{p_i\brace \beta_i}-\sum \frac {\partial}{\partial\beta_i} H_\alpha^{p_i\brace \beta_i}(1-\alpha)^{-1}H_\alpha^{p_i\brace \beta_i}=(1-\alpha)^{-1}H_1^{p_i\brace \beta_i}. \] Certain types of generalized average code lengths -- \(L^{p_i\brace \beta_i}(t)\) and \(L_M^{p_i\brace \beta_i}(t)\) of order \(t\) and type \(\{\beta_i\}\), resembling the mean lengths -- are defined. A lemma is proved for any code satisfying the inequality \(\sum p_i^{\beta_i-1}D^{-n_i}\leq p_i^{\beta_i}\) which, in some sense, is a generalization of Kraft's inequality for uniquely decipherable codes. The generalized coding theorem states that the average code length of order \(t\) and type \(\{\beta_i\}\) per input symbol, i.e. \(L_M^{p_i\brace \beta_i}(t)/M\), can be made as close to \(H_\alpha^{p_i\brace \beta_i}\) as desired by encoding sufficiently long sequences, but it is not possible to find a code whose average length of order \(t\) and type \(\{\beta_i\}\) is less than \(H_\alpha^{p_i\brace \beta_i}\), where \(\alpha=(1+t)^{-1}\) and \(-1\leq t\leq\infty\). The corresponding coding theorem for Rényi's entropy given recently by L. L. Campbell follows as a special case.




Cited in
(35)








This page was built for publication: On a generalized entropy and a coding theorem

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5587626)