Information-spectrum methods in information theory (Q4779829)

This book proposes a generalization of information theory in the sense of Shannon to a situation with sources and channels being possibly nonstationary and/or nonergodic. Moreover, the source/channel alphabets are supposed arbitrary (possibly infinite). For this purpose, one introduces new quantities and new operations for manipulating these quantities. In the case of source coding, if \(X^n\) is a random word with \(n\) letters from the source \(X\) with probability distribution \(P_{X^n}\) the ``entropy density rate'' is EDR\((X^n)= {1\over n}\log{1\over P_{X^n}(X^n)}\) and its distribution is the ``entropy-spectrum''. The new operation is called the ``limit superior in probability'' defined for a sequence of random variables \(Z_n\) by \[ p-\lim_{n\to\infty} \sup(Z_n)= \inf\Biggl\{\alpha|\lim_{n\to\infty}\text{Pr}\{Z_n> \alpha\}= 0\Biggr\}. \] One defines then the ``spectral sup-entropy rate'' of \(X\) by \[ \overline H(X)= p-\lim_{n\to\infty}\sup(\text{EDR}(X^n)). \] One sees this generalization is the right quantity for estimating the number of fixed-length coding rates. In the case of channel coding, the ``limit inferior in probability'' can be introduced to define a ``spectral inf-mutual information rate'' of a channel with inputs and outputs generalizing the usual mutual information of inputs and outputs. This information theoretic quantity is related to the channel capacity. These ideas are applied to various situations within the chapters of this book. Chapter one is devoted to source coding. In chapter two the focus is on random number generation. Chapter three deals with channel coding. Chapter four contains work on hypothesis testing to be used later. Chapter five is on rate-distortion theory. Here, the problem is that perfect encoding of a continuous source is not possible so that a finite distortion is allowed, which is an additional constraint when dealing with problems of coding rate minimization. Chapter six investigates ``identification code and channel resolvability''. Here the encoding process itself is random, which adds complexity, but the decoding and transmission will be coupled in such a way that messages not associated to prescribed, possibly self-intersecting identification domains will be discarded. The key quantities are then the probabilities that a decoder misjudges that a message is discarded or that it misjudges that it is not discarded. Due to the fact that the identification domains are possibly intersecting, these two probabilities are not related in a trivial way. Then one relates an identification channel capacity result to an approximation property for channels called ``channel resolvability''. Chapter seven is a development on multi-user information theory (several decoders or encoders). This book presents useful and important concepts in information theory arising from original ideas of the author and S. Verdú.