Dissimilarity measures for probability distributions

Publication date 18 November 2001

deviations dissimilarity index Kullback-Leibler coefficient probability distributions variational distance

Statistical aspects of information-theoretic topics (62B10) Characterization and structure theory of statistical distributions (62E10) Classification and discrimination; cluster analysis (statistical aspects) (62H30)

Review text

This section is designed as a prelude to the following section of this book, see the following review, Zbl 0977.62066, as it provides methods for comparing the elements \(\xi_{kj}\) and \(\xi_{lj}\) in the \(j\)-th column of a symbolic data array \(\underline{X}=(\xi_{kj})\) in the case where \(Y_j (\xi_{kj}=Y_j(k))\) is a symbolic modal variable. The entries \(\xi_{kj}=Y_j(k)\) and \(\xi_{lj}=Y_j(l)\) of the symbolic data array \(\underline{X}=(\xi_{kj})\) are measures, probability distributions or weightings. Let \(P, Q\) be probability distributions on the domain \(\mathcal Y\). Various measures \(d(P,Q)\) are described which characterize the distinctness of the distributions \(P\) and \(Q\) in terms of the densities \(p\) and \(q\). Most classical approaches to measuring the deviation between two probability distributions \(P\) and \(Q\) with densities \(p\) and \(q\) are special cases or transformations of the \(\phi\)-divergence or \(I\)-divergence, proposed by \textit{I. Csiszár} [Stud. Sci. Math. Hung. 2, 299-318 (1967; Zbl 0157.25802)], which use the (likelihood) ratio \(\lambda(y) \colon= q(y)/p(y)\) of the densities \(p\) and \(q:\) \[ d(P,Q):=d(P,Q;\phi):= \int_{\mathcal Y}\phi(\lambda(y))dP(y)= \int_{\mathcal Y}\phi(q(y)/p(y))p(y)d\mu(y). \] Here \(\phi(\cdot)\) is a given real-valued and convex function defined on \(R_+\) with \(\phi(1)=0\), and \(\mu\) is either the Lebesgue measure or the counting measure. Various divergence and dissimilarity indexes result by a special choice of the convex function \(\phi(\lambda)\). For the special choice \(\phi(\lambda) \colon= \lambda\log \lambda\) for \(\lambda>0\) and \(\phi(0)=0\) we get the \(I\)-coefficient, Kullback-Leibler divergence, relative or cross-entropy, or information distance. A symmetric version of the Kullback-Leibler coefficient, termed \(J\)-coefficient, is given by \(J(P,Q) \colon= I(P,Q)+I(Q,P)\), which is a \(\phi\)-divergence measure for \(\phi(\lambda) \colon= (\lambda-1)\log \lambda\). The function \(\phi(\lambda) \colon= (\lambda-1)^2\) induces a dissimilarity measure which is called \(\chi^2\)-divergence. Various dissimilarity measures defined by using the Hellinger coefficient result for the special choice \(\phi(\lambda)\colon=\lambda^3\) for \( 0<s<1\). The choice \(\phi(\lambda) \colon= |\lambda|\) yields a symmetric dissimilarity coefficient, termed variational distance (Minkowski \(L_1\) distance). For the entire collection see [Zbl 1039.62501].

Recommendations

Cited in

(11)

This page was built for publication: Dissimilarity measures for probability distributions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2756635)