Estimations of \(f\)- and Rényi divergences by using a cyclic refinement of the Jensen's inequality (Q2421416): Difference between revisions

A functional related to the \(f\)-divergence functional is introduced and then, some estimates for the new functional and the Rényi divergence are obtained by applying the cyclic refinement of the Jensen inequality. Let \(J\subset \mathbb{R}\) be an interval and let \(f:J\rightarrow \mathbb{R}\) be a function. Let \(\mathbf{p}=(p_1, \ldots, p_n) \in \mathbb{R}^n\) and \(\mathbf{q}=(q_1, \ldots, q_n) \in (0,\infty)^n\) such that \[ \frac{p_i}{q_i} \in J, \quad i=1, 2, \ldots,n. \] Then let \[ \hat{I}_f (\mathbf{p}, \mathbf{q}) := \sum_{i=1}^n q_i f(\frac{p_i}{q_i}). \] Theorem 1. Let \(2\leq k\leq n\) be integers and let \(\lambda :=(\lambda_1, \ldots, \lambda_n)\) be a positive probability distribution. Let \(J\subset \mathbb{R}\) be an interval, let \(\mathbf{p} \in \mathbb{R}^n\) and \(\mathbf{q} \in (0,\infty)^n\) such that \[ \frac{p_i}{q_i} \in J, \quad i=1, 2, \ldots,n. \] (a) If \(f:J\rightarrow \mathbb{R}\) is a convex function, then \[ \hat{I}_f (\mathbf{p}, \mathbf{q}) = \sum_{i=1}^n q_i f(\frac{p_i}{q_i}) \geq \sum_{i=1}^n \left( \sum_{j=0}^{k-1} \lambda_{j+1}q_{i+j}\right) f\left( \frac{\sum_{j=0}^{k-1} \lambda_{j+1}p_{i+j}}{\sum_{j=0}^{k-1} \lambda_{j+1}q_{i+j}}\right) \] \[ \geq f\left( \frac{\sum_{i=1}^n p_i}{\sum_{i=1}^n q_i}\right) \sum_{i=1}^n q_i. \] If \(f\) is a concave function, then the inequality signs are reversed. (b) If \(f:J\rightarrow \mathbb{R}\) is a function such that \(x\mapsto xf(x)\) \((x\in J)\) is convex, then \[ \hat{I}_{id_Jf} (\mathbf{p}, \mathbf{q}) = \sum_{i=1}^n p_i f(\frac{p_i}{q_i}) \geq \sum_{i=1}^n \left( \sum_{j=0}^{k-1} \lambda_{j+1}p_{i+j}\right) f\left( \frac{\sum_{j=0}^{k-1} \lambda_{j+1}p_{i+j}}{\sum_{j=0}^{k-1} \lambda_{j+1}q_{i+j}}\right) \] \[ \geq f\left( \frac{\sum_{i=1}^n p_i}{\sum_{i=1}^n q_i}\right) \sum_{i=1}^n p_i. \] If \(x\mapsto xf(x)\) is a concave function, then the inequality signs in the above inequalities are reversed. In all these inequalities, \(i+j\) means \(i+j-n\) in case of \(i+j>n\). Two particular cases of the previous result are given: the result for the Shannon entropy of a discrete probability distribution and a result for the Kullback-Leibler divergence. The second part of the paper is devoted to Rényi divergence and entropy. Namely, for positive probability distributions \(\mathbf{p} = (p_1, \ldots, p_n)\) and \(\mathbf{q} =(q_1, \ldots,q_n)\) and for \(\alpha \geq 0, \alpha \not= 1\), the Rényi divergence of order \(\alpha\) is defined by \[ D_{\alpha}(\mathbf{p}, \mathbf{q}):= \frac{1}{\alpha -1} \log \left( \sum_{i=1}^n q_i \left( \frac{p_i}{q_i}\right)^{\alpha}\right). \] We get the following result for the Rényi divergence. Theorem 2. Let \(2\leq k \leq n\) be integers, let \(\lambda :=(\lambda_1, \ldots, \lambda_n)\), \(\mathbf{p}\) and \(\mathbf{q}\) be positive probability distributions. (a) If \(0\leq \alpha \leq \beta\), \(\alpha, \beta \not= 1\) and the base of \(\log\) is greater than 1, then \[ D_{\alpha}(\mathbf{p}, \mathbf{q}) \leq \frac{1}{\beta -1} \log \left( \sum_{i=1}^n \left( \sum_{j=0}^{k-1} \lambda_{j+1} p_{i+j} \right) \left( \frac{\sum_{j=0}^{k-1} \lambda_{j+1}p_{i+j} \left( \frac{p_{i+j}}{q_{i+j}}\right)^{\alpha -1}}{\sum_{j=0}^{k-1} \lambda_{j+1}p_{i+j}} \right)^{\frac{\beta -1}{\alpha -1}}\right) \] \[ \leq D_{\beta}(\mathbf{p}, \mathbf{q}). \] The reverse inequalities hold if the base of \(\log\) is between 0 and 1. (b) If \(1<\beta\) and the base of \(\log\) is greater than 1, then \[ \sum_{i=1}^n p_i \log \left( \frac{p_i}{q_i} \right) \leq \frac{1}{\beta -1} \log \left( \sum_{i=1}^n \left( \sum_{j=0}^{k-1} \lambda_{j+1} p_{i+j} \right) \right. \times \] \[ \left. \times \exp \left( \frac{(\beta -1)\sum_{j=0}^{k-1} \lambda_{j+1}p_{i+j} \log \left( \frac{p_{i+j}}{q_{i+j}}\right) }{\sum_{j=0}^{k-1} \lambda_{j+1}p_{i+j}} \right) \right) \leq D_{\beta}(\mathbf{p}, \mathbf{q}), \] where the base of \(\exp\) is the same as the base of \(\log\). The reverse inequalities hold if the base of \(\log\) is between 0 and 1.