Individually Conditional Individual Mutual Information Bound on Generalization Error
From MaRDI portal
Abstract: We propose a new information-theoretic bound on generalization error based on a combination of the error decomposition technique of Bu et al. and the conditional mutual information (CMI) construction of Steinke and Zakynthinou. In a previous work, Haghifam et al. proposed a different bound combining the two aforementioned techniques, which we refer to as the conditional individual mutual information (CIMI) bound. However, in a simple Gaussian setting, both the CMI and the CIMI bounds are order-wise worse than that by Bu et al.. This observation motivated us to propose the new bound, which overcomes this issue by reducing the conditioning terms in the conditional mutual information. In the process of establishing this bound, a conditional decoupling lemma is established, which also leads to a meaningful dichotomy and comparison among these information-theoretic bounds.
Recommendations
- A Finite-Sample, Distribution-Free, Probabilistic Lower Bound on Mutual Information
- Conditional Mutual Information Estimation for Mixed, Discrete and Continuous Data
- Generalized mutual-information based independence tests
- On mutual information estimation for mixed-pair random variables
- Mutual Information and Optimality of Approximate Message-Passing in Random Linear Estimation
- A unified definition of mutual information with applications in machine learning
- Generalization Error Bounds via Rényi-, f-Divergences and Maximal Leakage
- Representation of Mutual Information Via Input Estimates
- Information-type divergence when the likelihood ratios are bounded
This page was built for publication: Individually Conditional Individual Mutual Information Bound on Generalization Error
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5088470)
