Stochastic feature mapping for PAC-Bayes classification

DOI10.1007/S10994-015-5525-9zbMATH Open1343.62036arXiv1204.2609OpenAlexW1944936460MaRDI QIDQ890313FDOQ890313

Authors: Xiong Li, Bin Wang, Tai Sing Lee, Yun-Cai Liu 

Publication date: 10 November 2015

Published in: Machine Learning (Search for Journal in Brave)

Abstract: Probabilistic generative modeling of data distributions can potentially exploit hidden information which is useful for discriminative classification. This observation has motivated the development of approaches that couple generative and discriminative models for classification. In this paper, we propose a new approach to couple generative and discriminative models in an unified framework based on PAC-Bayes risk theory. We first derive the model-parameter-independent stochastic feature mapping from a practical MAP classifier operating on generative models. Then we construct a linear stochastic classifier equipped with the feature mapping, and derive the explicit PAC-Bayes risk bounds for such classifier for both supervised and semi-supervised learning. Minimizing the risk bound, using an EM-like iterative procedure, results in a new posterior over hidden variables (E-step) and the update rules of model parameters (M-step). The derivation of the posterior is always feasible due to the way of equipping feature mapping and the explicit form of bounding risk. The derived posterior allows the tuning of generative models and subsequently the feature mappings for better classification. The derived update rules of the model parameters are same to those of the uncoupled models as the feature mapping is model-parameter-independent. Our experiments show that the coupling between data modeling generative model and the discriminative classifier via a stochastic feature mapping in this framework leads to a general classification tool with state-of-the-art performance.

Full work available at URL: https://arxiv.org/abs/1204.2609

Recommendations

• PAC-Bayesian stochastic model selection
• scientific article; zbMATH DE number 1881088
• A discriminative model for semi-supervised learning
• On the Combination of Information-Theoretic Kernels with Generative Embeddings
• Data dependent priors in PAC-Bayes bounds

zbMATH Keywords

hybrid generative-discriminative classificationPAC-Bayes generalization boundstochastic feature mapping

Mathematics Subject Classification ID

Classification and discrimination; cluster analysis (statistical aspects) (62H30) Learning and adaptive systems in artificial intelligence (68T05)

Cites Work

• 10.1162/153244303765208377
• Title not available (Why is that?)
• Modeling the shape of the scene: A holistic representation of the spatial envelope
• 10.1162/jmlr.2003.3.4-5.993
• A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains
• Localized algorithms for multiple kernel learning
• PAC-Bayesian Inequalities for Martingales
• An introduction to variational methods for graphical models
• Tutorial on practical prediction theory for classification
• Simplified PAC-Bayesian margin bounds.
• Some PAC-Bayesian theorems
• A New Discriminative Kernel from Probabilistic Models

Cited In (1)

• Coupled Generation

Uses Software

• LIBSVM
• SIFT
• SGTlight

This page was built for publication: Stochastic feature mapping for PAC-Bayes classification

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q890313)