Noise contrastive estimation: asymptotic properties, formal comparison with MC-MLE

DOI10.1214/18-EJS1485MaRDI QIDQ1616322zbMATH OpenOpenAlexFDO

Authors Lionel Riou-Durand, Nicolas Chopin

Publication date 1 November 2018

Published in Electronic Journal of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1801.10381, https://projecteuclid.org/euclid.ejs/1539741651

Mathematics Subject Classification ID

Point estimation (62F10) Monte Carlo methods (65C05)

Abstract: A statistical model is said to be un-normalised when its likelihood function involves an intractable normalising constant. Two popular methods for parameter inference for these models are MC-MLE (Monte Carlo maximum likelihood estimation), and NCE (noise contrastive estimation); both methods rely on simulating artificial data-points to approximate the normalising constant. While the asymptotics of MC-MLE have been established under general hypotheses (Geyer, 1994), this is not so for NCE. We establish consistency and asymptotic normality of NCE estimators under mild assumptions. We compare NCE and MC-MLE under several asymptotic regimes. In particular, we show that, when m goes to infinity while n is fixed (m and n being respectively the number of artificial data-points, and actual data-points), the two estimators are asymptotically equivalent. Conversely, we prove that, when the artificial data-points are IID, and when n goes to infinity while m/n converges to a positive constant, the asymptotic variance of a NCE estimator is always smaller than the asymptotic variance of the corresponding MC-MLE estimator. We illustrate the variance reduction brought by NCE through a numerical study.

Recommendations

• scientific article; zbMATH DE number 6276120
• Asymptotics of Monte Carlo maximum likelihood estimators
• The Poisson transform for unnormalised statistical models
• Some extensions of score matching
• A note on Monte Carlo maximization by the density ratio model

Cites work

• scientific article; zbMATH DE number 509169 (Why is no real title available?)
• scientific article; zbMATH DE number 2107836 (Why is no real title available?)
• scientific article; zbMATH DE number 6276120 (Why is no real title available?)
• An efficient learning procedure for deep Boltzmann machines
• Basic properties of strong mixing conditions. A survey and some open questions
• Concavity of certain maps on positive definite matrices and applications to Hadamard products
• General state space Markov chains and MCMC algorithms
• Markov chains and stochastic stability
• Maximum likelihood estimation for spatial models by Markov chain Monte Carlo stochastic approximation
• Note on the Consistency of the Maximum Likelihood Estimate
• On the Markov chain central limit theorem
• Posterior sampling when the normalizing constant is unknown
• The Poisson transform for unnormalised statistical models
• Variational Analysis

Cited in

(3)

• The Poisson transform for unnormalised statistical models
• scientific article; zbMATH DE number 7415101 (Why is no real title available?)
• scientific article; zbMATH DE number 6276120 (Why is no real title available?)

This page was built for publication: Noise contrastive estimation: asymptotic properties, formal comparison with MC-MLE

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1616322)