Noise contrastive estimation: asymptotic properties, formal comparison with MC-MLE

DOI10.1214/18-EJS1485zbMATH Open1407.62069arXiv1801.10381OpenAlexW2897723259MaRDI QIDQ1616322FDOQ1616322

Authors: Lionel Riou-Durand, Nicolas Chopin 

Publication date: 1 November 2018

Published in: Electronic Journal of Statistics (Search for Journal in Brave)

Abstract: A statistical model is said to be un-normalised when its likelihood function involves an intractable normalising constant. Two popular methods for parameter inference for these models are MC-MLE (Monte Carlo maximum likelihood estimation), and NCE (noise contrastive estimation); both methods rely on simulating artificial data-points to approximate the normalising constant. While the asymptotics of MC-MLE have been established under general hypotheses (Geyer, 1994), this is not so for NCE. We establish consistency and asymptotic normality of NCE estimators under mild assumptions. We compare NCE and MC-MLE under several asymptotic regimes. In particular, we show that, when m goes to infinity while n is fixed (m and n being respectively the number of artificial data-points, and actual data-points), the two estimators are asymptotically equivalent. Conversely, we prove that, when the artificial data-points are IID, and when n goes to infinity while m/n converges to a positive constant, the asymptotic variance of a NCE estimator is always smaller than the asymptotic variance of the corresponding MC-MLE estimator. We illustrate the variance reduction brought by NCE through a numerical study.

Full work available at URL: https://arxiv.org/abs/1801.10381

Recommendations

• scientific article; zbMATH DE number 6276120
• Asymptotics of Monte Carlo maximum likelihood estimators
• The Poisson transform for unnormalised statistical models
• Some extensions of score matching
• A note on Monte Carlo maximization by the density ratio model

Mathematics Subject Classification ID

Point estimation (62F10) Monte Carlo methods (65C05)

Cites Work

• Title not available (Why is that?)
• Variational Analysis
• Markov chains and stochastic stability
• General state space Markov chains and MCMC algorithms
• On the Markov chain central limit theorem
• Basic properties of strong mixing conditions. A survey and some open questions
• Concavity of certain maps on positive definite matrices and applications to Hadamard products
• Note on the Consistency of the Maximum Likelihood Estimate
• Title not available (Why is that?)
• Title not available (Why is that?)
• Maximum likelihood estimation for spatial models by Markov chain Monte Carlo stochastic approximation
• Posterior sampling when the normalizing constant is unknown
• An efficient learning procedure for deep Boltzmann machines
• The Poisson transform for unnormalised statistical models

Cited In (3)

• Title not available (Why is that?)
• Title not available (Why is that?)
• The Poisson transform for unnormalised statistical models

This page was built for publication: Noise contrastive estimation: asymptotic properties, formal comparison with MC-MLE

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1616322)