Adaptive importance sampling for control and inference

DOI10.1007/S10955-016-1446-7zbMATH Open1338.93166arXiv1505.01874OpenAlexW1752690783WikidataQ59438775 ScholiaQ59438775MaRDI QIDQ290478FDOQ290478

Authors: Hilbert J. Kappen, H. C. Ruiz 

Publication date: 1 June 2016

Published in: Journal of Statistical Physics (Search for Journal in Brave)

Abstract: Path integral (PI) control problems are a restricted class of non-linear control problems that can be solved formally as a Feyman-Kac path integral and can be estimated using Monte Carlo sampling. In this contribution we review path integral control theory in the finite horizon case. We subsequently focus on the problem how to compute and represent control solutions. Within the PI theory, the question of how to compute becomes the question of importance sampling. Efficient importance samplers are state feedback controllers and the use of these requires an efficient representation. Learning and representing effective state-feedback controllers for non-linear stochastic control problems is a very challenging, and largely unsolved, problem. We show how to learn and represent such controllers using ideas from the cross entropy method. We derive a gradient descent method that allows to learn feed-back controllers using an arbitrary parametrisation. We refer to this method as the Path Integral Cross Entropy method or PICE. We illustrate this method for some simple examples. The path integral control methods can be used to estimate the posterior distribution in latent state models. In neuroscience these problems arise when estimating connectivity from neural recording data using EM. We demonstrate the path integral control method as an accurate alternative to particle filtering.

Full work available at URL: https://arxiv.org/abs/1505.01874

Recommendations

• Learning effective state-feedback controllers through efficient multilevel importance samplers
• Optimal control as a graphical model inference problem
• A generalized path integral control approach to reinforcement learning
• Stochastic optimal control of state constrained systems
• EP for efficient stochastic control with obstacles

Mathematics Subject Classification ID

Dynamic programming (90C39) Brownian motion (60J65) Neural biology (92C20) Hamilton-Jacobi equations (35F21) Feedback control (93B52) Optimal stochastic control (93E20)

Cites Work

• Universal approximation bounds for superpositions of a sigmoidal function
• Title not available (Why is that?)
• Title not available (Why is that?)
• A sequential smoothing algorithm with linear computational cost
• Title not available (Why is that?)
• Smoothing algorithms for state-space models
• A tutorial on the cross-entropy method
• On Transforming a Certain Class of Stochastic Processes by Absolutely Continuous Substitution of Measures
• Title not available (Why is that?)
• Title not available (Why is that?)
• Differential dynamic programming and Newton's method for discrete optimal control problems
• A solution of the smoothing problem for linear dynamic systems
• Backward simulation methods for Monte Carlo statistical inference
• A generalized path integral control approach to reinforcement learning
• Efficient computation of optimal actions
• Functional Approximations and Dynamic Programming
• Learning Tetris Using the Noisy Cross-Entropy Method
• Optimal Control and Nonlinear Filtering for Nondegenerate Diffusion Processes
• Optimal control as a graphical model inference problem
• Variance Reduction for Simulated Diffusions
• An analysis of temporal-difference learning with function approximation
• Title not available (Why is that?)
• Importance Sampling, Large Deviations, and Differential Games
• Applications of the cross-entropy method to importance sampling and optimal control of diffusions
• A Bayesian approach for inferring neuronal connectivity from calcium fluorescent imaging data

Cited In (31)

• Iterative Path Integral Approach to Nonlinear Stochastic Optimal Control Under Compound Poisson Noise
• Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space
• An iterative Bayesian filtering framework for fast and automated calibration of DEM models
• Design of biased random walks on a graph with application to collaborative recommendation
• Controlled interacting particle algorithms for simulation-based reinforcement learning
• Learning-based importance sampling via stochastic optimal control for stochastic reaction networks
• Learning effective state-feedback controllers through efficient multilevel importance samplers
• Optimal control of probabilistic Boolean control networks: A scalable infinite horizon approach
• Adaptive path-integral autoencoder: representation learning and planning for dynamical systems
• Probabilistic control and majorisation of optimal control
• Adaptive sampling of large deviations
• Convergence rates for optimised adaptive importance samplers
• Daisee: Adaptive importance sampling by balancing exploration and exploitation
• Controlled sequential Monte Carlo
• Optimal control as a graphical model inference problem
• An Optimal Control Derivation of Nonlinear Smoothing Equations
• EP for efficient stochastic control with obstacles
• Nonparametric inference of stochastic differential equations based on the relative entropy rate
• Title not available (Why is that?)
• Implicitly adaptive importance sampling
• Action selection in growing state spaces: control of network structure growth
• Computable Primal and Dual Bounds for Stochastic Control
• Title not available (Why is that?)
• A Bayesian view on motor control and planning
• Variational Inference for Stochastic Differential Equations
• A generalized path integral control approach to reinforcement learning
• Data assimilation: The Schrödinger perspective
• Variational approach to rare event simulation using least-squares regression
• Diffusion Schrödinger bridges for Bayesian computation
• A multilevel approach for stochastic nonlinear optimal control
• An estimator for the relative entropy rate of path measures for stochastic differential equations

This page was built for publication: Adaptive importance sampling for control and inference

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q290478)