Experimental design for partially observed Markov decision processes

DOI10.1137/16M1084924MaRDI QIDQ3176233zbMATH OpenOpenAlexWikidataFDO

Authors Leifur Thorbergsson, Giles Hooker

Publication date 19 July 2018

Published in SIAM/ASA Journal on Uncertainty Quantification (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1209.4019

zbMATH Keywords

experimental designdynamic programmingMarkov decision processoptimal controlvalue iteration algorithm

Mathematics Subject Classification ID

Point estimation (62F10) Optimal statistical designs (62K05) Dynamic programming (90C39) Markov and semi-Markov decision processes (90C40)

Abstract: This paper deals with the question of how to most effectively conduct experiments in Partially Observed Markov Decision Processes so as to provide data that is most informative about a parameter of interest. Methods from Markov decision processes, especially dynamic programming, are introduced and then used in an algorithm to maximize a relevant Fisher Information. The algorithm is then applied to two POMDP examples. The methods developed can also be applied to stochastic dynamical systems, by suitable discretization, and we consequently show what control policies look like in the Morris-Lecar Neuron model, and simulation results are presented. We discuss how parameter dependence within these methods can be dealt with by the use of priors, and develop tools to update control policies online. This is demonstrated in another stochastic dynamical system describing growth dynamics of DNA template in a PCR model.

Recommendations

• A survey of solution techniques for the partially observed Markov decision process
• Observer and control design in partially observable finite Markov chains
• A tutorial on partially observable Markov decision processes
• Control Theory and Experimental Design in Diffusion Processes
• Computationally Feasible Bounds for Partially Observed Markov Decision Processes

Cites work

• scientific article; zbMATH DE number 5685899 (Why is no real title available?)
• Approximate Dynamic Programming
• Automated reverse engineering of nonlinear dynamical systems
• Bayesian experimental design: A review
• Branching Processes
• Control Theory and Experimental Design in Diffusion Processes
• Forcing Function Diagnostics for Nonlinear Dynamics
• Goodness of fit in nonlinear dynamics: misspecified rates or misspecified states?
• Inference in hidden Markov models.
• Mathematical foundations of neuroscience
• Optimal design for estimation in diffusion processes from first hitting times
• State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms

Cited in

(8)

• Control Theory and Experimental Design in Diffusion Processes
• Markovian experiments that approximate optimal designs for quantal response curves
• Goodness of fit in nonlinear dynamics: misspecified rates or misspecified states?
• Markov Bases and Designed Experiments
• Structured policies in the sequential design of experiments
• Design of experiments for the calibration of history-dependent models via deep reinforcement learning and an enhanced Kalman filter
• Transient dynamics of Pearson diffusions facilitates estimation of rate parameters
• Timing observations of diffusions

This page was built for publication: Experimental design for partially observed Markov decision processes

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3176233)