Experimental design for partially observed Markov decision processes

DOI10.1137/16M1084924zbMATH Open1391.90634arXiv1209.4019OpenAlexW2962776633WikidataQ130014496 ScholiaQ130014496MaRDI QIDQ3176233FDOQ3176233

Authors: Leifur Thorbergsson, Giles Hooker 

Publication date: 19 July 2018

Published in: SIAM/ASA Journal on Uncertainty Quantification (Search for Journal in Brave)

Abstract: This paper deals with the question of how to most effectively conduct experiments in Partially Observed Markov Decision Processes so as to provide data that is most informative about a parameter of interest. Methods from Markov decision processes, especially dynamic programming, are introduced and then used in an algorithm to maximize a relevant Fisher Information. The algorithm is then applied to two POMDP examples. The methods developed can also be applied to stochastic dynamical systems, by suitable discretization, and we consequently show what control policies look like in the Morris-Lecar Neuron model, and simulation results are presented. We discuss how parameter dependence within these methods can be dealt with by the use of priors, and develop tools to update control policies online. This is demonstrated in another stochastic dynamical system describing growth dynamics of DNA template in a PCR model.

Full work available at URL: https://arxiv.org/abs/1209.4019

Recommendations

• A survey of solution techniques for the partially observed Markov decision process
• Observer and control design in partially observable finite Markov chains
• A tutorial on partially observable Markov decision processes
• Control Theory and Experimental Design in Diffusion Processes
• Computationally Feasible Bounds for Partially Observed Markov Decision Processes

zbMATH Keywords

experimental designdynamic programmingMarkov decision processoptimal controlvalue iteration algorithm

Mathematics Subject Classification ID

Point estimation (62F10) Optimal statistical designs (62K05) Dynamic programming (90C39) Markov and semi-Markov decision processes (90C40)

Cites Work

• Bayesian experimental design: A review
• State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms
• Inference in hidden Markov models.
• Automated reverse engineering of nonlinear dynamical systems
• Mathematical foundations of neuroscience
• Branching Processes
• Approximate Dynamic Programming
• Title not available (Why is that?)
• Forcing Function Diagnostics for Nonlinear Dynamics
• Control Theory and Experimental Design in Diffusion Processes
• Goodness of fit in nonlinear dynamics: misspecified rates or misspecified states?
• Optimal design for estimation in diffusion processes from first hitting times

Cited In (8)

• Markovian experiments that approximate optimal designs for quantal response curves
• Goodness of fit in nonlinear dynamics: misspecified rates or misspecified states?
• Markov Bases and Designed Experiments
• Structured policies in the sequential design of experiments
• Design of experiments for the calibration of history-dependent models via deep reinforcement learning and an enhanced Kalman filter
• Transient dynamics of Pearson diffusions facilitates estimation of rate parameters
• Timing observations of diffusions
• Control Theory and Experimental Design in Diffusion Processes

This page was built for publication: Experimental design for partially observed Markov decision processes

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3176233)