Optimal control of continuous-time Markov chains with noise-free observation

DOI10.1137/17M1139989zbMATH Open1387.93184arXiv1707.07202WikidataQ129734828 ScholiaQ129734828MaRDI QIDQ4563379FDOQ4563379

Authors: Alessandro Calvia 

Publication date: 1 June 2018

Published in: SIAM Journal on Control and Optimization (Search for Journal in Brave)

Abstract: We consider an infinite horizon optimal control problem for a continuous-time Markov chain $X$ in a finite set $I$ with noise-free partial observation. The observation process is defined as $Y_t=h(X_t)$, $tgeq0$, where $h$ is a given map defined on $I$. The observation is noise-free in the sense that the only source of randomness is the process $X$ itself. The aim is to minimize a discounted cost functional and study the associated value function $V$. After transforming the control problem with partial observation into one with complete observation (the separated problem) using filtering equations, we provide a link between the value function $v$ associated to the latter control problem and the original value function $V$. Then, we present two different characterizations of $v$ (and indirectly of $V$): on one hand as the unique fixed point of a suitably defined contraction mapping and on the other hand as the unique constrained viscosity solution (in the sense of Soner) of a HJB integro-differential equation. Under suitable assumptions, we finally prove the existence of an optimal control.

Full work available at URL: https://arxiv.org/abs/1707.07202

Recommendations

• Stochastic filtering and optimal control of pure jump Markov processes with noise-free partial observation
• A partially observed control problem for Markov chains
• Optimal control of partially observable piecewise deterministic Markov processes
• Ergodic control of partially observed Markov chains
• The mean squared loss control problem for a partially observed Markov chain

zbMATH Keywords

viscosity solutionscontinuous-time Markov chainsBellman equationpiecewise deterministic Markov processespartial observation control problem

Mathematics Subject Classification ID

Markov chains (discrete-time Markov processes on discrete state spaces) (60J10) Viscosity solutions to Hamilton-Jacobi equations in optimal control and differential games (49L25) Optimal stochastic control (93E20)

Cites Work

• Multivariate point processes: predictable projection, Radon-Nikodym derivatives, representation of martingales
• Applied Probability and Queues
• Title not available (Why is that?)
• Functional analysis, Sobolev spaces and partial differential equations
• Viscosity solutions of Hamilton-Jacobi equations
• Controlled Markov processes and viscosity solutions
• Optimal control of piecewise deterministic markov process
• Title not available (Why is that?)
• Optimal Control with State-Space Constraint I
• Title not available (Why is that?)
• Title not available (Why is that?)
• Stochastic optimal control. The discrete time case
• Title not available (Why is that?)
• Markov Chains
• Filtering of continuous-time Markov chains with noise-free observation and applications
• Point processes and queues. Martingale dynamics
• Constrained and unconstrained optimal discounted control of piecewise deterministic Markov processes
• Optimal Control with State-Space Constraint. II
• Continuous average control of piecewise deterministic Markov processes
• Some topological properties of vector measures with bounded variation and its applications
• An introduction to stochastic filtering theory.
• Controlled Jump Markov Models
• A dynamic programming algorithm for the optimal control of piecewise deterministic Markov processes
• On Reducing a Jump Controllable Markov Model to a Model with Discrete Time
• Title not available (Why is that?)
• Necessary and sufficient optimality conditions for control of piecewise deterministic markov processes
• Nonlinear filtering with signal dependent observation noise
• Least-squares state estimation of systems with state-dependent observation noise
• Filtering for nonlinear systems driven by nonwhite noises:an approximation scheme
• On the optimal control of partially observed inventory systems
• Optimal control of piecewise deterministic Markov processes: a BSDE representation of the value function
• Title not available (Why is that?)
• Constrained BSDEs driven by a non-quasi-left-continuous random measure and optimal control of PDMPs on bounded domains

Cited In (5)

• State constrained control problems in Banach lattices and applications
• Stochastic filtering of a pure jump process with predictable jumps and path-dependent local characteristics
• Stochastic filtering and optimal control of pure jump Markov processes with noise-free partial observation
• Filtering method for linear and non-linear stochastic optimal control of partially observable systems. II
• Nonlinear filtering of partially observed systems arising in singular stochastic optimal control

This page was built for publication: Optimal control of continuous-time Markov chains with noise-free observation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4563379)