Dynamic Programming Conditions for Partially Observable Stochastic Systems
The problem of minimizing \[ J(U) = E \int_0^T c(t,x_t,u_t)(\,dt, \tag{1} \] over the class of controls \(u_t = u(t,x_t)\), \(u_t \in E\subset R^k\), subject to \[ dx_t = f(t,x_t,u_t)\,dt + \sigma(t,x_t)\,dB_t \tag{2} \] (Ito equation; \(B = \) Wiener process) centres around solution of the Bellman-Hamilton-Jacobi equation \(\Lambda W + min_{u\in\Xi} (\nabla W\cdot f + c) = 0\), where \[ W=W(t,x),\quad \nabla W = \frac{\partial W}{\partial x}\quad\text{and}\quad \Lambda W = \frac{\partial W}{\partial t} + \frac12 \sum_{ij} (\sigma\sigma')_{ij} \frac{\partial^2 W}{\partial x_i\partial x_j}. \] If this equation has a smooth solution \(W\), then (i) \(W\) is the``value function (the minimum cost over \([t,1]\) starting at \(x_t =x)\), (ii) \(W\) can be expanded (by the Ito differential rule) as \[ W_t = J(u^0) + \int_0^T \Lambda W \,dt + \int_0^T \nabla W \,dx \tag{3} \] and (iii) the optimal policy is \(u^0(t,x) = \lambda(t,x,\nabla W)\) where \(\lambda(t,x,p)\) is the value of \(u\in\Xi\) which minimizes \(p\cdot f(t,x,u)+h(t,x,u)\). See \textit{W. H. Fleming} [SIAM Rev. 11, 470--509 (1969; Zbl 0192.52501)]. The objective of this paper is to obtain similar characterizations of the optimal policy for problems similar to (1), (2) but possibly non-Markovian and with partial observations. The solution of (2) is defined by the Girsanov measure transformation technique. Then value functions (processes) \(W_1(t)\) are defined directly and shown to satisfy a version of Bellman's optimality principle. A control \(u\) is called ``value decreasing if the process \(W_n(t)\) is a supermartingale. For such controls we can apply the Meyer decomposition of supermartingales and write \(W_n(t) = -A_t + M_t\), where \(A_t\) is an increasing process and \(M_t\) a martingale. It is shown that \(A_t\) is absolutely continuous w.r.t. Lebesgue measure and \(M_t\) has a stochastic integral representation. Thus a representation of \(W_1(t)\) of the form (3) is obtained \((\Lambda W_n\) and \(\nabla W_n\) are now abstract processes) and conditions for optimality can be derived using this together with Bellman's principle. In the case of complete observations there is one value process \(W\) (the same for all controls), which has the above representation. In a subsequent paper [the first author, ibid. 11, 587--594 (1973; Zbl 0238.93044)] it has been shown that (iii) above holds, and this leads to an existence theorem for the optimal policy since the process \(W\) is defined whether or not an optimal policy exists.
- Dynamic programming for ergodic control with partial observations.
- Dynamic programming and stochastic control
- Optimality conditions for partially observable Markov decision processes
- scientific article; zbMATH DE number 4191401
- Dynamic programming in stochastic control of systems with delay
- scientific article; zbMATH DE number 49106
- Dynamic programming for multidimensional stochastic control problems
- Partially observable multistage stochastic programming
- Least-squares state estimation of systems with state-dependent observation noise
- The probabilistic structure of controlled diffusion processes
- Optimal stopping of continuous time stochastic processes and stochastic differential representations for the value functions
- Optimality criteria for controlled discontinuous processes
- Stochastic control by measure transformation: A general existence result
- On the optimal control of stochastic systems with an exponential-of- integral performance index
- Finite-dimensional attainable sets for stochastic control systems
- Optimal control of diffusion processes with reflection
- Stochastic control of system with unobserved jump parameter process
- On the separation principle with bounded controls
- Exit probabilities and optimal stochastic control
- Dynamic programming optimality criteria for stochastic systems in Riemannian manifolds
- Lagrange approach to the optimal control of diffusions
- Conjugate convex functions in optimal stochastic control
- Nonanticipative risk sensitive control: the martingale method.
- Existence and uniqueness results for BSDE with jumps: the whole nine yards
- Strategies using an observer for steering a random motion of a point in a multitarget environment
- A general problem of an optimal equivalent change of measure and contingent claim pricing in an incomplete market.
- Control with partial observations and an explicit solution of Mortensen's equation
- Control problem for diffusion-type random fields
- A unified approach to well-posedness of type-I backward stochastic Volterra integral equations
- Optimal contracting under mean-volatility joint ambiguity uncertainties
- Interview with Ulf Hashagen: exhibitions and mathematical models in the nineteenth and twentieth centuries
- On the stochastic control-stopping problem
- Geometry of information structures, strategic measures and associated stochastic control topologies
- Application of conditional-optimal filter for synthesis of suboptimal control in the problem of optimizing the output of a nonlinear differential stochastic system
- The optimal investment, liability and dividends in insurance
- Backward stochastic partial differential equations related to utility maximization and hedging
- Portfolio optimization for an investor with a benchmark
- Optimal compensation with hidden action and lump-sum payment in a continuous-time model
- Optimal compensation with adverse selection and dynamic actions
- An invariance principle in large population stochastic dynamic games
- Dynamic programming for ergodic control with partial observations.
- BSDEs and risk-sensitive control, zero-sum and nonzero-sum game problems of stochastic functional differential equations.
- A partial history of the early development of continuous-time nonlinear stochastic systems theory
- The optimal control of diffusions
- Zero-sum path-dependent stochastic differential games in weak formulation
- Sequential stochastic control (single or multi-agent) problems nearly admit change of measures with independent measurement
- Risk-Sensitive Mean-Field Type Control Under Partial Observation
- Decentralized optimality conditions of stochastic differential decision problems via Girsanov's measure transformation
- Adjoint processes in stochastic optimal control problems
- A necessary condition for optimality in a problem of stochastic control with discretized observations
- On the existence of solutions to stochastic differential equations on Loeb spaces
- Separation principle for impulse control with partial information
- Forward and backward semimartingale models for gaussian processes with stationary increments
- Optimal locally absolutely continuous change of measure. finite set of decisions. part i
- Optimal locally absolutely continuous change of measure. finite set of decisions. part ii:optimization problems
- Nash-equilibrium in stochastic differential games
- Strong envelopes of stochastic processes and a penalty method†
- scientific article; zbMATH DE number 3745962 (Why is no real title available?)
- Bang-bang partially observable feedback strategies for a rendezvous problem†
- Local optimality conditions for optimal stopping
- Optimal control of a jump process
- A stochastic minimum principle
- scientific article; zbMATH DE number 3554125 (Why is no real title available?)
- Techniques probabilistes dans le contrôle impulsionnel
- Transaction costs, trading volume, and the liquidity premium
- The value function in ergodic control of diffusion processes with partial observations
- Singular stochastic control and optimal stopping
- Optimal control of semi-Markov processes with a backward stochastic differential equations approach
- The limits of leverage
- Optimal switching problems of tandem type
- Young, timid, and risk takers
- Simulation-based optimization of Markov decision processes: an empirical process theory approach
- Encounters with Martingales in Stochastic Control
- Dissipative stochastic dynamical systems
- Operational absolutely optimal dynamic control of the stochastic differential plant's state by its output
- Optimal finite-dimensional controller of the stochastic differential object's state by its output. I: Incomplete precise measurements
- Reflections on BSDEs
- Gaussian agency problems with memory and linear contracts
- Lagrange lemma and the optimal control of diffusions. II: Nonlinear Lagrange functionals
- Zero-sum stochastic differential games and backward equations
- Verification theorem related to a zero sum stochastic differential game based on a chain rule for nonsmooth functions
- Mokobodzki's intervals: an approach to Dynkin games when value process is not a semimartingale
- Partially observable optimal control using exponential cost criterion
- Shortfall aversion on a finite horizon
- The dynamic programming method in systems with states in the form of distributions
- Computation of suboptimal randomized strategies for steering the random motion of a point under partial observation
- Martingale approach to stochastic differential games of control and stopping
This page was built for publication: Dynamic Programming Conditions for Partially Observable Stochastic Systems
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5675611)