Controlling a Stochastic Process with Unknown Parameters

From MaRDI portal





The problem of controlling a stochastic process with unknown parameters over an infinite horizon with discounting is considered. The derivations given are mathematically rigorous. The decisionmaker or agent faces a tradeoff between current reward and information which may be useful in the future. The emphasis is on the behaviour of the sequence of beliefs, i.e., how much learning occurs when the agent acts optionally. The paper starts with a brief overview of some existing literature on estimation and learning control problems; the bandit problem is a related problem in that the action space (the arms) is discrete, whereas in the current paper it may be continuous. Section 2 describes the decision problem in mathematical terms; it is discrete time. The underlying distribution function \(\mu\) which governs the evolution in time depends on a parameter \(\theta\) which is initially not known to the decisionmaker. The section concludes with a number of technical assumptions. In section 3 the functional equation for the value function is derived, using the dynamic programming approach, with beliefs (the prior distributions) as state variables. In section 4 the limit between behaviour of the sequence of beliefs \((\mu_ i\), \(i=1,2,...)\) is investigated. Convergence to a limit distribution is almost always assured. The problem is, however, that the limit distribution is not necessarily the true distribution, which is clearly indicated in sections 5 and 6. Section 5 deals with special cases such as one parameter problems and low discount rates. For a low discount rate certain convexity results can be proved. Section 6 deals with an example which clearly shows the features described above. For certain decisions the beliefs turn out to be invariant.




Cited in
(63)








This page was built for publication: Controlling a Stochastic Process with Unknown Parameters

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3808001)