Entropy Maximization for Partially Observable Markov Decision Processes

DOI10.1109/TAC.2022.3183564MaRDI QIDQ6053218zbMATH OpenOpenAlexFDO

Authors Yagiz Savas, Bo Wu, Takashi Tanaka, Ufuk Topcu

Publication date 26 September 2023

Published in IEEE Transactions on Automatic Control (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2105.07490

zbMATH Keywords

entropystochastic processesautonomous systems

Mathematics Subject Classification ID

Systems theory; control (93-XX)

Abstract: We study the problem of synthesizing a controller that maximizes the entropy of a partially observable Markov decision process (POMDP) subject to a constraint on the expected total reward. Such a controller minimizes the predictability of an agent's trajectories to an outside observer while guaranteeing the completion of a task expressed by a reward function. We first prove that an agent with partial observations can achieve an entropy at most as well as an agent with perfect observations. Then, focusing on finite-state controllers (FSCs) with deterministic memory transitions, we show that the maximum entropy of a POMDP is lower bounded by the maximum entropy of the parametric Markov chain (pMC) induced by such FSCs. This relationship allows us to recast the entropy maximization problem as a so-called parameter synthesis problem for the induced pMC. We then present an algorithm to synthesize an FSC that locally maximizes the entropy of a POMDP over FSCs with the same number of memory states. In numerical examples, we illustrate the relationship between the maximum entropy, the number of memory states in the FSC, and the expected reward.

Recommendations

• Maximizing entropy over Markov processes
• Maximizing entropy over Markov processes
• Entropy Maximization for Markov Decision Processes Under Temporal Logic Constraints
• Entropy maximization for Markov and semi-Markov processes
• scientific article; zbMATH DE number 5058481
• scientific article; zbMATH DE number 888812
• Empirical entropy, minimax regret and minimax risk
• On the way towards a generalized entropy maximization procedure
• scientific article; zbMATH DE number 4136860

Cited in

(3)

• Entropy Maximization for Markov Decision Processes Under Temporal Logic Constraints
• Infinite Time Horizon Maximum Causal Entropy Inverse Reinforcement Learning
• Probabilistic predictability of stochastic dynamical systems

This page was built for publication: Entropy Maximization for Partially Observable Markov Decision Processes

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6053218)