Categorical semantics of compositional reinforcement learning

MaRDI QIDQ6409140zbMATH OpenFDO

Authors Georgios Bakirtzis, Michail Savvas, Ufuk Topcu

Publication date 29 August 2022

Abstract: Reinforcement learning (RL) often requires decomposing a problem into subtasks and composing learned behaviors on these tasks. Compositionality in RL has the potential to create modular subtask units that interface with other system capabilities. However, generating compositional models requires the characterization of minimal assumptions for the robustness of the compositional feature. We develop a framework for a emph{compositional theory} of RL using a categorical point of view. Given the categorical representation of compositionality, we investigate sufficient conditions under which learning-by-parts results in the same optimal policy as learning on the whole. In particular, our approach introduces a category $mathsfMDP$, whose objects are Markov decision processes (MDPs) acting as models of tasks. We show that $mathsfMDP$ admits natural compositional operations, such as certain fiber products and pushouts. These operations make explicit compositional phenomena in RL and unify existing constructions, such as puncturing hazardous states in composite MDPs and incorporating state-action symmetry. We also model sequential task completion by introducing the language of zig-zag diagrams that is an immediate application of the pushout operation in $mathsfMDP$.

This page was built for publication: Categorical semantics of compositional reinforcement learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6409140)