Bayesian policy reuse

DOI10.1007/S10994-016-5547-YzbMATH Open1454.68129arXiv1505.00284OpenAlexW778742492MaRDI QIDQ1689554FDOQ1689554

Majd Hawasly, Benjamin Rosman, Subramanian Ramamoorthy

Publication date: 12 January 2018

Published in: Machine Learning (Search for Journal in Brave)

Abstract: A long-lived autonomous agent should be able to respond online to novel instances of tasks from a familiar domain. Acting online requires 'fast' responses, in terms of rapid convergence, especially when the task instance has a short duration, such as in applications involving interactions with humans. These requirements can be problematic for many established methods for learning to act. In domains where the agent knows that the task instance is drawn from a family of related tasks, albeit without access to the label of any given instance, it can choose to act through a process of policy reuse from a library, rather than policy learning from scratch. In policy reuse, the agent has prior knowledge of the class of tasks in the form of a library of policies that were learnt from sample task instances during an offline training phase. We formalise the problem of policy reuse, and present an algorithm for efficiently responding to a novel task instance by reusing a policy from the library of existing policies, where the choice is based on observed 'signals' which correlate to policy performance. We achieve this by posing the problem as a Bayesian choice problem with a corresponding notion of an optimal response, but the computation of that response is in many cases intractable. Therefore, to reduce the computation cost of the posterior, we follow a Bayesian optimisation approach and define a set of policy selection functions, which balance exploration in the policy library against exploitation of previously tried policies, together with a model of expected performance of the policy library on their corresponding task instances. We validate our method in several simulated domains of interactive, short-duration episodic tasks, showing rapid convergence in unknown task variations.

Full work available at URL: https://arxiv.org/abs/1505.00284

zbMATH Keywords

transfer learningBayesian decision theoryonline learningreinforcement learningBayesian optimisationonline banditspolicy reuse

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Online algorithms; streaming algorithms (68W27) Bayesian problems; characterization of Bayes procedures (62C10)

Cites Work

• Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
• Asymptotically efficient adaptive allocation rules
• Computing a classic index for finite-horizon bandits
• Finite-time analysis of the multiarmed bandit problem
• A Structured Multiarmed Bandit Problem and the Greedy Policy
• Transfer learning for reinforcement learning domains: a survey
• Title not available (Why is that?)
• Bayesian policy gradient and actor-critic algorithms
• Title not available (Why is that?)

Cited In (1)

• A Survey of Opponent Modeling in Adversarial Domains

This page was built for publication: Bayesian policy reuse

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1689554)