Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective

DOI10.1007/S11229-021-03141-4MaRDI QIDQ6182771zbMATH OpenOpenAlexFDO

Authors Tom Everitt, Marcus Hutter, Ramana Kumar, Victoria Krakovna

Publication date 26 January 2024

Published in Synthese (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1908.04734

zbMATH Keywords

decision theorycausalityreinforcement learningBayesian learningAGI safetycausal influence diagrams

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Causal inference from observational studies (62D20) General considerations in statistical decision theory (62C05) Agent technology and artificial intelligence (68T42)

Abstract: Can humans get arbitrarily capable reinforcement learning (RL) agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design principles that prevent instrumental goals for two different types of reward tampering (reward function tampering and RF-input tampering). Combined, the design principles can prevent both types of reward tampering from being instrumental goals. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations.

Recommendations

• Reward is enough
• Learning reward machines: a study in partially observable reinforcement learning
• Reward machines: exploiting reward function structure in reinforcement learning
• Framing reinforcement learning from human reward: reward positivity, temporal discounting, episodicity, and performance
• Specification-guided reinforcement learning

Cites work

• A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
• Artificial general intelligence 2008. Proceedings of the 1st AGI conference.
• Causality. Models, reasoning, and inference
• Complete identification methods for the causal hierarchy
• General time consistent discounting
• Multi-agent influence diagrams for representing and solving games.
• Planning and acting in partially observable stochastic domains
• Reinforcement learning. An introduction
• Representing and solving decision problems with limited information

Cited in

(1)

• On Imperfect Recall in Multi-Agent Influence Diagrams

This page was built for publication: Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6182771)