Reward Maximization Through Discrete Active Inference

DOI10.1162/NECO_A_01574zbMATH Open1520.91292arXiv2009.08111OpenAlexW4328052356WikidataQ123008069 ScholiaQ123008069MaRDI QIDQ6136191FDOQ6136191

Authors: Lancelot Da Costa, Noor Sajid, Thomas Parr, Karl J. Friston, R. N. Smith 

Publication date: 28 August 2023

Published in: Neural Computation (Search for Journal in Brave)

Abstract: Active inference is a probabilistic framework for modelling the behaviour of biological and artificial agents, which derives from the principle of minimising free energy. In recent years, this framework has successfully been applied to a variety of situations where the goal was to maximise reward, offering comparable and sometimes superior performance to alternative approaches. In this paper, we clarify the connection between reward maximisation and active inference by demonstrating how and when active inference agents perform actions that are optimal for maximising reward. Precisely, we show the conditions under which active inference produces the optimal solution to the Bellman equation--a formulation that underlies several approaches to model-based reinforcement learning and control. On partially observed Markov decision processes, the standard active inference scheme can produce Bellman optimal actions for planning horizons of 1, but not beyond. In contrast, a recently developed recursive active inference scheme (sophisticated inference) can produce Bellman optimal actions on any finite temporal horizon. We append the analysis with a discussion of the broader relationship between active inference and reinforcement learning.

Full work available at URL: https://arxiv.org/abs/2009.08111

Recommendations

• Active Inference: Demystified and Compared
• Realizing Active Inference in Variational Message Passing: The Outcome-Blind Certainty Seeker
• Active inference and agency: optimal control without cost functions
• Active inference on discrete state-spaces: a synthesis
• Active inference, belief propagation, and the Bethe approximation

Mathematics Subject Classification ID

Cognitive psychology (91E10) Markov and semi-Markov decision processes (90C40) Memory and learning in psychology (91E40)

Cites Work

• Online planning algorithms for POMDPS
• Title not available (Why is that?)
• Title not available (Why is that?)
• Pattern recognition and machine learning.
• Planning and acting in partially observable stochastic domains
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Statistical decision theory and Bayesian analysis. 2nd ed
• Game theory
• Information Theory and Statistical Mechanics
• Title not available (Why is that?)
• Stochastic optimal control. The discrete time case
• Prospect Theory: An Analysis of Decision under Risk
• On a Measure of the Information Provided by an Experiment
• Robot navigation as hierarchical active inference
• Efficient computation of optimal actions
• Optimal control as a graphical model inference problem
• Reinforcement learning. An introduction
• Title not available (Why is that?)
• Stochastic processes and applications. Diffusion processes, the Fokker-Planck and Langevin equations
• Active inference and agency: optimal control without cost functions
• Title not available (Why is that?)
• If multi-agent learning is the answer, what is the question?
• Optimal control
• Stochastic optimal control of state constrained systems
• Learning to optimize via posterior sampling
• A Bayesian approach for learning and planning in partially observable Markov decision processes
• Title not available (Why is that?)
• Active inference: a process theory
• Active inference, curiosity and insight
• Active inference on discrete state-spaces: a synthesis
• Bayesian reinforcement learning: a survey
• An information-theoretic analysis of Thompson sampling
• A Tutorial on Thompson Sampling
• Scalable and efficient Bayes-adaptive reinforcement learning based on Monte-Carlo tree search
• Title not available (Why is that?)
• Active Inference: Demystified and Compared
• A step-by-step tutorial on active inference and its application to empirical data
• Whence the expected free energy?
• Deep active inference as variational policy gradients
• Sophisticated inference
• Title not available (Why is that?)

Cited In (8)

• Active inference on discrete state-spaces: a synthesis
• Active inference, belief propagation, and the Bethe approximation
• Branching time active inference with Bayesian filtering
• A reinforcement meta-learning framework of executive function and information demand
• Active inference and agency: optimal control without cost functions
• The free energy principle made simpler but not too simple
• Predictive processing in cognitive robotics: a review
• Deep active inference as variational policy gradients

This page was built for publication: Reward Maximization Through Discrete Active Inference

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6136191)