Estimating scale-invariant future in continuous time

DOI10.1162/NECO_A_01171MaRDI QIDQ5154136zbMATH OpenOpenAlexDBLPWikidataFDO

Authors Zoran Tiganj, Samuel J. Gershman, Per B. Sederberg, Marc W. Howard

Publication date 1 October 2021

Published in Neural Computation (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1802.06426, http://europepmc.org/articles/pmc6959535

Mathematics Subject Classification ID

Neural networks for/in biological studies, artificial life and related topics (92B20) Memory and learning in psychology (91E40)

Abstract: Natural learners must compute an estimate of future outcomes that follow from a stimulus in continuous time. Widely used reinforcement learning algorithms discretize continuous time and estimate either transition functions from one step to the next (model-based algorithms) or a scalar value of exponentially-discounted future reward using the Bellman equation (model-free algorithms). An important drawback of model-based algorithms is that computational cost grows linearly with the amount of time to be simulated. On the other hand, an important drawback of model-free algorithms is the need to select a time-scale required for exponential discounting. We present a computational mechanism, developed based on work in psychology and neuroscience, for computing a scale-invariant timeline of future outcomes. This mechanism efficiently computes an estimate of inputs as a function of future time on a logarithmically-compressed scale, and can be used to generate a scale-invariant power-law-discounted estimate of expected future reward. The representation of future time retains information about what will happen when. The entire timeline can be constructed in a single parallel operation which generates concrete behavioral and neural predictions. This computational mechanism could be incorporated into future reinforcement learning algorithms.

Recommendations

• A scale-invariant internal representation of time
• scientific article; zbMATH DE number 1784904
• Hyperbolically Discounted Temporal Difference Learning
• Neural mechanism to simulate a scale-invariant future
• scientific article; zbMATH DE number 1683883

Cites work

• A distributed representation of temporal context
• A scale-invariant internal representation of time
• Optimally fuzzy temporal memory
• Reinforcement learning. An introduction
• Stimulus Representation and the Timing of Reward-Prediction Errors in Models of the Dopamine System

Cited in

(3)

• Strange properties of linear reservoirs in the infinitely large limit for prediction of continuous-time signals
• Neural mechanism to simulate a scale-invariant future
• A scale-invariant internal representation of time

This page was built for publication: Estimating scale-invariant future in continuous time

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5154136)