Robust risk-aware reinforcement learning

DOI10.1137/21M144640XMaRDI QIDQ5065087zbMATH OpenOpenAlexFDO

Authors Sebastian Jaimungal, Silvana M. Pesenti, Ye Sheng Wang, Hariom Tatsat

Publication date 18 March 2022

Published in SIAM Journal on Financial Mathematics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2108.10403

zbMATH Keywords

Wasserstein distancerisk measuresportfolio optimizationrobust optimizationreinforcement learningstatistical arbitrage

Mathematics Subject Classification ID

Statistical methods; risk measures (91G70) Portfolio theory (91G10) Robustness in mathematical programming (90C17)

Abstract: We present a reinforcement learning (RL) approach for robust optimisation of risk-aware performance criteria. To allow agents to express a wide variety of risk-reward profiles, we assess the value of a policy using rank dependent expected utility (RDEU). RDEU allows the agent to seek gains, while simultaneously protecting themselves against downside risk. To robustify optimal policies against model uncertainty, we assess a policy not by its distribution, but rather, by the worst possible distribution that lies within a Wasserstein ball around it. Thus, our problem formulation may be viewed as an actor/agent choosing a policy (the outer problem), and the adversary then acting to worsen the performance of that strategy (the inner problem). We develop explicit policy gradient formulae for the inner and outer problems, and show its efficacy on three prototypical financial problems: robust portfolio allocation, optimising a benchmark, and statistical arbitrage.

Recommendations

• Risk-sensitive reinforcement learning
• Risk-sensitive reinforcement learning
• Risk-constrained reinforcement learning with percentile risk criteria
• Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
• Reinforcement learning in robust Markov decision processes

Cites work

• scientific article; zbMATH DE number 1975241 (Why is no real title available?)
• scientific article; zbMATH DE number 7307478 (Why is no real title available?)
• scientific article; zbMATH DE number 2243395 (Why is no real title available?)
• Algorithmic trading with model uncertainty
• Ambiguity in portfolio selection
• Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations
• Envelope Theorems for Arbitrary Choice Sets
• Nonparametric kernel density estimation and its computational aspects
• On the intuition of rank-dependent utility
• Practical augmented Lagrangian methods for constrained optimization
• Risk-sensitive reinforcement learning
• The Dual Theory of Choice under Risk

Cited in

(8)

• Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning
• Recent advances in reinforcement learning in finance
• Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
• Recent developments in machine learning methods for stochastic control and games
• Reinforcement learning with dynamic convex risk measures
• Robust Risk-Aware Option Hedging
• Risk-sensitive reinforcement learning
• Risk-sensitive reinforcement learning algorithms with generalized average criterion

Describes a project that uses

Uses Software

• ALGENCAN

This page was built for publication: Robust risk-aware reinforcement learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5065087)