Robust risk-aware reinforcement learning

DOI10.1137/21M144640XzbMATH Open1484.91426arXiv2108.10403OpenAlexW4214588671MaRDI QIDQ5065087FDOQ5065087

Authors: Sebastian Jaimungal, Silvana M. Pesenti, Ye Sheng Wang, Hariom Tatsat 

Publication date: 18 March 2022

Published in: SIAM Journal on Financial Mathematics (Search for Journal in Brave)

Abstract: We present a reinforcement learning (RL) approach for robust optimisation of risk-aware performance criteria. To allow agents to express a wide variety of risk-reward profiles, we assess the value of a policy using rank dependent expected utility (RDEU). RDEU allows the agent to seek gains, while simultaneously protecting themselves against downside risk. To robustify optimal policies against model uncertainty, we assess a policy not by its distribution, but rather, by the worst possible distribution that lies within a Wasserstein ball around it. Thus, our problem formulation may be viewed as an actor/agent choosing a policy (the outer problem), and the adversary then acting to worsen the performance of that strategy (the inner problem). We develop explicit policy gradient formulae for the inner and outer problems, and show its efficacy on three prototypical financial problems: robust portfolio allocation, optimising a benchmark, and statistical arbitrage.

Full work available at URL: https://arxiv.org/abs/2108.10403

Recommendations

• Risk-sensitive reinforcement learning
• Risk-sensitive reinforcement learning
• Risk-constrained reinforcement learning with percentile risk criteria
• Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
• Reinforcement learning in robust Markov decision processes

zbMATH Keywords

Wasserstein distancerisk measuresportfolio optimizationrobust optimizationreinforcement learningstatistical arbitrage

Mathematics Subject Classification ID

Statistical methods; risk measures (91G70) Portfolio theory (91G10) Robustness in mathematical programming (90C17)

Cites Work

• Practical augmented Lagrangian methods for constrained optimization
• The Dual Theory of Choice under Risk
• Envelope Theorems for Arbitrary Choice Sets
• Title not available (Why is that?)
• Algorithmic trading with model uncertainty
• On the intuition of rank-dependent utility
• Ambiguity in portfolio selection
• Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations
• Nonparametric kernel density estimation and its computational aspects
• Risk-sensitive reinforcement learning
• Title not available (Why is that?)
• Title not available (Why is that?)

Cited In (8)

• Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning
• Recent advances in reinforcement learning in finance
• Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
• Recent developments in machine learning methods for stochastic control and games
• Reinforcement learning with dynamic convex risk measures
• Robust Risk-Aware Option Hedging
• Risk-sensitive reinforcement learning
• Risk-sensitive reinforcement learning algorithms with generalized average criterion

Uses Software

• ALGENCAN

This page was built for publication: Robust risk-aware reinforcement learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5065087)