Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
From MaRDI portal
(Redirected from Publication:5870485)
Abstract: Keeping risk under control is often more crucial than maximizing expected rewards in real-world decision-making situations, such as finance, robotics, autonomous driving, etc. The most natural choice of risk measures is variance, which penalizes the upside volatility as much as the downside part. Instead, the (downside) semivariance, which captures the negative deviation of a random variable under its mean, is more suitable for risk-averse proposes. This paper aims at optimizing the mean-semivariance (MSV) criterion in reinforcement learning w.r.t. steady reward distribution. Since semivariance is time-inconsistent and does not satisfy the standard Bellman equation, the traditional dynamic programming methods are inapplicable to MSV problems directly. To tackle this challenge, we resort to Perturbation Analysis (PA) theory and establish the performance difference formula for MSV. We reveal that the MSV problem can be solved by iteratively solving a sequence of RL problems with a policy-dependent reward function. Further, we propose two on-policy algorithms based on the policy gradient theory and the trust region method. Finally, we conduct diverse experiments from simple bandit problems to continuous control tasks in MuJoCo, which demonstrate the effectiveness of our proposed methods.
Recommendations
- Risk-sensitive reinforcement learning
- Mean-semivariance optimality for continuous-time Markov decision processes
- Robust risk-aware reinforcement learning
- Variance-constrained actor-critic algorithms for discounted and average reward MDPs
- Mean-variance optimization of discrete time discounted Markov decision processes
Cites work
- A comprehensive survey on safe reinforcement learning
- A multi-period fuzzy portfolio optimization model with minimum transaction lots
- A possibilistic mean-semivariance-entropy model for multi-period portfolio selection with transaction costs
- Computation of mean-semivariance efficient sets by the critical line algorithm
- Convex Approximations of Chance Constrained Programs
- Lectures on stochastic programming. Modeling and theory
- Mean-semivariance optimality for continuous-time Markov decision processes
- Mean-Variance Tradeoffs in an Undiscounted MDP: The Unichain Case
- Multi-period mean-semivariance portfolio optimization based on uncertain measure
- Multi-period semi-variance portfolio selection: model and numerical solution
- Optimal dynamic portfolio selection: multiperiod mean-variance formulation
- Optimization of Markov decision processes under the variance criterion
- Reinforcement learning. An introduction
- Risk-averse dynamic programming for Markov decision processes
- Risk-constrained reinforcement learning with percentile risk criteria
- Risk-Sensitive Markov Decision Processes
- Risk-Sensitive Optimal Control for Markov Decision Processes with Monotone Cost
- Sequential Decision Making With Coherent Risk
- Stochastic learning and optimization. A sensitivity-based approach.
- The variance of discounted Markov decision processes
- Variance-constrained actor-critic algorithms for discounted and average reward MDPs
- Variance-Penalized Markov Decision Processes
- Variance-penalized Markov decision processes: dynamic programming and reinforcement learning techniques
Cited in
(3)
This page was built for publication: Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5870485)
