Robust optimal well control using an adaptive multigrid reinforcement learning framework

DOI10.1007/S11004-022-10033-XMaRDI QIDQ6084341zbMATH OpenFDO

Authors Atish Dixit, Ahmed H. Elsheikh

Publication date 31 October 2023

Published in Mathematical Geosciences (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2207.03253

zbMATH Keywords

adaptivemultigrid frameworkreinforcement learningrobust optimal controltransfer learning

Mathematics Subject Classification ID

Multigrid methods; domain decomposition for boundary value problems involving PDEs (65N55) Learning and adaptive systems in artificial intelligence (68T05) Computational methods for problems pertaining to geophysics (86-08)

Abstract: Reinforcement learning (RL) is a promising tool to solve robust optimal well control problems where the model parameters are highly uncertain, and the system is partially observable in practice. However, RL of robust control policies often relies on performing a large number of simulations. This could easily become computationally intractable for cases with computationally intensive simulations. To address this bottleneck, an adaptive multi-grid RL framework is introduced which is inspired by principles of geometric multi-grid methods used in iterative numerical algorithms. RL control policies are initially learned using computationally efficient low fidelity simulations using coarse grid discretization of the underlying partial differential equations (PDEs). Subsequently, the simulation fidelity is increased in an adaptive manner towards the highest fidelity simulation that correspond to finest discretization of the model domain. The proposed framework is demonstrated using a state-of-the-art, model-free policy-based RL algorithm, namely the Proximal Policy Optimisation (PPO) algorithm. Results are shown for two case studies of robust optimal well control problems which are inspired from SPE-10 model 2 benchmark case studies. Prominent gains in the computational efficiency is observed using the proposed framework saving around 60-70% of computational cost of its single fine-grid counterpart.

Recommendations

• Reinforcement learning for a class of continuous-time input constrained optimal control problems
• A robust, multi-solution framework for well placement and control optimization
• Gradient-free strategies to robust well control optimization
• Reinforcement learning-based direct adaptive optimal control of JLQ model
• Robust reinforcement learning for stochastic linear quadratic control with multiplicative noise

Cites work

• Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
• Differential evolution -- a simple and efficient heuristic for global optimization over continuous spaces
• scientific article; zbMATH DE number 7626783 (Why is no real title available?)
• Robust control in water management
• Transfer learning for reinforcement learning domains: a survey

Cited in

(3)

• Efficient real-time reservoir management using adjoint-based optimal control and model updating
• Deep reinforcement learning for optimal well control in subsurface systems with uncertain geology
• Multi-asset closed-loop reservoir management using deep reinforcement learning

This page was built for publication: Robust optimal well control using an adaptive multigrid reinforcement learning framework

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6084341)