Learning in structured MDPs with convex cost functions: improved regret bounds for inventory management

DOI10.1287/OPRE.2022.2263zbMATH Open1494.90004arXiv1905.04337OpenAlexW2944461362MaRDI QIDQ5095166FDOQ5095166

Randy Jia, Shipra Agrawal

Publication date: 5 August 2022

Published in: Operations Research (Search for Journal in Brave)

Abstract: We consider a stochastic inventory control problem under censored demands, lost sales, and positive lead times. This is a fundamental problem in inventory management, with significant literature establishing near-optimality of a simple class of policies called ``base-stock policies for the underlying Markov Decision Process (MDP), as well as convexity of long run average-cost under those policies. We consider the relatively less studied problem of designing a learning algorithm for this problem when the underlying demand distribution is unknown. The goal is to bound regret of the algorithm when compared to the best base-stock policy. We utilize the convexity properties and a newly derived bound on bias of base-stock policies to establish a connection to stochastic convex bandit optimization. Our main contribution is a learning algorithm with a regret bound of $ildeO(LsqrtT+D)$ for the inventory control problem. Here $L$ is the fixed and known lead time, and $D$ is an unknown parameter of the demand distribution described roughly as the number of time steps needed to generate enough demand for depleting one unit of inventory. Notably, even though the state space of the underlying MDP is continuous and $L$-dimensional, our regret bounds depend linearly on $L$. Our results significantly improve the previously best known regret bounds for this problem where the dependence on $L$ was exponential and many further assumptions on demand distribution were required. The techniques presented here may be of independent interest for other settings that involve large structured MDPs but with convex cost functions.

Full work available at URL: https://arxiv.org/abs/1905.04337

Recommendations

• Technical note: Perishable inventory systems: convexity results for base-stock policies and learning algorithms under censored demand
• Dynamic Inventory Control with Fixed Setup Costs and Unknown Discrete Demand Distribution
• Near-optimal regret bounds for reinforcement learning
• Regret bounds for restless Markov bandits
• Nonparametric learning algorithms for joint pricing and inventory control with lost sales and censored demand

zbMATH Keywords

reinforcement learningregret boundsinventory control problemcensored demandonline convex optimization

Mathematics Subject Classification ID

Convex programming (90C25) Inventory, storage, reservoirs (90B05) Transportation, logistics and supply chain management (90B06)

Cites Work

• Partial Monitoring—Classification, Regret Bounds, and Algorithms
• An Adaptive Algorithm for Finding the Optimal Base-Stock Policy in Lost Sales Inventory Systems with Censored Demand
• Lost-sales inventory theory: a review
• Title not available (Why is that?)
• A Nonparametric Asymptotic Analysis of Inventory Planning with Censored Demand
• Near-optimal regret bounds for reinforcement learning
• A note on the convexity of performance measures of M/M/c queueing systems
• Lost-Sales Problems with Stochastic Lead Times: Convexity Results for Base-Stock Policies
• Asymptotic Optimality of Order-Up-To Policies in Lost Sales Inventory Systems
• Old and New Methods for Lost-Sales Inventory Systems
• Optimal Server Allocation in a System of Multi-Server Stations
• Note—On the Marginal Benefit of Adding Servers to G/GI/m Queues
• Non-stationary stochastic optimization

Cited In (3)

• A least squares temporal difference actor–critic algorithm with applications to warehouse management
• Title not available (Why is that?)
• Deep reinforcement learning for inventory control: a roadmap

This page was built for publication: Learning in structured MDPs with convex cost functions: improved regret bounds for inventory management

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5095166)