Uncovering instabilities in variational-quantum deep Q-networks

DOI10.1016/J.JFRANKLIN.2022.08.021MaRDI QIDQ6136452zbMATH OpenOpenAlexFDO

Authors Maja Franz, Lucas Wolf, Maniraman Periyasamy, Christian Ufrecht, Daniel D. Scherer, Axel Plinge, Christopher Mutschler, Wolfgang Mauerer

Publication date 17 January 2024

Published in Journal of the Franklin Institute (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2202.05195

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Quantum computation (81P68)

Abstract: Deep Reinforcement Learning (RL) has considerably advanced over the past decade. At the same time, state-of-the-art RL algorithms require a large computational budget in terms of training time to converge. Recent work has started to approach this problem through the lens of quantum computing, which promises theoretical speed-ups for several traditionally hard tasks. In this work, we examine a class of hybrid quantum-classical RL algorithms that we collectively refer to as variational quantum deep Q-networks (VQ-DQN). We show that VQ-DQN approaches are subject to instabilities that cause the learned policy to diverge, study the extent to which this afflicts reproduciblity of established results based on classical simulation, and perform systematic experiments to identify potential explanations for the observed instabilities. Additionally, and in contrast to most existing work on quantum reinforcement learning, we execute RL algorithms on an actual quantum processing unit (an IBM Quantum Device) and investigate differences in behaviour between simulated and physical quantum systems that suffer from implementation deficiencies. Our experiments show that, contrary to opposite claims in the literature, it cannot be conclusively decided if known quantum approaches, even if simulated without physical imperfections, can provide an advantage as compared to classical approaches. Finally, we provide a robust, universal and well-tested implementation of VQ-DQN as a reproducible testbed for future experiments.

Recommendations

• An efficient and scalable variational quantum circuits approach for deep reinforcement learning
• Quantum reinforcement learning. Comparing quantum annealing and gate-based quantum computing with classical deep reinforcement learning
• Variational quantum algorithms: fundamental concepts, applications and challenges
• An RNN-policy gradient approach for quantum architecture search
• A hybrid quantum-classical generative adversarial networks algorithm based on inherited layerwise learning with circle-connectivity circuit

Cites work

• scientific article; zbMATH DE number 1256737 (Why is no real title available?)
• scientific article; zbMATH DE number 1775388 (Why is no real title available?)
• An analysis of temporal-difference learning with function approximation
• Deep learning
• End-to-end training of deep visuomotor policies
• Machine learning. A probabilistic perspective
• Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer
• THE PROBABLE ERROR OF A MEAN
• \({\mathcal Q}\)-learning

Cited in

(2)

• An efficient and scalable variational quantum circuits approach for deep reinforcement learning
• Automatic evolutionary design of quantum rule-based systems and applications to quantum reinforcement learning

This page was built for publication: Uncovering instabilities in variational-quantum deep Q-networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6136452)