Uncovering instabilities in variational-quantum deep Q-networks

DOI10.1016/J.JFRANKLIN.2022.08.021arXiv2202.05195OpenAlexW4293051376MaRDI QIDQ6136452FDOQ6136452

Authors: Maja Franz, Lucas Wolf, Maniraman Periyasamy, Christian Ufrecht, Daniel D. Scherer, Axel Plinge, Christopher Mutschler, Wolfgang Mauerer 

Publication date: 17 January 2024

Published in: Journal of the Franklin Institute (Search for Journal in Brave)

Abstract: Deep Reinforcement Learning (RL) has considerably advanced over the past decade. At the same time, state-of-the-art RL algorithms require a large computational budget in terms of training time to converge. Recent work has started to approach this problem through the lens of quantum computing, which promises theoretical speed-ups for several traditionally hard tasks. In this work, we examine a class of hybrid quantum-classical RL algorithms that we collectively refer to as variational quantum deep Q-networks (VQ-DQN). We show that VQ-DQN approaches are subject to instabilities that cause the learned policy to diverge, study the extent to which this afflicts reproduciblity of established results based on classical simulation, and perform systematic experiments to identify potential explanations for the observed instabilities. Additionally, and in contrast to most existing work on quantum reinforcement learning, we execute RL algorithms on an actual quantum processing unit (an IBM Quantum Device) and investigate differences in behaviour between simulated and physical quantum systems that suffer from implementation deficiencies. Our experiments show that, contrary to opposite claims in the literature, it cannot be conclusively decided if known quantum approaches, even if simulated without physical imperfections, can provide an advantage as compared to classical approaches. Finally, we provide a robust, universal and well-tested implementation of VQ-DQN as a reproducible testbed for future experiments.

Full work available at URL: https://arxiv.org/abs/2202.05195

Recommendations

• An efficient and scalable variational quantum circuits approach for deep reinforcement learning
• Quantum reinforcement learning. Comparing quantum annealing and gate-based quantum computing with classical deep reinforcement learning
• Variational quantum algorithms: fundamental concepts, applications and challenges
• An RNN-policy gradient approach for quantum architecture search
• A hybrid quantum-classical generative adversarial networks algorithm based on inherited layerwise learning with circle-connectivity circuit

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Quantum computation (81P68)

Cites Work

• Machine learning. A probabilistic perspective
• Deep learning
• \({\mathcal Q}\)-learning
• Title not available (Why is that?)
• Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer
• An analysis of temporal-difference learning with function approximation
• THE PROBABLE ERROR OF A MEAN
• Title not available (Why is that?)
• End-to-end training of deep visuomotor policies

Cited In (2)

• An efficient and scalable variational quantum circuits approach for deep reinforcement learning
• Automatic evolutionary design of quantum rule-based systems and applications to quantum reinforcement learning

This page was built for publication: Uncovering instabilities in variational-quantum deep Q-networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6136452)