Recovering Markov models from closed-loop data

DOI10.1016/J.AUTOMATICA.2019.01.022MaRDI QIDQ1737809zbMATH OpenOpenAlexWikidataFDO

Authors Jonathan P. Epperlein, Sergiy M. Zhuk, Robert N. Shorten

Publication date 24 April 2019

Published in Automatica (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1706.06359

zbMATH Keywords

Markov models recoveringrecommenders

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Markov chains (discrete-time Markov processes on discrete state spaces) (60J10) Feedback control (93B52) Stochastic systems in control theory (general) (93E03)

Abstract: Situations in which recommender systems are used to augument decision making are becoming prevalent in many application domains. Almost always, these prediction tools (recommenders) are created with a view to affecting behavioural change. Clearly, successful applications actuating behavioural change, affect the original model underpinning the predictor, leading to an inconsistency. This feedback loop is often not considered in standard so-called Big Data learning techniques which rely upon machine learning/statistical learning machinery. The objective of this paper is to develop tools that recover unbiased user models in the presence of recommenders. More specifically, we assume that we observe a time series which is a trajectory of a Markov chain $R$ modulated by another Markov chain $S$, i.e. the transition matrix of $R$ is unknown and depends on the current state of $S$. The transition matrix of the latter is also unknown. In other words, at each time instant, $S$ selects a transition matrix for $R$ within a given set which consists of known and unknown matrices. The state of $S$, in turn, depends on the current state of $R$ thus introducing a feedback loop. We propose an Expectation-Maximization (EM) type algorithm, which estimates the transition matrices of $S$ and $R$. Experimental results are given to demonstrate the efficacy of the approach.

Recommendations

• An MDP-based recommender system
• Modeling mutual feedback between users and recommender systems
• Technical note: An expectation-maximization algorithm to estimate the parameters of the Markov chain choice model
• On targeting Markov segments
• Learning Markov models via low-rank optimization

Cites work

• scientific article; zbMATH DE number 1818892 (Why is no real title available?)
• scientific article; zbMATH DE number 44386 (Why is no real title available?)
• scientific article; zbMATH DE number 1493045 (Why is no real title available?)
• scientific article; zbMATH DE number 2136426 (Why is no real title available?)
• A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains
• An EM Algorithm for Markov Modulated Markov Processes
• An Introduction to Regime Switching Time Series Models
• Closed-loop identification revisited
• Counterfactual reasoning and learning systems: the example of computational advertising
• Identification and control -- Closed-loop issues
• Markov chains and mixing times. With a chapter on ``Coupling from the past by James G. Propp and David B. Wilson.
• Mostly harmless econometrics. An empiricist's companion.
• On the Identifiability Problem for Functions of Finite Markov Chains
• On the Identifiability Problem for Functions of Finite Markov Chains
• Probabilistic Functions of Finite State Markov Chains

Cited in

(4)

• Analysing plant closure effects using time-varying mixture-of-experts Markov chain clustering
• An MDP-based recommender system
• Personalized optimization with user's feedback
• Closure–state specification for markov–process models with incomplete micro data

This page was built for publication: Recovering Markov models from closed-loop data

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1737809)