Analysis of inhomogeneous Markov reward models (Q1434438)

Let \(\{Z(t),\, t\geq0\}\) be an inhomogeneous continuous-time Markov chain (CTMC) on state space \(S=\{1,2,\dots,N\}\) with initial probability vector \(\pi\) and generator \({\mathbb Q}(t,w)\) depending both on time \(t\) and the level of accumulated reward \(w\) so that \({\mathbb Q}(t,w)\) is irreducible for \(t,w\geq0\). Whenever the CTMC stays in state \(i\) at time \(t\) and the level of accumulated reward is \(w\), the reward is accumulated at rate \(r_ i(t,w)\in[0,\infty)\). The amount of accumulated reward \(B(t)\) at time \(t\) is defined by \(dB(t)\over {dt}=r_ {Z(t)}(t,w)\), \(B(0)=0\). The authors provide forward partial differential equations describing the distributions of \(B(t)\) and of the completion time \(C(w):=\min\{t:B(t)\geq w\}\). On the basis of these descriptions, a set of ordinary differential equations describing the moments of these reward measures is obtained. This description allows the authors to give an effective numerical analysis of rather large inhomogeneous Markov reward models with rate and impulse reward accumulation. A numerical example is given to demonstrate an application of inhomogeneous Markov reward models in practice.