Generalized Mehler semigroups and catalytic branching processes with immigration (Q1876625)

Let \(H\) be a real separable Hilbert space, \((T_t)_{t\geq 0}\) a strongly continuous semigroup of linear operators on \(H\), and \((\mu_t)_{t\geq 0}\) a family of probability measures on \(H\). If the convolution equation \(\mu_{r+t}= (T_t \mu_r)*\mu_t\), \(r,t\geq 0\), is satisfied, then \((\mu_t)_{t\geq 0}\) is called a skew-convolution semigroup (SC-semigroup). In this case there is an associated Markov transition semigroup \((Q_t^\mu)_{t\geq 0}\) on \(H\) given by \[ Q_t^\mu f(x)= \int_H f(T_tx+y)\mu_t(dy),\quad x\in H, \] for bounded Borel measurable functions \(f\) on \(H\), and \((Q_t^\mu)_{t\geq 0}\) is called a generalized Mehler semigroup, which corresponds to a generalized Ornstein-Uhlenbeck (OU) process. On the other hand, SC-semigroups play a key role in immigration branching processes on a Lusin space \(E\). Let \(M(E)\) denote the space of finite Borel measures on \(E\), \((Q_t)_{t\geq 0}\) the transition semigroup of a measure-valued branching process (superprocess) \(X\) with state space \(M(E)\), and \((N_t)_{t\geq 0}\) a family of probability measures on \(M(E)\). If \((N_t)_{t\geq 0}\) satisfies the equation \(N_{r+t} =(N_rQ_t)* N_t\), \(r,t\geq 0\), then \((N_t)_{t\geq 0}\) is called a SC-semigroup associated with \((Q_t)_{t\geq 0}\). In this case \(Q^N_t(\nu, \cdot):= Q_t(\nu,\cdot)*N_t\), \(t\geq 0\), \(\nu\in M(E)\), is the semigroup of a Markov process \(Y\) in \(M(E)\), called the immigration process associated with \(X\). This paper builds on the connection between these two subjects. The main result is a characterization of general skew-convolution semigroups on \(H\) whose characteristic functional is not necessarily differentiable at the initial time. Other topics discussed in the paper are a connection between this subject and catalytic branching superprocesses through fluctuation limits, which provides a rich class of non-differentiable SC-semigroups, and path regularity of the corresponding generalized OU processes.