Sequential quantiles via Hermite series density estimation

From MaRDI portal



Abstract: Sequential quantile estimation refers to incorporating observations into quantile estimates in an incremental fashion thus furnishing an online estimate of one or more quantiles at any given point in time. Sequential quantile estimation is also known as online quantile estimation. This area is relevant to the analysis of data streams and to the one-pass analysis of massive data sets. Applications include network traffic and latency analysis, real time fraud detection and high frequency trading. We introduce new techniques for online quantile estimation based on Hermite series estimators in the settings of static quantile estimation and dynamic quantile estimation. In the static quantile estimation setting we apply the existing Gauss-Hermite expansion in a novel manner. In particular, we exploit the fact that Gauss-Hermite coefficients can be updated in a sequential manner. To treat dynamic quantile estimation we introduce a novel expansion with an exponentially weighted estimator for the Gauss-Hermite coefficients which we term the Exponentially Weighted Gauss-Hermite (EWGH) expansion. These algorithms go beyond existing sequential quantile estimation algorithms in that they allow arbitrary quantiles (as opposed to pre-specified quantiles) to be estimated at any point in time. In doing so we provide a solution to online distribution function and online quantile function estimation on data streams. In particular we derive an analytical expression for the CDF and prove consistency results for the CDF under certain conditions. In addition we analyse the associated quantile estimator. Simulation studies and tests on real data reveal the Gauss-Hermite based algorithms to be competitive with a leading existing algorithm.


Let \(x_1,x_2,\dots,x_n\) be iid random variables with density function \(f(x)\) and distribution function \(F(x)\). The empirical distribution function is defined as \(\widehat{F}(x)=\frac{1}{n}\sum_{i=1}^{n}I\{x_i \leq x\}\), where \(I\) is the usual indicator function. Clearly, \(\widehat{F}(x)\) can also be written in terms of order statistics \(x_{(i)}\), \(i\geq 1\). The kernel distribution function estimator is defined as \(\widehat{F_1}(x)=\frac{1}{n}\sum_{i=1}^{n}\int_{-\infty}^{\infty} \frac{1}{h_n} K\Big(\frac{x_i-x}{h_n}\Big) dx\), where the kernel function \(K\) is a non-negative function integrating to one with mean zero and bandwidth \(h_n >0\) as a smoothing parameter. The quantile function is defined as: \(q(p)=\text{inf}\{x:F(x)\geq p \}\). The \(p\)-th quantile can be estimated from the order statistics. For example, if \(p\in \Big(\frac{i-1}{n},\frac{i}{n}\Big]\), then \(\hat{q}(p)=x_{(i)}\). For certain values of \(p\), \(\hat{q}(p)\) is interpolated between \(x_{(i)}\) and \(x_{(i+1)}\). To improve the efficiency of such estimators, certain weighted average of several order statistics with suitable weights (known as L-estimators) is used as an alternative estimator. Perhaps the most popular class of such estimators uses the kernel as the weight function. Thus the kernel quantile estimator is defined as: \(\hat{q}(p)=\sum_{i=1}^{n} \int_{\frac{i-1}{n}}^{\frac{i}{n}} \frac{1}{h_n}K\Big(\frac{s-p}{h_n}\Big)\, x_{(i)}\, ds\). After providing some background, the authors use Hermite polynomials to develop the Gauss-Hermite expansion of the density function \(f(x)\), which leads to a Gauss-Hermite based estimator of the cumulative distribution function (CDF). They apply these methods to a obtain sequential quantile estimation in the settings of static and dynamic quantile estimation. The treatment of quantile estimation is made concrete by considering cases of independent and identically distributed and non-identically distributed data streams. Observations are continuous random variables observed sequentially one at a time. The quality of the Gauss-Hermite based CDF estimator and the quantile estimation is investigated. The proposed techniques are compared with existing leading algorithms for both simulated data and real data, and suitable conclusions are made.











This page was built for publication: Sequential quantiles via Hermite series density estimation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q89436)