Sequential estimation of Spearman rank correlation using Hermite series estimators (Q89438)

The authors study sequential online algorithms enabling to estimate the most popular measure of nonparametric correlation, the Spearman rank correlation coefficient, in both stationary (static) and non-stationary (dynamic) settings. The algorithms are based on bivariate Hermite series density estimators and Hermite series based distribution function estimators. The stationary setting corresponds to bivariate data streams where the Spearman rank correlation coefficient is constant. This setting also includes the one-pass estimation of the Spearman rank correlation for massive data sets. The algorithm can be applied in decentralized settings where Hermite series based estimates of the Spearman rank correlation, generated on separate portions of a larger dataset, can be combined to form an overall estimate. In the stationary iid setting it is shown that the Hermite series based Spearman rank correlation estimator converges to the grade correlation which is the constant that is unbiasedly estimated by the standard Spearman rank correlation estimator in large samples. The non-stationary (dynamic) setting corresponds to situations where the Spearman rank correlation of a bivariate data stream is expected to vary over time. In order to treat these scenarios an exponentially weighted version of the Hermite series based Spearman rank correlation estimator is introduced. The results describing variance for the exponentially weighted estimator in the iid scenario are derived, providing rule of thumb guidance for the selection of the weighting parameter. The effectiveness of the Hermite series based Spearman rank estimator is demonstrated in the stationary setting by means of a simulation study which revealed the Hermite series based estimator to be competitive with an existing approach. The effectiveness of the exponentially weighted Hermite estimator through simulation and real data studies is also demonstrated. The approach can be useful in a variety of settings, particularly those where robustness to outliers and errors is required in addition to fast online calculation of correlation which may vary over time and in financial applications involving high frequency data streams. Moreover, similar approach can also be applied to define a Hermite series based Kendall rank correlation estimator.