Robust change detection for large-scale data streams
From MaRDI portal
Abstract: Robust change-point detection for large-scale data streams has many real-world applications in industrial quality control, signal detection, biosurveillance. Unfortunately, it is highly non-trivial to develop efficient schemes due to three challenges: (1) the unknown sparse subset of affected data streams, (2) the unexpected outliers, and (3) computational scalability for real-time monitoring and detection. In this article, we develop a family of efficient real-time robust detection schemes for monitoring large-scale independent data streams. For each data stream, we propose to construct a new local robust detection statistic called -CUSUM statistic that can reduce the effect of outliers by using the Box-Cox transformation of the likelihood function. Then the global scheme will raise an alarm based upon the sum of the shrinkage transformation of these local -CUSUM statistics so as to filter out unaffected data streams. In addition, we propose a new concept called {em false alarm breakdown point} to measure the robustness of online monitoring schemes and propose a worst-case detection efficiency score to measure the detection efficiency when the data contain outliers. We then characterize the breakdown point and the efficiency score of our proposed schemes. Asymptotic analysis and numerical simulations are conducted to illustrate the robustness and efficiency of our proposed schemes.
Recommendations
- Efficient scalable schemes for monitoring a large number of data streams
- Scalable sum-shrinkage schemes for distributed monitoring large-scale data streams
- Changepoint Detection in the Presence of Outliers
- Sequential multi-sensor change-point detection
- Multiple changepoint detection in categorical data streams
Cites work
- A robust surveillance scheme for stochastically ordered alternatives
- A Robust Version of the Probability Ratio Test
- An efficient sequential nonparametric scheme for detecting a change of distribution
- An online kernel change detection algorithm
- Asymptotic statistical properties of communication-efficient quickest detection schemes in sensor networks
- Asymptotically Optimal Quickest Change Detection in Distributed Sensor Systems
- Efficient Bounded-Influence Regression Estimation
- Efficient scalable schemes for monitoring a large number of data streams
- High breakdown-point and high efficiency robust estimates for regression
- Large-Scale Multi-Stream Quickest Change Detection via Shrinkage Post-Change Estimation
- Least Median of Squares Regression
- Maximum L\(q\)-likelihood estimation
- Minimax Robust Quickest Change Detection
- Optimal sequential detection in multi-stream data
- Optimal stopping times for detecting changes in distributions
- Procedures for Reacting to a Change in Distribution
- Robust and efficient estimation by minimising a density power divergence
- Robust Bounded-Influence Tests in General Parametric Models
- Robust Estimation of a Location Parameter
- Robust hypothesis testing via L_q-likelihood
- Robust Inference for Generalized Linear Models
- Robust Statistics
- Scalable sum-shrinkage schemes for distributed monitoring large-scale data streams
- Second-Order Asymptotic Optimality in Multisensor Sequential Change Detection
- Sequential analysis. Tests and confidence intervals
- Sequential multi-sensor change-point detection
Cited in
(9)- Efficient scalable schemes for monitoring a large number of data streams
- Using labeled data to evaluate change detectors in a multivariate streaming environment
- Forman-Ricci flow for change detection in large dynamic data sets
- Large-Scale Multi-Stream Quickest Change Detection via Shrinkage Post-Change Estimation
- Scalable sum-shrinkage schemes for distributed monitoring large-scale data streams
- Large-Scale Datastreams Surveillance via Pattern-Oriented-Sampling
- A precise statistical approach for concept change detection in unlabeled data streams
- A two-stage monitoring scheme for multiple high-dimensional data streams
- Data line change detection with application to Mud logging data processing
This page was built for publication: Robust change detection for large-scale data streams
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5085241)
