Anomaly Detection in High Dimensional Data

DOI10.48550/ARXIV.1908.04000arXiv1908.04000MaRDI QIDQ74767FDOQ74767

Rob J. Hyndman, Kate Smith-Miles, Priyanga Dilini Talagala

Publication date: 12 August 2019

Abstract: The HDoutliers algorithm is a powerful unsupervised algorithm for detecting anomalies in high-dimensional data, with a strong theoretical foundation. However, it suffers from some limitations that significantly hinder its performance level, under certain circumstances. In this article, we propose an algorithm that addresses these limitations. We define an anomaly as an observation that deviates markedly from the majority with a large distance gap. An approach based on extreme value theory is used for the anomalous threshold calculation. Using various synthetic and real datasets, we demonstrate the wide applicability and usefulness of our algorithm, which we call the stray algorithm. We also demonstrate how this algorithm can assist in detecting anomalies present in other data structures using feature engineering. We show the situations where the stray algorithm outperforms the HDoutliers algorithm both in accuracy and computational time. This framework is implemented in the open source R package stray.

Cited In (2)

• stray
• dsos

Summary: This article introduces a novel algorithm for detecting anomalies in high-dimensional data, known as the stray algorithm. Developed to overcome limitations in the performance of existing algorithms like HDoutliers, this method identifies anomalies based on extreme value theory by calculating thresholds for large distance gaps between observations. Extensive testing with both synthetic and real datasets has demonstrated that the stray algorithm not only outperforms its predecessor but also excels in terms of accuracy and computational efficiency. The stray algorithm is available as an open-source R package, further highlighting its versatility and potential impact on anomaly detection methods.

Summary_simple: This paper introduces a novel algorithm for spotting unusual data points in big piles of information—the stray algorithm. It was created because another tool (HDoutliers) has some flaws that make it less useful under specific conditions. The new algorithm looks at how much something stands out from the rest, using fancy math called extreme value theory to find special cases. Tests on pretend and real data showed that the stray does a better job of finding weird stuff than its predecessor, being more accurate and quicker too. You can get it for free as an R package that anyone can use!

This page was built for publication: Anomaly Detection in High Dimensional Data

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q74767)