Extreme entropy machines: robust information theoretic classification

DOI10.1007/S10044-015-0497-8MaRDI QIDQ2337517zbMATH OpenOpenAlexDBLPWikidataFDO

Authors Wojciech M. Czarnecki, Jacek Tabor

Publication date 20 November 2019

Published in PAA. Pattern Analysis and Applications (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/1501.05279

zbMATH Keywords

classificationentropyextreme learning machinesrandom projectionsrapid learning

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Measures of information, entropy (94A17)

Abstract: Most of the existing classification methods are aimed at minimization of empirical risk (through some simple point-based error measured with loss function) with added regularization. We propose to approach this problem in a more information theoretic way by investigating applicability of entropy measures as a classification model objective function. We focus on quadratic Renyi's entropy and connected Cauchy-Schwarz Divergence which leads to the construction of Extreme Entropy Machines (EEM). The main contribution of this paper is proposing a model based on the information theoretic concepts which on the one hand shows new, entropic perspective on known linear classifiers and on the other leads to a construction of very robust method competetitive with the state of the art non-information theoretic ones (including Support Vector Machines and Extreme Learning Machines). Evaluation on numerous problems spanning from small, simple ones from UCI repository to the large (hundreads of thousands of samples) extremely unbalanced (up to 100:1 classes' ratios) datasets shows wide applicability of the EEM in real life problems and that it scales well.

Recommendations

• Entropy-based estimation in classification problems
• Error Entropy in Classification Problems: A Univariate Data Analysis
• Minimum error entropy classification.
• Learning from examples with information theoretic criteria
• Relation of information entropy and support vector

Cites work

• A well-conditioned estimator for large-dimensional covariance matrices
• Benchmarking least squares support vector machine classifiers
• Chaos control using least-squares support vector machines
• Cross-entropy clustering
• scientific article; zbMATH DE number 107482 (Why is no real title available?)
• scientific article; zbMATH DE number 3567782 (Why is no real title available?)
• scientific article; zbMATH DE number 4001209 (Why is no real title available?)
• scientific article; zbMATH DE number 3023295 (Why is no real title available?)
• Learning from examples with information theoretic criteria
• Learning pattern classification-a survey
• On Estimation of a Probability Density Function and Mode
• On the Nyström method for approximating a gram matrix for improved kernel-based learning
• Regularization algorithms for learning that are equivalent to multilayer networks
• Scikit-learn: machine learning in Python
• Statistical analysis of finite mixture distributions
• Support-vector networks
• The Cauchy--Schwarz divergence and Parzen windowing: Connections to graph theory and Mercer kernels
• Weighted least squares support vector machines: robustness and sparse approximation

Cited in

(6)

• Soft randomized machine learning
• Entropy-based estimation in classification problems
• Information-extreme method for classification of observations with categorical attributes
• eSPA+: Scalable Entropy-Optimal Machine Learning Classification for Small Data Problems
• Error Entropy in Classification Problems: A Univariate Data Analysis
• Relation of information entropy and support vector

Describes a project that uses

Uses Software

• NumPy
• Scikit
• LIBSVM
• UCI
• SciPy

This page was built for publication: Extreme entropy machines: robust information theoretic classification

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2337517)