Autoencoding slow representations for semi-supervised data-efficient regression

DOI10.1007/S10994-022-06299-1MaRDI QIDQ6134318zbMATH OpenOpenAlexFDO

Authors Oliver Struckmeier, Kshitij Tiwari, Ville Kyrki

Publication date 22 August 2023

Published in Machine Learning (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2012.06279

zbMATH Keywords

unsupervised representation learningslowness principledata-efficient downstream tasks

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05)

Abstract: The slowness principle is a concept inspired by the visual cortex of the brain. It postulates that the underlying generative factors of a quickly varying sensory signal change on a slower time scale. Unsupervised learning of intermediate representations utilizing abundant unlabeled sensory data can be leveraged to perform data-efficient supervised downstream regression. In this paper, we propose a general formulation of slowness for unsupervised representation learning adding a slowness regularization term to the estimate lower bound of the beta-VAE to encourage temporal similarity in observation and latent space. Within this framework we compare existing slowness regularization terms such as the L1 and L2 loss used in existing end-to-end methods, the SlowVAE and propose a new term based on Brownian motion. We empirically evaluate these slowness regularization terms with respect to their downstream task performance and data efficiency. We find that slow representations lead to equal or better downstream task performance and data efficiency in different experiment domains when compared to representations without slowness regularization. Finally, we discuss how the Frechet Inception Distance (FID), traditionally used to determine the generative capabilities of GANs, can serve as a measure to predict the performance of pre-trained Autoencoder model in a supervised downstream task and accelerate hyperparameter search.

Recommendations

• A New Framework to Train Autoencoders Through Non-Smooth Regularization
• Semi-supervised learning for regression based on the diffusion matrix
• Robust sparse coding via self-paced learning for data representation
• Efficient and adaptive linear regression in semi-supervised settings

Cites work

• scientific article; zbMATH DE number 1928596 (Why is no real title available?)
• scientific article; zbMATH DE number 7306898 (Why is no real title available?)
• scientific article; zbMATH DE number 3067835 (Why is no real title available?)
• Elements of causal inference. Foundations and learning algorithms
• Semi-supervised learning in causal and anticausal settings
• Slow Feature Analysis: Unsupervised Learning of Invariances

This page was built for publication: Autoencoding slow representations for semi-supervised data-efficient regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6134318)