On the benefits of representation regularization in invariance based domain generalization

DOI10.1007/S10994-021-06080-WMaRDI QIDQ2673317zbMATH OpenOpenAlexFDO

Authors Changjian Shui, Boyu Wang, Christian Gagné

Publication date 9 June 2022

Published in Machine Learning (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2105.14529

zbMATH Keywords

transfer learningrepresentation learningdomain generalization

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Problem solving in the context of artificial intelligence (heuristics, search strategies, etc.) (68T20)

Abstract: A crucial aspect in reliable machine learning is to design a deployable system in generalizing new related but unobserved environments. Domain generalization aims to alleviate such a prediction gap between the observed and unseen environments. Previous approaches commonly incorporated learning invariant representation for achieving good empirical performance. In this paper, we reveal that merely learning invariant representation is vulnerable to the unseen environment. To this end, we derive novel theoretical analysis to control the unseen test environment error in the representation learning, which highlights the importance of controlling the smoothness of representation. In practice, our analysis further inspires an efficient regularization method to improve the robustness in domain generalization. Our regularization is orthogonal to and can be straightforwardly adopted in existing domain generalization algorithms for invariant representation learning. Empirical results show that our algorithm outperforms the base versions in various dataset and invariance criteria.

Recommendations

• Robust domain adaptation
• Learning smooth representations with generalized softmax for unsupervised domain adaptation
• MapFlow: latent transition via normalizing flow for unsupervised domain adaptation
• Emergence of invariance and disentanglement in deep representations
• scientific article; zbMATH DE number 6982327

Cites work

• scientific article; zbMATH DE number 1448976 (Why is no real title available?)
• A theory of learning from different domains
• Application of the information-percolation method to reconstruction problems on graphs
• Covariate shift adaptation by importance weighted cross validation
• Divergence measures based on the Shannon entropy
• Domain-adversarial training of neural networks
• Emergence of invariance and disentanglement in deep representations
• Invariance, causality and robustness

Cited in

(1)

• Expanding domain generalization theory: error bound beyond convex combination assumption

Describes a project that uses

Uses Software

• mixup

This page was built for publication: On the benefits of representation regularization in invariance based domain generalization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2673317)