WAD-CMSN: Wasserstein distance-based cross-modal semantic network for zero-shot sketch-based image retrieval

DOI10.1142/S0219691322500540MaRDI QIDQ5880604zbMATH OpenOpenAlexFDO

Authors Jia Cai

Publication date 3 March 2023

Published in International Journal of Wavelets, Multiresolution and Information Processing (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2202.05465

zbMATH Keywords

Wasserstein distancezero-shot learningsketch-based image retrievalidentity matching loss

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Information storage and retrieval of data (68P20) Computing methodologies for image processing (68U10)

Abstract: Zero-shot sketch-based image retrieval (ZSSBIR), as a popular studied branch of computer vision, attracts wide attention recently. Unlike sketch-based image retrieval (SBIR), the main aim of ZSSBIR is to retrieve natural images given free hand-drawn sketches that may not appear during training. Previous approaches used semantic aligned sketch-image pairs or utilized memory expensive fusion layer for projecting the visual information to a low dimensional subspace, which ignores the significant heterogeneous cross-domain discrepancy between highly abstract sketch and relevant image. This may yield poor performance in the training phase. To tackle this issue and overcome this drawback, we propose a Wasserstein distance based cross-modal semantic network (WAD-CMSN) for ZSSBIR. Specifically, it first projects the visual information of each branch (sketch, image) to a common low dimensional semantic subspace via Wasserstein distance in an adversarial training manner. Furthermore, identity matching loss is employed to select useful features, which can not only capture complete semantic knowledge, but also alleviate the over-fitting phenomenon caused by the WAD-CMSN model. Experimental results on the challenging Sketchy (Extended) and TU-Berlin (Extended) datasets indicate the effectiveness of the proposed WAD-CMSN model over several competitors.

Recommendations

• Semantically tied paired cycle consistency for any-shot sketch-based image retrieval
• scientific article; zbMATH DE number 6747204
• Cross knowledge-based generative zero-shot learning approach with taxonomy regularization
• Single color sketch-based image retrieval in HSV color space
• Manifold regularized cross-modal embedding for zero-shot learning

Cites work

• Bit-Scalable Deep Hashing With Regularized Similarity Learning for Image Retrieval and Person Re-Identification
• Cross-Paced Representation Learning With Partial Curricula for Sketch-Based Image Retrieval
• Progressive Cross-Modal Semantic Network for Zero-Shot Sketch-Based Image Retrieval
• Semantically tied paired cycle consistency for any-shot sketch-based image retrieval
• Visualizing data using t-SNE

Cited in

(2)

• Single color sketch-based image retrieval in HSV color space
• Semantically tied paired cycle consistency for any-shot sketch-based image retrieval

Describes a project that uses

Uses Software

• Wasserstein GAN
• ImageNet
• word2vec
• WordNet

This page was built for publication: WAD-CMSN: Wasserstein distance-based cross-modal semantic network for zero-shot sketch-based image retrieval

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5880604)