WAD-CMSN: Wasserstein distance-based cross-modal semantic network for zero-shot sketch-based image retrieval

DOI10.1142/S0219691322500540zbMATH Open1506.68177arXiv2202.05465OpenAlexW4308915925MaRDI QIDQ5880604FDOQ5880604

Authors: Jia Cai 

Publication date: 3 March 2023

Published in: International Journal of Wavelets, Multiresolution and Information Processing (Search for Journal in Brave)

Abstract: Zero-shot sketch-based image retrieval (ZSSBIR), as a popular studied branch of computer vision, attracts wide attention recently. Unlike sketch-based image retrieval (SBIR), the main aim of ZSSBIR is to retrieve natural images given free hand-drawn sketches that may not appear during training. Previous approaches used semantic aligned sketch-image pairs or utilized memory expensive fusion layer for projecting the visual information to a low dimensional subspace, which ignores the significant heterogeneous cross-domain discrepancy between highly abstract sketch and relevant image. This may yield poor performance in the training phase. To tackle this issue and overcome this drawback, we propose a Wasserstein distance based cross-modal semantic network (WAD-CMSN) for ZSSBIR. Specifically, it first projects the visual information of each branch (sketch, image) to a common low dimensional semantic subspace via Wasserstein distance in an adversarial training manner. Furthermore, identity matching loss is employed to select useful features, which can not only capture complete semantic knowledge, but also alleviate the over-fitting phenomenon caused by the WAD-CMSN model. Experimental results on the challenging Sketchy (Extended) and TU-Berlin (Extended) datasets indicate the effectiveness of the proposed WAD-CMSN model over several competitors.

Full work available at URL: https://arxiv.org/abs/2202.05465

Recommendations

• Semantically tied paired cycle consistency for any-shot sketch-based image retrieval
• scientific article; zbMATH DE number 6747204
• Cross knowledge-based generative zero-shot learning approach with taxonomy regularization
• Single color sketch-based image retrieval in HSV color space
• Manifold regularized cross-modal embedding for zero-shot learning

zbMATH Keywords

Wasserstein distancezero-shot learningsketch-based image retrievalidentity matching loss

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Information storage and retrieval of data (68P20) Computing methodologies for image processing (68U10)

Cites Work

• Visualizing data using t-SNE
• Bit-Scalable Deep Hashing With Regularized Similarity Learning for Image Retrieval and Person Re-Identification
• Semantically tied paired cycle consistency for any-shot sketch-based image retrieval
• Cross-Paced Representation Learning With Partial Curricula for Sketch-Based Image Retrieval
• Progressive Cross-Modal Semantic Network for Zero-Shot Sketch-Based Image Retrieval

Cited In (2)

• Single color sketch-based image retrieval in HSV color space
• Semantically tied paired cycle consistency for any-shot sketch-based image retrieval

Uses Software

• WordNet
• word2vec
• ImageNet
• Wasserstein GAN

This page was built for publication: WAD-CMSN: Wasserstein distance-based cross-modal semantic network for zero-shot sketch-based image retrieval

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5880604)