Abstract
In this article, we present two algorithms that discover the discriminative structures of sketches, given pairs of sketches and photos in sketch-based image retrieval (SBIR) scenarios. Unlike the existing approaches, we aim at the few-shot and domain adaptation (DA) problems, and set up a module with canonical correlation analysis (CCA) technique in our algorithms to improve retrieval performance. For single source domain settings, our first algorithm can effectively transfer a classifier trained on a known dataset to a new one. For multisource settings, our second algorithm sophisticatedly combines multisource domain data to yield a classifier on the target domain. To the best of our knowledge, these two works are the first research in SBIR field. Experiments on the Sketchy and TU-Berlin sketch benchmark datasets demonstrate the effectiveness of our algorithms and compelling performance. Compared with the state-of-the-art methods, our algorithms do not use text based semantic information, but achieve competitive results. Furthermore, experiments also turn out that feature representation by available trained deep networks has distinct advantages and combination with traditional machine learning methods brings substantial improvements against the state-of-the-art methods on image retrievals. The complete source code of the proposed algorithms will be released on gitHub: [
- [1] . 2016. Sketch me that shoe. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 799–807.Google Scholar
Cross Ref
- [2] . 2017. Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In Proceedings of the IEEE International Conference on Computer Vision. 5551–5560.Google Scholar
Cross Ref
- [3] . 2017. Zero-shot recognition using dual visual-semantic mapping paths. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3279–3287.Google Scholar
Cross Ref
- [4] . 2018. Learning cross-modal deep embeddings for multi-object image retrieval using text and sketch. In Proceedings of the 24th International Conference on Pattern Recognition. IEEE, 916–921.Google Scholar
Cross Ref
- [5] . 2017. Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In Proceedings of the IEEE International Conference on Computer Vision. 5551–5560.Google Scholar
Cross Ref
- [6] . 2018. Zero-shot sketch-image hashing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3598–3607.Google Scholar
Cross Ref
- [7] . 2018. A zero-shot framework for sketch based image retrieval. In Proceedings of the European Conference on Computer Vision. 300–317.Google Scholar
Digital Library
- [8] . 2019. Doodle to search: Practical zero-shot sketch-based image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2179–2188.Google Scholar
Cross Ref
- [9] Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceeding of the International Conference on Learning Representations.Google Scholar
- [10] . 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.Google Scholar
Cross Ref
- [11] . 2021. Domain-Smoothing network for zero-shot sketch-based image retrieval. In Proceedings of the 30th International Joint Conference on Artificial Intelligence. 1143–1149.Google Scholar
Cross Ref
- [12] . 2013. A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Computer Vision and Image Understanding 117, 7 (2013), 790–806.Google Scholar
Digital Library
- [13] . 2014. Sketch based image retrieval using a soft computation of the histogram of edge local orientations (s-helo). In Proceedings of the IEEE International Conference on Image Processing. 2998–3002.Google Scholar
Cross Ref
- [14] . 2015. Sketch based image retrieval using learned keyshapes (LKS). Proceedings of BMVC 1, 2 (2015), 7.Google Scholar
Cross Ref
- [15] . 2014. Fine-grained sketch-based image retrieval by matching deformable part models. In Proceedings of the British Machine Vision Conference.Google Scholar
- [16] . 2016. The sketchy database: Learning to retrieve badly drawn bunnies. ACM Transactions on Graphics 35, 4 (2016), 1–12.Google Scholar
Digital Library
- [17] . 2016. Sketchnet: Sketch classification with web images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1105–1113.Google Scholar
Cross Ref
- [18] . 2019. Edge-guided cross-domain learning with shape regression for sketch-based image retrieval. IEEE Access 7 (2019), 32393–32399.Google Scholar
Cross Ref
- [19] . 2017. Deep sketch hashing: Fast free-hand sketch-based image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2862–2871.Google Scholar
Cross Ref
- [20] . 2018. Generative domain-migration hashing for sketch-to-image retrieval. In Proceedings of the European Conference on Computer Vision. 297–314.Google Scholar
Digital Library
- [21] . 2019. f-vaegan-d2: A feature generating framework for any-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10275–10284.Google Scholar
Cross Ref
- [22] . 2019. Generalized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8247–8255.Google Scholar
Cross Ref
- [23] . 1993. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of the 27th Asilomar Conference on Signals, Systems and Computers. 40–44.Google Scholar
Cross Ref
- [24] . 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 248–255.Google Scholar
Cross Ref
- [25] . 2012. How do humans sketch objects? ACM Transactions on Graphics 31, 4 (2012), 1–10.Google Scholar
Digital Library
- [26] . 2019. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision. 1406–1415.Google Scholar
Cross Ref
- [27] . 2020. Towards recognizing unseen categories in unseen domains. In Proceedings of the 16th European Conference on Computer Vision. 466–483.Google Scholar
Digital Library
- [28] . 2018. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3964–3973.Google Scholar
Cross Ref
- [29] . 2020. Semantically tied paired cycle consistency for any-shot sketch-based image retrieval. International Journal of Computer Vision 128, 10 (2020), 2684–2703.Google Scholar
Digital Library
- [30] . 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1532–1543.Google Scholar
Cross Ref
- [31] Quick, Draw! Retrieved from https://quickdraw.withgoogle.com/.Google Scholar
- [32] . 2005. Totally positive bases and progressive iteration approximation. Computers & Mathematics with Applications 50, 3–4 (2005), 575–586.Google Scholar
Digital Library
- [33] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceeding of the International Conference on Learning Representations.Google Scholar
- [34] . 2015. Siamese neural networks for one-shot image recognition. In Proceedings of the ICML Deep Learning Workshop, 2.Google Scholar
- [35] . 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, 11 (2008), 2579–2605.Google Scholar
- [36] . 2021. Learning task-aware local representations for few-shot learning. In Proceedings of the 29th International Conference on International Joint Conferences on Artificial Intelligence. 716–722.Google Scholar
- [37] . 2018. Low-shot learning from imaginary data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7278–7286.Google Scholar
Cross Ref
- [38] Victor Garcia and Joan Bruna. 2017. Few-shot learning with graph neural networks. arXiv:1711.04043.Google Scholar
- [39] . 2021. Domain-aware SE network for sketch-based image retrieval with multiplicative euclidean margin softmax. In Proceedings of the 29th ACM International Conference on Multimedia. 3418–3426.Google Scholar
Digital Library
- [40] . 2015. A relationship between the average precision and the area under the ROC curve. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval. 349–352.Google Scholar
Digital Library
- [41] S. J. Pan and Q. Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22, 10 (2010), 1345--1359.Google Scholar
- [42] . 2019. Transfer independently together: A generalized framework for domain adaptation. IEEE Transactions on Cybernetics 49, 6 (2019), 2144–2155.
DOI: Google ScholarCross Ref
- [43] . 2021. Maximum density divergence for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 11 (2021), 3918–3930.
DOI: Google ScholarCross Ref
- [44] . 2018. Heterogeneous domain adaptation through progressive alignment. IEEE Transactions on Neural Networks and Learning Systems 30, 5 (2018), 1381–1391.Google Scholar
Cross Ref
- [45] . 2015. Learning transferable features with deep adaptation networks. In Proceedings of the International Conference on Machine Learning. PMLR, 97–105.Google Scholar
- [46] . 2015. Unsupervised domain adaptation by backpropagation. In Proceedings of the International Conference on Machine Learning. PMLR, 1180–1189.Google Scholar
- [47] . 2021. Divergence-agnostic unsupervised domain adaptation by adversarial attacks. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 11 (2021), 8196–8211.
DOI: Google ScholarCross Ref
- [48] . 2021. Faster domain adaptation networks. In IEEE Transactions on Knowledge and Data Engineering.
DOI: Google ScholarCross Ref
Index Terms
Domain Adaptation Problem in Sketch Based Image Retrieval
Recommendations
Free-hand sketch based image and video retrieval
MM '10: Proceedings of the 18th ACM international conference on MultimediaWe present an overview of our work to date on a sketch based retrieval of image and video. We present a fast technique for extracting motion trajectories from videos and a Viterbi matching approach for retrieving video clips using free-hand sketched ...
Localized content based image retrieval
MIR '05: Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrievalClassic Content-Based Image Retrieval (CBIR) takes a single non-annotated query image, and retrieves similar images from an image repository. Such a search must rely upon a holistic (or global) view of the image. Yet often the desired content of an ...
A Zero-Shot Framework for Sketch Based Image Retrieval
Computer Vision – ECCV 2018AbstractSketch-based image retrieval (SBIR) is the task of retrieving images from a natural image database that correspond to a given hand-drawn sketch. Ideally, an SBIR model should learn to associate components in the sketch (say, feet, tail, etc.) with ...






Comments