skip to main content
research-article

Domain Adaptation Problem in Sketch Based Image Retrieval

Published:25 February 2023Publication History
Skip Abstract Section

Abstract

In this article, we present two algorithms that discover the discriminative structures of sketches, given pairs of sketches and photos in sketch-based image retrieval (SBIR) scenarios. Unlike the existing approaches, we aim at the few-shot and domain adaptation (DA) problems, and set up a module with canonical correlation analysis (CCA) technique in our algorithms to improve retrieval performance. For single source domain settings, our first algorithm can effectively transfer a classifier trained on a known dataset to a new one. For multisource settings, our second algorithm sophisticatedly combines multisource domain data to yield a classifier on the target domain. To the best of our knowledge, these two works are the first research in SBIR field. Experiments on the Sketchy and TU-Berlin sketch benchmark datasets demonstrate the effectiveness of our algorithms and compelling performance. Compared with the state-of-the-art methods, our algorithms do not use text based semantic information, but achieve competitive results. Furthermore, experiments also turn out that feature representation by available trained deep networks has distinct advantages and combination with traditional machine learning methods brings substantial improvements against the state-of-the-art methods on image retrievals. The complete source code of the proposed algorithms will be released on gitHub: [ GitHub- ADAM0912/Domain-Adaptation-Problem-in-Sketch-Based-Image-Retrieval: The code repository for the articleDomain Adaptation Problem in Sketch Based Image Retrieval].

REFERENCES

  1. [1] Yu Qian, Liu Feng, Song Yi-Zhe, Xiang Tao, Hospedales Timothy M., and Loy Chen-Change. 2016. Sketch me that shoe. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 799807.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Song Jifei, Yu Qian, Song Yi-Zhe, Xiang Tao, and Hospedales Timothy M.. 2017. Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In Proceedings of the IEEE International Conference on Computer Vision. 55515560.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Li Yanan, Wang Donghui, Hu Huanhang, Lin Yuetan, and Zhuang Yueting. 2017. Zero-shot recognition using dual visual-semantic mapping paths. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 32793287.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Dey Sounak, Dutta Anjan, Ghosh Suman K., Valveny Ernest, Lladós Josep, and Pal Umapada. 2018. Learning cross-modal deep embeddings for multi-object image retrieval using text and sketch. In Proceedings of the 24th International Conference on Pattern Recognition. IEEE, 916921.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Song Jifei, Yu Qian, Song Yi-Zhe, Xiang Tao, and Hospedales Timothy M.. 2017. Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In Proceedings of the IEEE International Conference on Computer Vision. 55515560.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Shen Yuming, Liu Li, Shen Fumin, and Shao Ling. 2018. Zero-shot sketch-image hashing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 35983607.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Yelamarthi Sasi Kiran, Reddy Shiva Krishna, Mishra Ashish, and Mittal Anurag. 2018. A zero-shot framework for sketch based image retrieval. In Proceedings of the European Conference on Computer Vision. 300317.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Dey Sounak, Riba Pau, Dutta Anjan, Llados Josep, and Song Yi-Zhe. 2019. Doodle to search: Practical zero-shot sketch-based image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 21792188.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceeding of the International Conference on Learning Representations.Google ScholarGoogle Scholar
  10. [10] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770778.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Wang Zhipeng, Wang Hao, Yan Jiexi, Wu Aming, and Deng Cheng. 2021. Domain-Smoothing network for zero-shot sketch-based image retrieval. In Proceedings of the 30th International Joint Conference on Artificial Intelligence. 11431149.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Hu Rui and Collomosse John. 2013. A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Computer Vision and Image Understanding 117, 7 (2013), 790806.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Saavedra Jose M.. 2014. Sketch based image retrieval using a soft computation of the histogram of edge local orientations (s-helo). In Proceedings of the IEEE International Conference on Image Processing. 29983002.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Saavedra Jose M., Manuel Barrios Juan, and Orand S.. 2015. Sketch based image retrieval using learned keyshapes (LKS). Proceedings of BMVC 1, 2 (2015), 7.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Li Yi, Hospedales Timothy M., Song Yi-Zhe, and Gong Shaogang. 2014. Fine-grained sketch-based image retrieval by matching deformable part models. In Proceedings of the British Machine Vision Conference.Google ScholarGoogle Scholar
  16. [16] Sangkloy Patsorn, Burnell Nathan, Ham Cusuh, and Hays James. 2016. The sketchy database: Learning to retrieve badly drawn bunnies. ACM Transactions on Graphics 35, 4 (2016), 112.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Zhang Hua, Liu Si, Zhang Changqing, Ren Wenqi, Wang Rui, and Cao Xiaochun. 2016. Sketchnet: Sketch classification with web images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11051113.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Song Y., Lei J., Peng B., Zheng K., Yang B., and Jia Y.. 2019. Edge-guided cross-domain learning with shape regression for sketch-based image retrieval. IEEE Access 7 (2019), 3239332399.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Liu Li, Shen Fumin, Shen Yuming, Liu Xianglong, and Shao Ling. 2017. Deep sketch hashing: Fast free-hand sketch-based image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 28622871.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Zhang Jingyi, Shen Fumin, Liu Li, Zhu Fan, Yu Mengyang, Shao Ling, Shen Heng Tao, and Gool Luc Van. 2018. Generative domain-migration hashing for sketch-to-image retrieval. In Proceedings of the European Conference on Computer Vision. 297314.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Xian Yongqin, Sharma Saurabh, Schiele Bernt, and Akata Zeynep. 2019. f-vaegan-d2: A feature generating framework for any-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1027510284.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Schonfeld Edgar, Ebrahimi Sayna, Sinha Samarth, Darrell Trevor, and Akata Zeynep. 2019. Generalized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 82478255.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Pati Yagyensh Chandra, Rezaiifar Ramin, and Krishnaprasad Perinkulam Sambamurthy. 1993. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of the 27th Asilomar Conference on Signals, Systems and Computers. 4044.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Deng Jia, Dong Wei, Socher Richard, Li Li-Jia, Li Kai, and Fei-Fei Li. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 248255.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Eitz Mathias, Hays James, and Alexa Marc. 2012. How do humans sketch objects? ACM Transactions on Graphics 31, 4 (2012), 110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Peng Xingchao, Bai Qinxun, Xia Xide, Huang Zijun, Saenko Kate, and Wang Bo. 2019. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision. 14061415.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Mancini Massimiliano, Akata Zeynep, Ricci Elisa, and Caputo Barbara. 2020. Towards recognizing unseen categories in unseen domains. In Proceedings of the 16th European Conference on Computer Vision. 466483.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Xu Ruijia, Chen Ziliang, Zuo Wangmeng, Yan Junjie, and Lin Liang. 2018. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 39643973.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Dutta Anjan and Akata Zeynep. 2020. Semantically tied paired cycle consistency for any-shot sketch-based image retrieval. International Journal of Computer Vision 128, 10 (2020), 26842703.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Pennington Jeffrey, Socher Richard, and Manning Christopher D.. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 15321543.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Quick, Draw! Retrieved from https://quickdraw.withgoogle.com/.Google ScholarGoogle Scholar
  32. [32] Lin Hong-Wei, Bao Hu-Jun, and Wang Guo-Jin. 2005. Totally positive bases and progressive iteration approximation. Computers & Mathematics with Applications 50, 3–4 (2005), 575586.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceeding of the International Conference on Learning Representations.Google ScholarGoogle Scholar
  34. [34] Koch Gregory, Zemel Richard, and Salakhutdinov Ruslan. 2015. Siamese neural networks for one-shot image recognition. In Proceedings of the ICML Deep Learning Workshop, 2.Google ScholarGoogle Scholar
  35. [35] Van der Maaten Laurens and Hinton Geoffrey. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, 11 (2008), 25792605.Google ScholarGoogle Scholar
  36. [36] Dong Chuanqi, Li Wenbin, Huo Jing, Gu Zheng, and Gao Yang. 2021. Learning task-aware local representations for few-shot learning. In Proceedings of the 29th International Conference on International Joint Conferences on Artificial Intelligence. 716722.Google ScholarGoogle Scholar
  37. [37] Wang Yu-Xiong, Girshick Ross, Hebert Martial, and Hariharan Bharath. 2018. Low-shot learning from imaginary data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 72787286.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Victor Garcia and Joan Bruna. 2017. Few-shot learning with graph neural networks. arXiv:1711.04043.Google ScholarGoogle Scholar
  39. [39] Lu Peng, Huang Gao, Lin Hangyu, Yang Wenming, Guo Guodong, and Fu Yanwei. 2021. Domain-aware SE network for sketch-based image retrieval with multiplicative euclidean margin softmax. In Proceedings of the 29th ACM International Conference on Multimedia. 34183426.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. [40] Su Wanhua, Yuan Yan, and Zhu Mu. 2015. A relationship between the average precision and the area under the ROC curve. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval. 349352.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. [41] S. J. Pan and Q. Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22, 10 (2010), 1345--1359.Google ScholarGoogle Scholar
  42. [42] Li J., Lu K., Huang Z., Zhu L., and Shen H. T.. 2019. Transfer independently together: A generalized framework for domain adaptation. IEEE Transactions on Cybernetics 49, 6 (2019), 21442155. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Li J., Chen E., Ding Z., Zhu L., Lu K., and Shen H. T.. 2021. Maximum density divergence for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 11 (2021), 39183930. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Li J., Lu K., Huang Z., Zhu L., and Shen H. T.. 2018. Heterogeneous domain adaptation through progressive alignment. IEEE Transactions on Neural Networks and Learning Systems 30, 5 (2018), 13811391.Google ScholarGoogle ScholarCross RefCross Ref
  45. [45] Long M., Cao Y., Wang J., and Jordan M.. 2015. Learning transferable features with deep adaptation networks. In Proceedings of the International Conference on Machine Learning. PMLR, 97105.Google ScholarGoogle Scholar
  46. [46] Ganin Y. and Lempitsky V.. 2015. Unsupervised domain adaptation by backpropagation. In Proceedings of the International Conference on Machine Learning. PMLR, 11801189.Google ScholarGoogle Scholar
  47. [47] Li J., Du Z., Zhu L., Ding Z., Lu K., and Shen H. T.. 2021. Divergence-agnostic unsupervised domain adaptation by adversarial attacks. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 11 (2021), 81968211. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Li J., Jing M., Su H., Lu K., Zhu L., and Shen H. T.. 2021. Faster domain adaptation networks. In IEEE Transactions on Knowledge and Data Engineering. DOI:Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Domain Adaptation Problem in Sketch Based Image Retrieval

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 3
        May 2023
        514 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3582886
        • Editor:
        • Abdulmotaleb El Saddik
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 25 February 2023
        • Online AM: 8 October 2022
        • Accepted: 26 September 2022
        • Revised: 8 July 2022
        • Received: 28 January 2022
        Published in tomm Volume 19, Issue 3

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
      • Article Metrics

        • Downloads (Last 12 months)139
        • Downloads (Last 6 weeks)19

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!