skip to main content
research-article
Free Access

CBIR Using Features Derived by Deep Learning

Published:03 September 2021Publication History
Skip Abstract Section

Abstract

In a Content-based Image Retrieval (CBIR) System, the task is to retrieve similar images from a large database given a query image. The usual procedure is to extract some useful features from the query image and retrieve images that have a similar set of features. For this purpose, a suitable similarity measure is chosen, and images with high similarity scores are retrieved. Naturally, the choice of these features play a very important role in the success of this system, and high-level features are required to reduce the “semantic gap.”

In this article, we propose to use features derived from pre-trained network models from a deep-learning convolution network trained for a large image classification problem. This approach appears to produce vastly superior results for a variety of databases, and it outperforms many contemporary CBIR systems. We analyse the retrieval time of the method and also propose a pre-clustering of the database based on the above-mentioned features, which yields comparable results in a much shorter time in most of the cases.

References

  1. Stanford University. [n.d.]. CS231n: Convolutional Neural Networks for Visual Recognition. Retrieved from http://cs231n.stanford.edu/.Google ScholarGoogle Scholar
  2. Khawaja Ahmed, Shahida, and Muhammad Iqbal. 2018. Content-based image retrieval using image features information fusion. Info. Fusion 51 (Nov. 2018), 76–99. https://doi.org/10.1016/j.inffus.2018.11.004Google ScholarGoogle Scholar
  3. K. T. Ahmed, S. A. H. Naqvi, A. Rehman, and T. Saba. 2019. Convolution, approximation and spatial information based object and color signatures for content based image retrieval. In Proceedings of the International Conference on Computer and Information Sciences (ICCIS’19). 1–6. https://doi.org/10.1109/ICCISci.2019.8716437Google ScholarGoogle Scholar
  4. S. Aksoy and R. M. Haralick. 2000. Probabilistic vs. geometric similarity measures for image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’00), Vol. 2. 357–362. https://doi.org/10.1109/CVPR.2000.854847Google ScholarGoogle Scholar
  5. Mutasem K. Alsmadi. 2020. Content-based image retrieval using color, shape and texture descriptors and features. Arab. J. Sci. Eng. 45, 4 (2020), 3317–3330. https://doi.org/10.1007/s13369-020-04384-yGoogle ScholarGoogle Scholar
  6. Rehan Ashraf, Mudassar Ahmed, Sohail Jabbar, Shehzad Khalid, Awais Ahmad, Sadia Din, and Gwangil Jeon. 2018. Content based image retrieval by using color descriptor and discrete wavelet transform. J. Med. Syst. 17, 6 (Mar. 2018), 3552–3580. https://doi.org/10.1007/s10916-017-0880-7Google ScholarGoogle Scholar
  7. Rehan Ashraf, Khalid Bashir, Aun Irtaza, and Muhammad Mahmood. 2015. Content based image retrieval using embedded neural networks with bandletized regions. Entropy 17 (June 2015), 3552–3580. https://doi.org/10.3390/e17063552Google ScholarGoogle Scholar
  8. Mohamed Uvaze Ahamed Ayoobkhan, C. Eswaran, and Kannan Ramakrishnan. 2017. CBIR system based on prediction errors. J. Info. Sci. Eng. 33 (Mar. 2017), 347–365. https://doi.org/10.1688/JISE.2017.33.2.5Google ScholarGoogle Scholar
  9. Smarajit Bose, Amita Pal, Disha Chakrabarti, and Taranga Mukherjee. 2017. Improved content-based image retrieval via discriminant analysis. Int. J. Mach. Learn. Comput. 7 (June 2017), 44–48. https://doi.org/10.18178/ijmlc.2017.7.3.618Google ScholarGoogle Scholar
  10. Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. 2016. An analysis of deep neural network models for practical applications. Retrieved from https://arXiv:1605.07678.Google ScholarGoogle Scholar
  11. François Chollet. 2016. Xception: Deep learning with depthwise separable convolutions. Retrieved from https://arXiv:1610.02357.Google ScholarGoogle Scholar
  12. Francois Chollet. 2017. Deep Learning with Python (1st ed.). Manning Publications, Greenwich, CT. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 248–255. https://doi.org/10.1109/CVPR.2009.5206848Google ScholarGoogle ScholarCross RefCross Ref
  14. Meenakshi Garg and Gaurav Dhiman. 2021. A novel content-based image retrieval approach for classification using GLCM features and texture fused LBP variants. Neural Comput. Appl. 33 (2021), 1311–1328. https://doi.org/10.1007/s00521-020-05017-zGoogle ScholarGoogle ScholarDigital LibraryDigital Library
  15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. Retrieved from https://arXiv:1512.03385.Google ScholarGoogle Scholar
  16. Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2016. Densely connected convolutional networks. Retrieved from https://arXiv:1608.06993.Google ScholarGoogle Scholar
  17. Jing Huang. 1998. Color-spatial Image Indexing and Applications. Ph.D. Dissertation. Ithaca, NY. Advisor(s) Zabih, Ramin. Google ScholarGoogle Scholar
  18. D. Mansoor Hussain and D. Surendran. 2020. The efficient fast-response content-based image retrieval using spark and MapReduce model framework. J. Ambient Intell. Human. Comput. 12, 3 (2020), 4049–4056. https://doi.org/10.1007/s12652-020-01775-9Google ScholarGoogle Scholar
  19. Safia Jabeen, Zahid Mehmood, Toqeer Mahmood, Tanzila Saba, Amjad Rehman, and Muhammad Mahmood. 2018. An effective content-based image retrieval technique for image visuals representation based on the bag-of-visual-words model. PLoS ONE 13, 4 (Mar. 2018). https://doi.org/10.1371/journal.pone.0194526Google ScholarGoogle Scholar
  20. Aamir Khan and Anand Jalal. 2021. A visual saliency-based approach for content-based image retrieval. International J. Cogn. Info. Natural Intell. 15 (Jan. 2021), 1–15. https://doi.org/10.4018/IJCINI.2021010101Google ScholarGoogle Scholar
  21. Suman Khokhar and Satya Verma. 2017. Content based image retrieval with multi-feature classification by back-propagation neural network. Int. J. Comput. Appl. Technol. Res. 6 (July 2017), 278–284. https://doi.org/10.7753/IJCATR0607.1002Google ScholarGoogle Scholar
  22. Harald Kosch. 2003. Distributed Multimedia Database Technologies Supported by MPEG-7 and MPEG-21. CRC Press. Google ScholarGoogle Scholar
  23. Li Fei-Fei, R. Fergus, and P. Perona. 2004. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop. 178–178. https://doi.org/10.1109/CVPR.2004.383Google ScholarGoogle Scholar
  24. Mr. Yogen Mahesh Lohite and Prof. Sushant J. Pawar. 2017. A novel method for content based image retrieval using local features and SVM classifier. Int. Res. J. Eng. Technol. 4, 7 (2017).Google ScholarGoogle Scholar
  25. B. S. Manjunath, J. Ohm, V. V. Vasudevan, and A. Yamada. 2001. Color and texture descriptors. IEEE Trans. Circ. Syst. Video Technol. 11, 6 (June 2001), 703–715. https://doi.org/10.1109/76.927424Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Pedro Marcelino. [n.d.]. Transfer learning from pre-trained models. Retrieved from https://towardsdatascience.com/transfer-learning-from-pre-trained-models- f2393f124751.Google ScholarGoogle Scholar
  27. Zahid Mehmood, Toqeer Mahmood, and Muhammad Arshad Javid. 2018. Content-based image retrieval and semantic automatic image annotation based on the weighted average of triangular histograms using support vector machine. Appl. Intell. 48, 1 (Jan. 2018), 166–181. https://doi.org/10.1007/s10489-017-0957-5Google ScholarGoogle Scholar
  28. Wayne Niblack, Ronald Barber, William Equitz, Myron Flickner, Eduardo Glasman, Dragutin Petkovic, Peter Yanker, Christos Faloutsos, and Gabriel Taubin. 1993. The QBIC Project: Querying images by content, using color, texture, and shape. InProceedings of the SPIE Conference on Storage and Retrieval for Image and Video Databases. 173–187.Google ScholarGoogle Scholar
  29. A. Obulesu, Vakulabharanam Vijaya Kumar, and Sumalatha Lingamgunta. 2018. Content based image retrieval using multi motif co-occurrence matrix. Int. J. Image Graph. Signal Process. 10 (Apr. 2018), 59–72. https://doi.org/10.5815/ijigsp.2018.04.07Google ScholarGoogle Scholar
  30. T. Ojala, Mika Rautiainen, Esa Matinmikko, and M. Aittola. 2001. Semantic image retrieval with HSV correlograms. (Jan. 2001).Google ScholarGoogle Scholar
  31. Michael Ortega-Binderberger. [n.d.]. Corel Image Features Data Set. Retrieved from https://archive.ics.uci.edu/ml/datasets/corel+image+features.Google ScholarGoogle Scholar
  32. Jing Peng, Bir Bhanu, and Shan Qing. 1999. Probabilistic feature relevance learning for content-based image retrieval. Comput. Vision Image Understand. 75 (1999), 150–164.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Soumya Rana, Maitreyee Dey, and Siarry Patrick. 2018. Boosting content based image retrieval performance through integration of parametric & nonparametric approaches. J. Visual Commun. Image Represent. 58 (Nov. 2018), 205–219. https://doi.org/10.1016/j.jvcir.2018.11.015Google ScholarGoogle Scholar
  34. A. Rashno and S. Sadri. 2017. Content-based image retrieval with color and texture features in neutrosophic domain. In Proceedings of the 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA’17). 50–55. https://doi.org/10.1109/PRIA.2017.7983063Google ScholarGoogle Scholar
  35. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. MobileNetV2: Inverted residuals and linear bottlenecks. Retrieved from https://arXiv:1801.04381.Google ScholarGoogle Scholar
  36. Amna Sarwar, Zahid Mehmood, Tanzila Saba, Khurram Ashfaq Qazi, Ahmed Adnan, and Habibullah Jamal. 2019. A novel method for content-based image retrieval to improve the effectiveness of the bag-of-words model using a support vector machine. J. Info. Sci. 45, 1 (2019), 117–135. https://doi.org/10.1177/0165551518782825arXiv:https://doi.org/10.1177/0165551518782825Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. G. V. Satya Kumar and P. G. Krishna Mohan. 2018. Local mean differential excitation pattern for content based image retrieval. SN Appl. Sci. 1, 1 (Nov. 2018), 46. https://doi.org/10.1007/s42452-018-0047-2Google ScholarGoogle Scholar
  38. Seong-O Shim and Tae-Sun Choi. 2003. Image indexing by modified color co-occurrence matrix. In Proceedings of the International Conference on Image Processing, Vol. 3. III–493. https://doi.org/10.1109/ICIP.2003.1247289Google ScholarGoogle Scholar
  39. Uzma Sharif, Zahid Mehmood, Toqeer Mahmood, Dr. Javid, Amjad Rehman, and Tanzila Saba. 2018. Scene analysis and search using local features and support vector machine for effective content-based image retrieval. Artific. Intell. Rev. 52, 2 (June 2018), 901–925. https://doi.org/10.1007/s10462-018-9636-0Google ScholarGoogle Scholar
  40. Jonathon Shlens. 2014. A tutorial on principal component analysis. Retrieved from https://arXiv:1404.1100.Google ScholarGoogle Scholar
  41. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. Retrieved from https://arXiv:1409.1556.Google ScholarGoogle Scholar
  42. Sachendra Singh and Shalini Batra. 2020. An efficient bi-layer content based image retrieval system. Multimedia Tools Appl. 79, 25 (July 2020), 17731–17759. https://doi.org/10.1007/s11042-019-08401-7Google ScholarGoogle Scholar
  43. Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi. 2016. Inception-v4, inception-resnet and the impact of residual connections on learning. Retrieved from https://arXiv:1602.07261.Google ScholarGoogle Scholar
  44. Jian Xu, Cunzhao Shi, Chengzuo Qi, Chunheng Wang, and Baihua Xiao. 2017. Unsupervised part-based weighting aggregation of deep convolutional features for image retrieval. Retrieved from https://arXiv:1705.01247.Google ScholarGoogle Scholar
  45. G. Yosr, N. Baklouti, H. Hagras, M. Ben ayed, and A. M. Alimi. 2021. Interval Type-2 beta fuzzy near sets approach to content-based image retrieval. IEEE Trans. Fuzzy Syst. (2021), 1–1. https://doi.org/10.1109/TFUZZ.2021.3049900Google ScholarGoogle Scholar
  46. Muhammad Yousuf, Zahid Mehmood, Hafiz Adnan Habib, Toqeer Mahmood, Tanzila Saba, Amjad Rehman, and Muhammad Rashid. 2018. A novel technique based on visual words fusion analysis of sparse features for effective content-based image retrieval. Math. Problems Eng. 2018 (Mar. 2018), 13. https://doi.org/10.1155/2018/2134395Google ScholarGoogle Scholar
  47. M. D. Zeiler, G. W. Taylor, and R. Fergus. 2011. Adaptive deconvolutional networks for mid- and high-level feature learning. In Proceedings of the International Conference on Computer Vision. 2018–2025. https://doi.org/10.1109/ICCV.2011.6126474Google ScholarGoogle Scholar
  48. Wengang Zhou, Houqiang Li, and Qi Tian. 2017. Recent advance in content-based image retrieval: A literature survey. Retrieved from https://arXiv:1706.06064.Google ScholarGoogle Scholar
  49. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2017. Learning transferable architectures for scalable image recognition. Retrieved from https://arXiv:1707.07012.Google ScholarGoogle Scholar

Index Terms

  1. CBIR Using Features Derived by Deep Learning

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!