skip to main content
research-article

Improving Concept-Based Image Retrieval with Training Weights Computed from Tags

Published:02 November 2015Publication History
Skip Abstract Section

Abstract

This article presents a novel approach to training classifiers for concept detection using tags and a variant of Support Vector Machine that enables the usage of training weights per sample. Combined with an appropriate tag weighting mechanism, more relevant samples play a more important role in the calibration of the final concept-detector model. We propose a complete, automated framework that (i) calculates relevance scores for each image-concept pair based on image tags, (ii) transforms the scores into relevance probabilities and automatically annotates each image according to this probability, (iii) transforms either the relevance scores or the probabilities into appropriate training weights and finally, (iv) incorporates the training weights and the visual features into a Fuzzy Support Vector Machine classifier to build the concept-detector model. The framework can be applied to online public collections, by gathering a large pool of diverse images, and using the calculated probability to select a training set and the associated training weights. To evaluate our argument, we experiment on two large annotated datasets. Experiments highlight the retrieval effectiveness of the proposed approach. Furthermore, experiments with various levels of annotation error show that using weights derived from tags significantly increases the robustness of the resulting concept detectors.

References

  1. Avi Arampatzis and Jaap Kamps. 2009. A signal-to-noise approach to score normalization. In Proceedings of the 18th ACM Conference on Information and Knowledge Management. ACM, 797--806. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Avi Arampatzis and Stephen Robertson. 2011. Modeling score distributions in information retrieval. Information Retrieval 14, 1, 26--46. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Avi Arampatzis and André van Hameran. 2001. The score-distributional threshold optimization for adaptive binary classification tasks. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 285--293. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. In Proceedings of the Biennial GSCL Conference. 31--40.Google ScholarGoogle Scholar
  5. Francesca Bovolo, Lorenzo Bruzzone, and Lorenzo Carlin. 2010. A novel technique for subpixel image classification based on support vector machine. IEEE Transactions on Image Processing 19, 11, 2983--2999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Christopher J. C. Burges. 1998. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery 2, 2, 121--167. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. 2009. NUS-WIDE: A real-world web image database from National University of Singapore. In Proceedings of the ACM International Conference on Image and Video Retrieval. ACM, 48. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Ritendra Datta, Jia Li, and James Z. Wang. 2005. Content-based image retrieval: Approaches and trends of the new age. In Proceedings of the 7th ACM SIGMM International Workshop on Multimedia Information Retrieval. ACM, 253--262. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Christos Diou, George Stephanopoulos, Panagiotis Panagiotopoulos, Christos Papachristou, Nikos Dimitriou, and Anastasios Delopoulos. 2010. Large-scale concept detection in multimedia data using small training sets and cross-domain concept fusion. IEEE Transactions on Circuits and Systems for Video Technology 20, 12, 1808--1821. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Ralph Ewerth, Khalid Ballafkir, M. Muhling, Dominik Seiler, and Bernd Freisleben. 2012. Long-term incremental web-supervised learning of visual concepts via random savannas. IEEE Transactions on Multimedia 14, 4, 1008--1020. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Amirhossein Habibian and Cees G. M. Snoek. 2014. Recommendations for recognizing video events by concept vocabularies. Computer Vision and Image Understanding 124, 110--122.Google ScholarGoogle ScholarCross RefCross Ref
  12. Mark J. Huiskes and Michael S. Lew. 2008. The MIR flickr retrieval evaluation. In Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval. ACM, 39--43. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Mark J. Huiskes, Bart Thomee, and Michael S. Lew. 2010. New trends and ideas in visual concept detection: The MIR flickr retrieval evaluation initiative. In Proceedings of the International Conference on Multimedia Information Retrieval. ACM, 527--536. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Yu-Gang Jiang, Chong-Wah Ngo, and Jun Yang. 2007. Towards optimal bag-of-features for object categorization and semantic video retrieval. In Proceedings of the 6th ACM International Conference on Image and Video Retrieval. ACM, 494--501. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Yu-Gang Jiang, Jun Yang, Chong-Wah Ngo, and Alexander G. Hauptmann. 2010. Representations of keypoint-based semantic concept detection: A comprehensive study. IEEE Transactions on Multimedia 12, 1, 42--53. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Xue-Ming Leng and Yi-Ding Wang. 2008. Gender classification based on fuzzy SVM. In Proceedings of the 2008 International Conference on Machine Learning and Cybernetics, Vol. 3. IEEE, 1260--1264.Google ScholarGoogle ScholarCross RefCross Ref
  17. Han-Xiong Li, Jing-Lin Yang, Geng Zhang, and Bi Fan. 2013. Probabilistic support vector machines for classification of noise affected data. Information Sciences 221, 60--71. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Chun-Fu Lin and Sheng-De Wang. 2002. Fuzzy support vector machines. IEEE Transactions on Neural Networks 13, 2, 464--471. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Chun-fu Lin and Sheng-de Wang. 2004. Training algorithms for fuzzy support vector machines with noisy data. Pattern Recognition Letters 25, 14, 1647--1656. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Yi Liu and Yuan F. Zheng. 2007. Soft SVM and its application in video-object extraction. IEEE Transactions on Signal Processing 55, 7, 3272--3282. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Michael I. Mandel, Razvan Pascanu, Douglas Eck, Yoshua Bengio, Luca M. Aiello, Rossano Schifanella, and Filippo Menczer. 2011. Contextual tag inference. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP) 7, 1, 32. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. R. Manmatha, T. Rath, and Fangfang Feng. 2001. Modeling score distributions for combining the outputs of search engines. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 267--275. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Ilya Markov, Avi Arampatzis, and Fabio Crestani. 2012. Unsupervised linear score normalization revisited. In Proceedings of the 35th international ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1161--1162. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Rui Min and H. D. Cheng. 2009. Effective image retrieval using dominant color descriptor and fuzzy support vector machine. Pattern Recognition 42, 1, 147--157. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Henrik Nottelmann and Norbert Fuhr. 2003. From uncertain inference to probability of relevance for advanced IR applications. In Advances in Information Retrieval. Springer, 235--250. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Stefanie Nowak and Stefan Rüger. 2010. How reliable are annotations via crowdsourcing: A study about inter-annotator agreement for multi-label image annotation. In Proceedings of the International Conference on Multimedia Information Retrieval. ACM, 557--566. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Martin F. Porter. 1980. An algorithm for suffix stripping. Program: Electronic Library and Information Systems 14, 3, 130--137.Google ScholarGoogle ScholarCross RefCross Ref
  28. Gholamreza Rafiee, Satnam Singh Dlay, and Wai Lok Woo. 2010. A review of content-based image retrieval. In Proceedings of the 2010 7th International Symposium on Communication Systems Networks and Digital Signal Processing (CSNDSP'10). IEEE, 775--779.Google ScholarGoogle ScholarCross RefCross Ref
  29. Yong Rao, Padma Mundur, and Yelena Yesha. 2006. Fuzzy SVM ensembles for relevance feedback in image retrieval. In Image and Video Retrieval. Springer, 350--359. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Miriam Redi and Bernard Merialdo. 2012. A multimedia retrieval framework based on automatic graded relevance judgments. In Advances in Multimedia Modeling. Springer, 300--311. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Jitao Sang, Changsheng Xu, and Dongyuan Lu. 2012. Learn to personalized image search from the photo sharing websites. IEEE Transactions on Multimedia 14, 4, 963--974. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Börkur Sigurbjörnsson and Roelof Van Zwol. 2008. Flickr tag recommendation based on collective knowledge. In Proceedings of the 17th International Conference on World Wide Web. ACM, 327--336. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Cees G. M. Snoek and Marcel Worring. 2009. Concept-Based Video Retrieval. Foundations and Trends in Information Retrieval 4, 2, 215--322. Invited review paper, covering 300 references. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Abu Sayeed Md Sohail, Prabir Bhattacharya, Sudhir P. Mudur, and Srinivasan Krishnamurthy. 2011. Classification of ultrasound medical images using distance based feature selection and fuzzy-SVM. In Pattern Recognition and Image Analysis. Springer, 176--183. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Zheng Sun, Dianxu Ruan, Yun Ma, Xiaolei Hu, and Xiao-guang Zhang. 2009. Crack defects detection in radiographic weldment images using FSVM and beamlet transform. In Proceedings of the 6th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD'09), Vol. 3. IEEE, 402--406. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Jinhui Tang, Qiang Chen, Meng Wang, Shuicheng Yan, Tat-Seng Chua, and Ramesh Jain. 2013. Towards optimizing human labeling for interactive image tagging. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP) 9, 4, 29. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Theodora Tsikrika, Christos Diou, Arjen P. de Vries, and Anastasios Delopoulos. 2009. Image annotation using clickthrough data. In Proceedings of the 8th ACM International Conference on Image and Video Retrieval (CIVR'09). Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Theodora Tsikrika, Christos Diou, Arjen P. de Vries, and Anastasios Delopoulos. 2011. Reliability and effectiveness of clickthrough data for automatic image annotation. Multimedia Tools and Applications 55, 1, 27--52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Triantafillos Tsirelis and Anastasios Delopoulos. 2011. Automatic ground-truth image generation from user tags. In 12th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS 2011).Google ScholarGoogle Scholar
  40. Adrian Ulges, Markus Koch, Damian Borth, and Thomas M. Breuel. 2009. Tubetagger-youtube-based concept detection. In Proceedings of the IEEE International Conference on Data Mining Workshops (ICDMW'09). IEEE, 190--195. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. K. E. A. van de Sande, T. Gevers, and C. G. M. Snoek. 2008. A comparison of color features for visual concept classification. In Proceedings of the ACM International Conference on Image and Video Retrieval. 141--150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Meng Wang and Xian-Sheng Hua. 2011. Active learning in multimedia annotation and retrieval: A survey. ACM Transactions on Intelligent Systems and Technology (TIST) 2, 2, 10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Surong Wang, Manoranjan Dash, Liang-Tien Chia, and Min Xu. 2007. Efficient sampling of training set in large and noisy multimedia data. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP) 3, 3, 14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Yongqiao Wang, Shouyang Wang, and Kin Keung Lai. 2005. A new fuzzy support vector machine to evaluate credit risk. IEEE Transactions on Fuzzy Systems 13, 6, 820--831. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Kui Wu and Kim-Hui Yap. 2008. Soft-labeling image scheme using fuzzy support vector machine. In Computational Intelligence in Multimedia Processing: Recent Advances. Springer, 271--290.Google ScholarGoogle Scholar
  46. Guang-ming Xian. 2010. An identification method of malignant and benign liver tumors from ultrasonography based on GLCM texture features and fuzzy SVM. Expert Systems with Applications 37, 10, 6737--6741. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Linjun Yang, Bo Geng, Alan Hanjalic, and Xian-Sheng Hua. 2012. A unified context model for web image retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP) 8, 3, 28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Jun Zhang and Lei Ye. 2009. Content based image retrieval using unclean positive examples. IEEE Transactions on Image Processing 18, 10, 2370--2375. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Lei Zhang and Yong Rui. 2013. Image search from thousands to billions in 20 years. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP) 9, 1s, 36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Shiai Zhu, Chong-Wah Ngo, and Yu-Gang Jiang. 2012. Sampling and ontologically pooling web images for visual concept learning. IEEE Transactions on Multimedia 14, 4, 1068--1078. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Improving Concept-Based Image Retrieval with Training Weights Computed from Tags

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!