skip to main content
research-article

Personalized Photograph Ranking and Selection System Considering Positive and Negative User Feedback

Published:04 July 2014Publication History
Skip Abstract Section

Abstract

In this article, we propose a novel personalized ranking system for amateur photographs. The proposed framework treats the photograph assessment as a ranking problem and we introduce the idea of personalized ranking, which ranks photographs considering both their aesthetic qualities and personal preferences. Photographs are described using three types of features: photo composition, color and intensity distribution, and personalized features. An aesthetic prediction model is learned from labeled photographs by using the proposed image features and RBF-ListNet learning algorithm. The experimental results show that the proposed framework outperforms in the ranking performance: a Kendall's tau value of 0.432 is significantly higher than those obtained by the features proposed in one of the state-of-the-art approaches (0.365) and by learning based on support vector regression (0.384). To realize personalization in ranking, three approaches are proposed: the feature-based approach allows users to select photographs with specific rules, the example-based approach takes the positive feedback from users to rerank the photograph, and the list-based approach takes both positive and negative feedback from users into consideration. User studies indicate that all three approaches are effective in both aesthetic and personalized ranking.

References

  1. Subhabrata Bhattacahrya, Rahul Sukthankar, and Mubarak Shah. 2010. A framework for photo-quality assessment and enhancement based on visual aesthetics. In Proceedings of the International Conference on Multimedia (MM'10). ACM Press, New York, 271--280. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. John Canny. 1986. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6, 679--698. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: From pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning. ACM Press, New York, 129--136. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 3, 27:1--27:27. http://www.csie.ntu.edu.tw/∼cjlin/libsvm. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Rama Chellappa. 1989. Two-dimensional discrete Gaussian markov random field models for image processing. J. Institut. Electron. Telecomm. Engin. 35, 2, 114--120.Google ScholarGoogle Scholar
  6. Tianping Chen and Hong Chen. 1995. Approximation capability to functions of several variables, nonlinear functionals, and operators by radial basis function neural networks. IEEE Trans. Neural Netw. 6, 4, 904--910. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Bin Cheng, Bingbing Ni, Shuicheng Yan, and Qi Tian. 2010. Learning to photograph. In Proceedings of the International Conference on Multimedia (MM'10). ACM Press, New York, 291--300. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Daniel Cohen-Or, Olga Sorkine, Ran Gal, Tommer Leyvand, and Ying-Qing Xu. 2006. Color harmonization. ACM Trans. Graph. 25, 3, 624--630. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Mach. Learn. 20, 3, 273--297. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Gabriella Csurka, Christopher R. Dance, Lixin Fan, Jutta Willamowski, and Cedric Bray. 2004. Visual categorization with bags of keypoints. In Proceedings of the Workshop on Statistical Learning in Computer Vision (ECCV'04). 1--22.Google ScholarGoogle Scholar
  11. Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z. Wang. 2006. Studying aesthetics in photographic images using a computational approach. In Proceedings of the 9th European Conference on Computer Vision (ECCV'06). 7--13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Richard O. Duda and Peter E. Hart. 1972. Use of the hough transformation to detect lines and curves in pictures. Comm. ACM 15, 1, 11--15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Pedro F. Felzenszwalb and Daniel P. Huttenlocher. 2004. Efficient graph-based image segmentation. Int. J. Comput. Vis. 59, 2, 167--181. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Tom Grill and Mark Scanlon. 1990. Photographic Composition. Amphoto Books.Google ScholarGoogle Scholar
  15. Jonathan Harel, Christof Koch, and Pietro Perona. 2007. Graph-based visual saliency. Adv. Neural Inf. Process. Syst. 19, 545--552.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Chung-Jung Hu. 2007. A real-time skin-color-enhanced face detection algorithm. Masters thesis, National Taiwan University, Taipei, Taiwan.Google ScholarGoogle Scholar
  17. Nicolaos B. Karayiannis and Mary M. Randolph-Gips. 2003. On the construction and training of reformulated radial basis function neural networks. IEEE Trans. Neural Netw. 14, 4, 835--846. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Yan Ke, Xiaoou Tang, and Feng Jing. 2006. The design of high-level features for photo quality assessment. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Vol. 1, 419--426. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Bert P. Krages. 2005. Photography: The Art of Composition. Allworth Press.Google ScholarGoogle Scholar
  20. William H. Kruskal. 1958. Ordinal measures of association. J. Amer. Statist. Assoc. 53, 284, 814--861.Google ScholarGoogle ScholarCross RefCross Ref
  21. Michael S. Lew, Nicu Sebe, Chabane Djeraba, and Ramesh Jain. 2006. Content-based multimedia information retrieval: State of the art and challenges. ACM Trans. Multimedia Comput. Comm. Appl. 2, 1, 1--19. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Ligang Liu, Renjie Chen, Lior Wolf, and Daniel Cohen-Or. 2010. Optimizing photo composition. Comput. Graph. Forum 29, 2.Google ScholarGoogle ScholarCross RefCross Ref
  23. Yiwen Luo and Xiaoou Tang. 2008. Photo and video quality evaluation: Focusing on the subject. In Proceedings of the 10th European Conference on Computer Vision (ECCV'08). Springer, 386--399. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Bangalore S. Manjunath and Wei-Ying Ma. 1996. Texture features for browsing and retrieval of image data. IEEE Trans. Pattern Anal. Mach. Intell. 18, 8, 837--842. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Luca Marchesotti, Florent Perronnin, Diane Larlus, and Gabriela Csurka. 2011. Assessing the aesthetic quality of photographs using generic image descriptors. In Proceedings of the International Conference on Computer Vision (ICCV'11). 1784--1791. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Benjamin Martinez and Jacqueline Block. 1988. Visual Forces: An Introduction to Design. Prentice Hall.Google ScholarGoogle Scholar
  27. Masashi Nishiyama, Takahiro Okabe, Yoichi Sato, and Imari Sato. 2009. Sensation-based photo cropping. In Proceedings of the 17th ACM International Conference on Multimedia (MM'09). ACM Press, New York, 669--672. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Aude Oliva and Antonio Torralba. 2001. Modeling the shape of the scene: A holistic representation of the spatial envelope. Int. J. Comput. Vis. 42, 3, 145--175. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Florent Perronnin and Christopher Dance. 2007. Fisher kernels on visual vocabularies for image categorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'07). 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  30. Gabriele Peters. 2007. Aesthetic primitives of images for visualization. In Proceedings of the 11th International Conference on Information Visualization (IV'07). 316--325. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Vera Rivotti, Joao Proenaa, Joaquim Jorge, and Mario Sousa. 2007. Composition principles for quality depiction and aesthetics. In Proceedings of the International Symposium on Computational Aesthetics in Graphics, Visualization, and Imaging. 37--44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Yong Man Ro, Munchurl Kim, H. K. Kang, B. S. Manjunath, and Jinwoong Kim. 2001. MPEG-7 homogeneous texture descriptor. ETRI J. 23, 2, 41--51.Google ScholarGoogle ScholarCross RefCross Ref
  33. Yong Rui and Thomas Huang. 2000. Optimizing learning in image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Vol. 1. 236--243.Google ScholarGoogle ScholarCross RefCross Ref
  34. Yong Rui, Thomas S. Huang, Michael Ortega, and Sharad Mehrotra. 1998. Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Trans. Circ. Syst. Video Technol. 8, 5, 644--655. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Jose San Pedro and Stefan Siersdorfer. 2009. Ranking and classifying attractiveness of photos in folksonomies. In Proceedings of the 18th International Conference on World Wide Web (WWW'09). ACM Press, New York, 771--780. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Anthony Santella, Maneesh Agrawala, Doug DeCarlo, David Salesin, and Michael Cohen. 2006. Gaze-based interaction for semi-automatic photo cropping. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'06). ACM Press, New York, 771--780. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Gaurav Sharma, Wencheng Wu, and Edul N. Dalal. 2005. The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Res. Appl. 30, 1, 21--30.Google ScholarGoogle ScholarCross RefCross Ref
  38. Hamid R. Sheikh, Alan C. Bovik, and Gustavo de Veciana. 2005. An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans. Image Process. 14, 12, 2117--2128. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Xiaoshuai Sun, Hongxun Yao, Rongrong Ji, and Shaohui Liu. 2009. Photo assessment based on computational visual attention model. In Proceedings of the 17th ACM International Conference on Multimedia (MM'09). ACM Press, New York, 541--544. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Hanghang Tong, Mingjing Li, Hong-Jiang Zhang, Jingrui He, and Changshui Zhang. 2004. Classification of digital photos taken by photographers or home users. In Proceedings of the 5th Pacific Rim Conference on Advances in Multimedia Information Processing. Lecture Notes in Computer Science, vol. 3331, Springer, 198--205. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4, 600--612. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Zhou Wang, Hamid R. Sheikh, and Alan C. Bovik. 2002. No-reference perceptual quality assessment of jpeg compressed images. In Proceedings of the International Conference on Image Processing. Vol. 1. 477--480.Google ScholarGoogle Scholar
  43. Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008. Listwise approach to learning to rank: Theory and algorithm. In Proceedings of the 25th International Conference on Machine Learning (ICML'08). ACM Press, New York, 1192--1199. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Yang Yang Xiang and Mohan S. Kankanhalli. 2010. Automated aesthetic enhancement of videos. In Proceedings of the International Conference on Multimedia (MM'10). ACM Press, New York, 281--290. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Seungji Yang, Sang-Kyun Kim, and Yong Man Ro. 2007. Semantic home photo categorization. IEEE Trans. Circ. Syst. Video Technol. 17, 3, 324--335. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Yi-Hsuan Yang and Homer H. Chen. 2011. Ranking-based emotion recognition for music organization and retrieval. IEEE Trans. Audio Speech Lang. Process. 19, 4, 762--774. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Che-Hua Yeh, Yuan-Chen Ho, Brian A. Barsky, and Ming Ouhyoung. 2010. Personalized photograph ranking and selection system. In Proceedings of the International Conference on Multimedia (MM'10). ACM Press, New York, 211--220. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Xiang Sean Zhou and Thomas S. Huang. 2003. Relevance feedback in image retrieval: A comprehensive review. Multimedia Syst. 8, 6, 536--544.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Personalized Photograph Ranking and Selection System Considering Positive and Negative User Feedback

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!