skip to main content
research-article

Deep-based Self-refined Face-top Coordination

Published:22 July 2021Publication History
Skip Abstract Section

Abstract

Face-top coordination, which exists in most clothes-fitting scenarios, is challenging due to varieties of attributes, implicit correlations, and tradeoffs between general preferences and individual preferences. We present a Deep-Based Self-Refined (DBSR) system to simulate face-top coordination based on intuition evaluation. To this end, we first establish a well-coordinated face-top (WCFT) dataset from fashion databases and communities. Then, we use a jointly trained CNN Deep Canonical Correlation Analysis (DCCA) method to bridge the semantic face-top gap based on the WCFT dataset to deal with general preferences. Subsequently, an irrelevance-based Optimum-path Forest (OPF) method is developed to adapt the results to individual preferences iteratively. Experimental results and user study demonstrate the effectiveness of our method.

References

  1. Kaori Abe, Teppei Suzuki, Shunya Ueta, Akio Nakamura, Yutaka Satoh, and Hirokatsu Kataoka. 2017.Changing fashion cultures. arXiv preprint arXiv:1703.07920 (2017).Google ScholarGoogle Scholar
  2. Stuti Ajmani, Hiranmay Ghosh, Anupama Mallik, and Santanu Chaudhury. 2013.An ontology based personalized garment recommendation system. In Proceedings of the IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT’13), Vol. 3. IEEE, 17–20. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Shotaro Akaho. 2006.A kernel method for canonical correlation analysis. arXiv preprint cs/0609071 (2006).Google ScholarGoogle Scholar
  4. Ziad Al-Halah, Rainer Stiefelhagen, and Kristen Grauman. 2017.Fashion forward: Forecasting visual style in fashion. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). IEEE, 388–397.Google ScholarGoogle ScholarCross RefCross Ref
  5. Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013.Deep canonical correlation analysis. In Proceedings of the International Conference on Machine Learning (ICML’13). ACM, 1247–1255. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Jean-Yves Baudouin and Guy Tiberghien. 2004.Symmetry, averageness, and feature size in the facial attractiveness of women. Acta Psychol. 117, 3 (2004), 313–332.Google ScholarGoogle ScholarCross RefCross Ref
  7. Andrea Bottino and Aldo Laurentini. 2010.The analysis of facial beauty: An emerging area of research in pattern analysis. In Proceedings of the International Conference Image Analysis and Recognition. Springer, 425–435. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Wen Chen, Pipei Huang, Jiaming Xu, Xin Guo, Cheng Guo, Fei Sun, Chao Li, Andreas Pfadler, Huan Zhao, and Binqiang Zhao. 2019.POG: Personalized outfit generation for fashion recommendation at Alibaba iFashion. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2662–2670. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2019.Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 765–774. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005.Learning a similarity metric discriminatively, with application to face verification. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 1. IEEE, 539–546. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Andre Tavares Da Silva, Alexandre Xavier Falcão, and Léo Pini Magalhães. 2011.Active learning paradigms for CBIR systems based on optimum-path forest classification. Pattern Recog. 44, 12 (2011), 2971–2978. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Yael Eisenthal, Gideon Dror, and Eytan Ruppin. 2006.Facial attractiveness: Beauty and the machine. Neural Comput. 18, 1 (2006), 119–142. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Ruining He, Chunbin Lin, Jianguo Wang, and Julian McAuley. 2016.Sherlock: Sparse hierarchical embeddings for visually-aware one-class collaborative filtering. In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI’16). Morgan Kaufmann, 3740–3746. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Ruining He and Julian McAuley. 2016.VBPR: Visual Bayesian Personalized Ranking from implicit feedback. In Proceedings of the 30th AAAI Conference on Artificial Intelligence. AAAI, 403–410. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Ruining He, Charles Packer, and Julian McAuley. 2016.Learning compatibility across categories for heterogeneous item recommendation. In Proceedings of the IEEE 16th International Conference on Data Mining (ICDM’16). IEEE, 937–942.Google ScholarGoogle ScholarCross RefCross Ref
  16. Tong He and Yang Hu. 2018.FashionNet: Personalized outfit recommendation with deep neural network. arXiv preprint arXiv:1810.02443 (2018).Google ScholarGoogle Scholar
  17. Wanjia He, Weiran Wang, and Karen Livescu. 2017.Multi-view recurrent neural acoustic word embeddings. In Proceedings of the 5th International Conference on Learning Representations (ICLR’17).Google ScholarGoogle Scholar
  18. Shintami Chusnul Hidayati, Cheng-Chun Hsu, Yu-Ting Chang, Kai-Lung Hua, Jianlong Fu, and Wen-Huang Cheng. 2018.What dress fits me best? Fashion recommendation on the clothing style for personal body shape. In Proceedings of the 26th ACM International Conference on Multimedia (ACM MM’18). ACM, 438–446. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Harold Hotelling. 1992.Relations between two sets of variates. Biometrika 28, 3/4 (1936), 321–377.Google ScholarGoogle ScholarCross RefCross Ref
  20. Min Hou, Le Wu, Enhong Chen, Zhi Li, Vincent W. Zheng, and Qi Liu. 2019. Explainable fashion recommendation: A semantic attribute region guided approach. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI’19). Macao, 4681–4688. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Wei-Lin Hsiao and Kristen Grauman. 2017.Learning the latent “look”: Unsupervised discovery of a style-coherent embedding from fashion images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). IEEE, 4213–4222.Google ScholarGoogle ScholarCross RefCross Ref
  22. Wei-Lin Hsiao and Kristen Grauman. 2018.Creating capsule wardrobes from fashion images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). IEEE, 7161–7170.Google ScholarGoogle ScholarCross RefCross Ref
  23. Chia-Wei Hsieh, Chieh-Yun Chen, Chien-Lung Chou, Hong-Han Shuai, Jiaying Liu, and Wen-Huang Cheng. 2019.FashionOn: Semantic-guided image-based virtual try-on with detailed human and clothing information. In Proceedings of the 27th ACM International Conference on Multimedia (ACM MM’19). ACM, 275–283. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Yang Hu, Xi Yi, and Larry S. Davis. 2015.Collaborative fashion recommendation: A functional tensor factorization approach. In Proceedings of the 23rd ACM International Conference on Multimedia (ACM MM’15). ACM, 129–138. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Tomoharu Iwata, Shinji Wanatabe, and Hiroshi Sawada. 2011.Fashion coordinates recommender system using photographs from fashion magazines. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI’11), Vol. 3. Morgan Kaufmann, 2262–2267. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Vignesh Jagadeesh, Robinson Piramuthu, Anurag Bhardwaj, Wei Di, and Neel Sundaresan. 2014.Large scale visual recommendations from street fashion images. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1925–1934. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Wang-Cheng Kang, Eric Kim, Jure Leskovec, Charles Rosenberg, and Julian McAuley. 2019.Complete the look: Scene-based complementary product recommendation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19). IEEE, 10532–10541.Google ScholarGoogle ScholarCross RefCross Ref
  28. M. Hadi Kiapour, Kota Yamaguchi, Alexander C. Berg, and Tamara L. Berg. 2014.Hipster wars: Discovering elements of fashion styles. In Proceedings of the European Conference on Computer Vision (ECCV’14). Springer, 472–488.Google ScholarGoogle Scholar
  29. Dmitry Kornilov. 2019. Recommendation system based on a user’s physical features. US Patent App. 15/826,533.Google ScholarGoogle Scholar
  30. Sudhir Kumar and Mithun Das Gupta. 2019. GAN: Complementary fashion item recommendation. arXiv preprint arXiv:1906.05596 (2019).Google ScholarGoogle Scholar
  31. Honglin Li, Masahiro Toyoura, Kazumi Shimizu, Wei Yang, and Xiaoyang Mao. 2016.Retrieval of clothing images based on relevance feedback with focus on collar designs. Vis. Comput. 32, 10 (2016), 1351–1363. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Yuncheng Li, Liangliang Cao, Jiang Zhu, and Jiebo Luo. 2017.Mining fashion outfit composition using an end-to-end deep learning approach on set data. IEEE Trans. Multim. 19, 8 (2017), 1946–1955.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Lizi Liao, Xiangnan He, Bo Zhao, Chong-Wah Ngo, and Tat-Seng Chua. 2018.Interpretable multimodal retrieval for fashion products. In Proceedings of the 26th ACM International Conference on Multimedia (ACM MM’18). ACM, 1571–1579. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Yujie Lin, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Jun Ma, and Maarten De Rijke. 2020.Explainable outfit recommendation with joint outfit matching and comment generation. IEEE Trans. Knowl. Data Eng. 32, 8 (2020), 1502–1516.Google ScholarGoogle ScholarCross RefCross Ref
  35. Jian Liu, Pengpeng Zhao, Yanchi Liu, Victor S. Sheng, Fuzheng Zhuang, Jiajie Xu, Xiaofang Zhou, and Hui Xiong. 2019.Deep cross networks with aesthetic preference for cross-domain recommendation. arXiv preprint arXiv:1905.13030 (2019).Google ScholarGoogle Scholar
  36. Luoqi Liu, Junliang Xing, Si Liu, Hui Xu, Xi Zhou, and Shuicheng Yan. 2014.Wow! you are so beautiful today!ACM Trans. Multim. Comput. Commun. Applic. 11, 1s (2014), 1–20. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Si Liu, Jiashi Feng, Zheng Song, Tianzhu Zhang, Hanqing Lu, Changsheng Xu, and Shuicheng Yan. 2012.Hi, magic closet, tell me what to wear! In Proceedings of the 20th ACM International Conference on Multimedia (ACM MM’12). ACM, 619–628. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Zhi Lu, Yang Hu, Yunchao Jiang, Yan Chen, and Bing Zeng. 2019.Learning binary code for personalized fashion recommendation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). IEEE, 10562–10570.Google ScholarGoogle ScholarCross RefCross Ref
  39. Yihui Ma, Jia Jia, Suping Zhou, Jingtian Fu, Yejun Liu, and Zijian Tong. 2017.Towards better understanding the clothing fashion styles: A multimodal deep learning approach. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI, 38–44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Yunshan Ma, Xun Yang, Lizi Liao, Yixin Cao, and Tat-Seng Chua. 2019.Who, where, and what to wear? extracting fashion knowledge from social media. In Proceedings of the 27th ACM International Conference on Multimedia (ACM MM’19). ACM, 257–265. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Utkarsh Mall, Kevin Matzen, Bharath Hariharan, Noah Snavely, and Kavita Bala. 2019.GeoStyle: Discovering fashion trends and events. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’19). IEEE, 411–420.Google ScholarGoogle ScholarCross RefCross Ref
  42. Kevin Matzen, Kavita Bala, and Noah Snavely. 2017.StreetStyle: Exploring world-wide clothing styles from millions of photos. arXiv preprint arXiv:1706.01869 (2017).Google ScholarGoogle Scholar
  43. Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015.Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 43–52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Mahesh Chandra Mukkamala and Matthias Hein. 2017.Variants of RMSProp and Adagrad with logarithmic regret bounds. In Proceedings of the 34th International Conference on Machine Learning (ICML’17), Vol. 70. ACM, 2545–2553. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Tam V. Nguyen, Si Liu, Bingbing Ni, Jun Tan, Yong Rui, and Shuicheng Yan. 2012.Sense beauty via face, dressing, and/or voice. In Proceedings of the 20th ACM International Conference on Multimedia (ACM MM’12). ACM, 239–248. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Hosnieh Sattar, Gerard Pons-Moll, and Mario Fritz. 2019.Fashion is taking shape: Understanding clothing preference based on body shape from online sources. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV’19). IEEE, 968–977.Google ScholarGoogle ScholarCross RefCross Ref
  47. Edgar Simo-Serra, Sanja Fidler, Francesc Moreno-Noguer, and Raquel Urtasun. 2015.Neuroaesthetics in fashion: Modeling the perception of fashionability. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). IEEE, 869–877.Google ScholarGoogle ScholarCross RefCross Ref
  48. Karen Simonyan and Andrew Zisserman. 2014.Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google ScholarGoogle Scholar
  49. Guang-Lu Sun, Zhi-Qi Cheng, Xiao Wu, and Qiang Peng. 2018.Personalized clothing recommendation combining user social circle and fashion style consistency. Multim. Tools Applic. 77, 14 (2018), 17731–17754. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Moeko Takagi, Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2017.What makes a style: Experimental analysis of fashion prediction. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW’17). IEEE, 2247–2253.Google ScholarGoogle ScholarCross RefCross Ref
  51. Pongsate Tangseng and Takayuki Okatani. 2020.Toward explainable fashion recommendation. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. IEEE, 2153–2162.Google ScholarGoogle ScholarCross RefCross Ref
  52. Kristen Vaccaro, Sunaya Shivakumar, Ziqiao Ding, Karrie Karahalios, and Ranjitha Kumar. 2016.The elements of fashion style. In Proceedings of the 29th Symposium on User Interface Software and Technology. 777–785. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Dario Riccardo Valenzano, Andrea Mennucci, Giandonato Tartarelli, and Alessandro Cellerino. 2006.Shape analysis of female facial attractiveness. Vis. Res. 46, 8–9 (2006), 1282–1291.Google ScholarGoogle ScholarCross RefCross Ref
  54. Andreas Veit, Balazs Kovacs, Sean Bell, Julian McAuley, Kavita Bala, and Serge Belongie. 2015.Learning visual clothing style with heterogeneous dyadic co-occurrences. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). IEEE, 4642–4650. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. 2017.Adversarial cross-modal retrieval. In Proceedings of the 25th ACM International Conference on Multimedia (ACM MM’17). ACM, 154–162. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. 2015.On deep multi-view representation learning. In Proceedings of the 32nd International Conference on Machine Learning (ICML’15), Vol. 37. ACM, 1083–1092. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Zhonghua Wu, Guosheng Lin, Qingyi Tao, and Jianfei Cai. 2019.M2E-Try On Net: Fashion from model to everyone. In Proceedings of the 27th ACM International Conference on Multimedia (ACM MM’19). ACM, 293–301. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Duorui Xie, Lingyu Liang, Lianwen Jin, Jie Xu, and Mengru Li. 2015.SCUT-FBP: A benchmark dataset for facial beauty perception. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 1821–1826.Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Kota Yamaguchi, Tamara L. Berg, and Luis E. Ortiz. 2014.Chic or social: Visual popularity analysis in online fashion networks. In Proceedings of the 22nd ACM International Conference on Multimedia (ACM MM’14). ACM, 773–776. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Wei Yang, Masahiro Toyoura, and Xiaoyang Mao. 2012.Hairstyle suggestion using statistical learning. In Proceedings of the International Conference on Multimedia Modeling (MMM’12). Springer, 277–287. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Xun Yang, Xiangnan He, Xiang Wang, Yunshan Ma, Fuli Feng, Meng Wang, and Tat-Seng Chua. 2019.Interpretable fashion matching with rich attributes. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 775–784. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Xun Yang, Yunshan Ma, Lizi Liao, Meng Wang, and Tat-Seng Chua. 2019.TransNFCM: Translation-based neural fashion compatibility modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. AAAI, 403–410.Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Lap-Fai Yu, Sai Kit Yeung, Demetri Terzopoulos, and Tony F. Chan. 2012.DressUp!: Outfit synthesis through automatic optimization.ACM Trans. Graph. 31, 6 (2012). Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Wenhui Yu, Huidi Zhang, Xiangnan He, Xu Chen, Li Xiong, and Zheng Qin. 2018.Aesthetic-based clothing recommendation. In Proceedings of the World Wide Web Conference (WWW’18). ACM, 649–658. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Yi Yu, Suhua Tang, Francisco Raposo, and Lei Chen. 2019.Deep cross-modal correlation learning for audio and lyrics in music retrieval. ACM Trans. Multim. Comput. Commun. Applic. 15, 1 (2019), 1–16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. Liangli Zhen, Peng Hu, Xu Wang, and Dezhong Peng. 2019.Deep supervised cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). IEEE, 10386–10395.Google ScholarGoogle ScholarCross RefCross Ref
  67. Zhengzhong Zhou, Xiu Di, Wei Zhou, and Liqing Zhang. 2018.Fashion sensitive clothing recommendation using hierarchical collocation model. In Proceedings of the 26th ACM International Conference on Multimedia (ACM MM’18). ACM, 1119–1127. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Deep-based Self-refined Face-top Coordination

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!