skip to main content
research-article
Open Access

Market2Dish: Health-aware Food Recommendation

Published:16 April 2021Publication History
Skip Abstract Section

Abstract

With the rising incidence of some diseases, such as obesity and diabetes, the healthy diet is arousing increasing attention. However, most existing food-related research efforts focus on recipe retrieval, user-preference-based food recommendation, cooking assistance, or the nutrition and calorie estimation of dishes, ignoring the personalized health-aware food recommendation. Therefore, in this work, we present a personalized health-aware food recommendation scheme, namely, Market2Dish, mapping the ingredients displayed in the market to the healthy dishes eaten at home. The proposed scheme comprises three components, namely, recipe retrieval, user health profiling, and health-aware food recommendation. In particular, recipe retrieval aims to acquire the ingredients available to the users and then retrieve recipe candidates from a large-scale recipe dataset. User health profiling is to characterize the health conditions of users by capturing the textual health-related information crawled from social networks. Specifically, to solve the issue that the health-related information is extremely sparse, we incorporate a word-class interaction mechanism into the proposed deep model to learn the fine-grained correlations between the textual tweets and pre-defined health concepts. For the health-aware food recommendation, we present a novel category-aware hierarchical memory network–based recommender to learn the health-aware user-recipe interactions for better food recommendation. Moreover, extensive experiments demonstrate the effectiveness of the health-aware food recommendation scheme.

References

  1. Sofiane Abbar, Yelena Mejova, and Ingmar Weber. 2015. You tweet what you eat: Studying food consumption through Twitter. In Proceedings of the Conference on Human Factors in Computing Systems. ACM, 3197–3206.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Yongsheng An, Yu Cao, Jingjing Chen, Chong-Wah Ngo, Jia Jia, Huanbo Luan, and Tat-Seng Chua. 2017. PIC2DISH: A customized cooking assistant system. In Proceedings of the International Conference on Multimedia. ACM, 1269–1273.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Oren Barkan and Noam Koenigstein. 2016. Item2Vec: Neural item embedding for collaborative filtering. In Proceedings of the International Workshop on Machine Learning for Signal Processing. IEEE, 1–6.Google ScholarGoogle ScholarCross RefCross Ref
  4. Immanuel Bayer, Xiangnan He, Bhargav Kanagal, and Steffen Rendle. 2017. A generic coordinate descent framework for learning from implicit feedback. In Proceedings of the International Conference on World Wide Web. ACM, 1341–1350.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Alexandre Blansché, Julien Cojan, Valmi Dufour-Lussier, Jean Lieber, Pascal Molli, Emmanuel Nauer, Hala Skaf-Molli, and Yannick Toussaint. 2010. Taaable 3: Adaptation of ingredient quantities and of textual preparations. In Proceedings of the International Conference on Case-based Reasoning Workshop. 189–198.Google ScholarGoogle Scholar
  6. Micael Carvalho, Rémi Cadène, David Picard, Laure Soulier, Nicolas Thome, and Matthieu Cord. 2018. Cross-modal retrieval in the cooking context: Learning semantic text-image embeddings. In Proceedings of the International SIGIR Conference on Research & Development in Information Retrieval. ACM, 35–44.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Jingjing Chen, Lei Pang, and Chong-Wah Ngo. 2017. Cross-modal recipe retrieval: How to cook this dish? In Proceedings of the International Conference on Multimedia Modeling. Springer, 588–600.Google ScholarGoogle ScholarCross RefCross Ref
  8. Jing-jing Chen, Chong-Wah Ngo, and Tat-Seng Chua. 2017. Cross-modal recipe retrieval with rich food attributes. In Proceedings of the International Conference on Multimedia. ACM, 1771–1779.Google ScholarGoogle Scholar
  9. Jing-Jing Chen, Chong-Wah Ngo, Fu-Li Feng, and Tat-Seng Chua. 2018. Deep understanding of cooking procedure for cross-modal recipe retrieval. In Proceedings of the International Conference on Multimedia. ACM, 1020–1028.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Zhiyong Cheng and Jialie Shen. 2014. Just-for-Me: An adaptive personalization system for location-aware social music recommendation. In Proceedings of the International Conference on Multimedia Retrieval. ACM, 185–192.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Zhiyong Cheng and Jialie Shen. 2016. On effective location-aware music recommendation. ACM Trans. Inf. Syst. 34, 2 (2016).Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12, Aug. (2011), 2493–2537.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Du Cunxiao, Chen Zhaozheng, Feng Fuli, Zhu Lei, Gan Tian, and Nie Liqiang. 2019. Explicit interaction model towards text classification. In Proceedings of the International AAAI Conference on Artificial Intelligence. AAAI Press, 6359–6366.Google ScholarGoogle Scholar
  14. Munmun De Choudhury, Sanket Sharma, and Emre Kiciman. 2016. Characterizing dietary choices, nutrition, and language in food deserts via social media. In Proceedings of the Conference on Computer-supported Cooperative Work & Social Computing. ACM, 1157–1170.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. David Elsweiler, Christoph Trattner, and Morgan Harvey. 2017. Exploiting food choice biases for healthier recipe recommendation. In Proceedings of the International SIGIR Conference on Research and Development in Information Retrieval. ACM, 575–584.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Aleksandr Farseev and Tat-Seng Chua. 2017. Tweet can be fit: Integrating data from wearable sensors and multiple social networks for wellness profile learning. Trans. Inf. Syst. 35, 4 (2017), 42.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Fuli Feng, Xiangnan He, Yiqun Liu, Liqiang Nie, and Tat-Seng Chua. 2018. Learning on partial-order hypergraphs. In Proceedings of the World Wide Web Conference. ACM, 1523–1532.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Jill Freyne and Shlomo Berkovsky. 2010. Intelligent food planning: Personalized recipe recommendation. In Proceedings of the International Conference on Intelligent User Interfaces. ACM, 321–324.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Mouzhi Ge, Mehdi Elahi, Ignacio Fernaández-Tobías, Francesco Ricci, and David Massimo. 2015. Using tags and latent factors in a food recommender system. In Proceedings of the International Conference on Digital Health. ACM, 105–112.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Mouzhi Ge, Francesco Ricci, and David Massimo. 2015. Health-aware food recommender system. In Proceedings of the Conference on Recommender Systems. ACM, 333–334.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Jun Harashima, Yuichiro Someya, and Yohei Kikuta. 2017. Cookpad image dataset: An image collection as infrastructure for food research. In Proceedings of the International SIGIR Conference on Research and Development in Information Retrieval. ACM, 1229–1232.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Morgan Harvey, Bernd Ludwig, and David Elsweiler. 2013. You are what you eat: Learning user tastes for rating prediction. In Proceedings of the International Symposium on String Processing and Information Retrieval. Springer, 153–164.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the International Conference on World Wide Web. ACM, 173–182.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Xiangnan He, Hanwang Zhang, Min-Yen Kan, and Tat-Seng Chua. 2016. Fast matrix factorization for online recommendation with implicit feedback. In Proceedings of the International SIGIR Conference on Research and Development in Information Retrieval. ACM, 549–558.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Jonathan L. Herlocker, Joseph A. Konstan, Al Borchers, and John Riedl. 1999. An algorithmic framework for performing collaborative filtering. In Proceedings of the International SIGIR Conference on Research and Development in Information Retrieval. ACM, 230–237.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Jose Antonio Iglesias, Plamen Angelov, Agapito Ledezma, and Araceli Sanchis. 2011. Creating evolving user behavior profiles automatically. Trans. Knowl. Data Eng. 24, 5 (2011), 854–867.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Hao Jiang, Wenjie Wang, Meng Liu, Liqiang Nie, Ling-Yu Duan, and Changsheng Xu. 2019. Market2Dish: A health-aware food recommendation system. In Proceedings of the International Conference on Multimedia. ACM, 2188–2190.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics. ACL, 427–431.Google ScholarGoogle ScholarCross RefCross Ref
  29. Hokuto Kagaya, Kiyoharu Aizawa, and Makoto Ogawa. 2014. Food detection and recognition using convolutional neural network. In Proceedings of the International Conference on Multimedia. ACM, 1085–1088.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. ACL, 1746–1751.Google ScholarGoogle ScholarCross RefCross Ref
  31. Keigo Kitamura, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2008. Food log by analyzing food images. In Proceedings of the International Conference on Multimedia. ACM, 999–1000.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Tomasz Kusmierczyk, Christoph Trattner, and Kjetil Nørvåg. 2015. Temporality in online food recipe consumption and production. In Proceedings of the International Conference on World Wide Web. ACM, 55–56.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In Proceedings of the International AAAI Conference on Artificial Intelligence. AAAI Press, 2267–2273.Google ScholarGoogle Scholar
  34. Yongqi Li, Meng Liu, Jianhua Yin, Chaoran Cui, Xin-Shun Xu, and Liqiang Nie. 2019. Routing micro-videos via a temporal graph-guided recommendation system. In Proceedings of the International Conference on Multimedia. ACM, 1464–1472.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Yelena Mejova, Hamed Haddadi, Anastasios Noulas, and Ingmar Weber. 2015.  # foodporn: Obesity patterns in culinary interactions. In Proceedings of the International Conference on Digital Health. ACM, 51–58.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. The MIT Press, 3111–3119.Google ScholarGoogle Scholar
  37. Weiqing Min, Bing-Kun Bao, Shuhuan Mei, Yaohui Zhu, Yong Rui, and Shuqiang Jiang. 2017. You are what you eat: Exploring rich recipe information for cross-region food analysis. Transactions on Multimedia 20, 4 (2017), 950--964.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain. 2018. A survey on food computing. arXiv preprint arXiv:1808.07202 (2018).Google ScholarGoogle Scholar
  39. Weiqing Min, Shuqiang Jiang, Shuhui Wang, Jitao Sang, and Shuhuan Mei. 2017. A delicious recipe analysis framework for exploring multi-modal recipes with various attributes. In Proceedings of the International Conference on Multimedia. ACM, 402–410.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Weiqing Min, Linhu Liu, Zhengdong Luo, and Shuqiang Jiang. 2019. Ingredient-guided cascaded multi-attention network for food recognition. In Proceedings of the International Conference on Multimedia. 1331--1339.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Liqiang Nie, Wenjie Wang, Richang Hong, Meng Wang, and Qi Tian. 2019. Multimodal dialog system: Generating responses via adaptive decoders. In Proceedings of the International Conference on Multimedia. ACM, 1098–1106.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Ferda Ofli, Yusuf Aytar, Ingmar Weber, Raggi Al Hammouri, and Antonio Torralba. 2017. The food perception gap on Instagram and its relation to health. In Proceedings of the International Conference on World Wide Web. ACM, 509–518.Google ScholarGoogle Scholar
  43. Parisa Pouladzadeh and Shervin Shirmohammadi. 2017. Mobile multi-food recognition using deep learning. Trans. Multimedia Comput. Commun. Applic. 13, 3s (Aug. 2017), 36:1–36:21.Google ScholarGoogle Scholar
  44. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics. ACL, 157–163.Google ScholarGoogle ScholarCross RefCross Ref
  45. Markus Rokicki, Christoph Trattner, and Eelco Herder. 2018. The impact of recipe features, social cues and demographics on estimating the healthiness of online recipes. In Proceedings of the International Conference on Web and Social Media. AAAI Press, 310–319.Google ScholarGoogle Scholar
  46. Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. 2017. Learning cross-modal embeddings for cooking recipes and food images. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, 3020–3028.Google ScholarGoogle ScholarCross RefCross Ref
  47. Satoshi Sanjo and Marie Katsurai. 2017. Recipe popularity prediction with deep visual-semantic fusion. In Proceedings of the Conference on Information and Knowledge Management. ACM, 2279–2282.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Yuka Shidochi, Tomokazu Takahashi, Ichiro Ide, and Hiroshi Murase. 2009. Finding replaceable materials in cooking recipe texts considering characteristic cooking actions. In Proceedings of the Multimedia Workshop on Multimedia for Cooking and Eating Activities. ACM, 9–14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google ScholarGoogle Scholar
  50. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, 2818–2826.Google ScholarGoogle ScholarCross RefCross Ref
  51. Christoph Trattner and David Elsweiler. 2017. Food recommender systems: Important contributions, challenges and future research directions. arXiv preprint arXiv:1711.02760 (2017).Google ScholarGoogle Scholar
  52. Christoph Trattner and David Elsweiler. 2017. Investigating the healthiness of internet-sourced recipes: Implications for meal planning and recommender systems. In Proceedings of the International Conference on World Wide Web. ACM, 489–498.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Liping Wang, Qing Li, Na Li, Guozhu Dong, and Yu Yang. 2008. Substructure similarity measurement in Chinese recipes. In Proceedings of the International Conference on World Wide Web. ACM, 979–988.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Shuohang Wang and Jing Jiang. 2016. Learning natural language inference with LSTM. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. ACL, 1442–1451.Google ScholarGoogle ScholarCross RefCross Ref
  55. Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2020. Denoising implicit feedback for recommendation. arXiv preprint arXiv:2006.04153 (2020).Google ScholarGoogle Scholar
  56. Wenjie Wang, Minlie Huang, Xin-Shun Xu, Fumin Shen, and Liqiang Nie. 2018. Chat more: Deepening and widening the chatting topic via a deep model. In Proceedings of the International SIGIR Conference on Research & Development in Information Retrieval. ACM, 255–264.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural graph collaborative filtering. In Proceedings of the International SIGIR Conference on Research & Development in Information Retrieval. ACM, 165–174.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Hui Wu, Michele Merler, Rosario Uceda-Sosa, and John R. Smith. 2016. Learning to make better mistakes: Semantics-aware visual food recognition. In Proceedings of the International Conference on Multimedia. ACM, 172–176.Google ScholarGoogle Scholar
  59. Shulin Yang, Mei Chen, Dean Pomerleau, and Rahul Sukthankar. 2010. Food recognition using statistics of pairwise local features. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, 2249–2256.Google ScholarGoogle Scholar
  60. Yiming Yang. 1999. An evaluation of statistical approaches to text categorization. Inf. Retr. 1, 1–2 (1999), 69–90.Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. ACL, 1480–1489.Google ScholarGoogle ScholarCross RefCross Ref
  62. Xun Yi, Elisa Bertino, Fang-Yu Rao, Kwok-Yan Lam, Surya Nepal, and Athman Bouguettaya. 2019. Privacy-preserving user profile matching in social networks. Trans. Knowl. Data Eng. 32, 8 (2019), 1572--1585.Google ScholarGoogle ScholarCross RefCross Ref
  63. Zhou Zhao, Hanqing Lu, Deng Cai, Xiaofei He, and Yueting Zhuang. 2016. User preference learning for online social recommendation. Trans. Knowl. Data Eng. 28, 9 (2016), 2522–2534.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Market2Dish: Health-aware Food Recommendation

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM Transactions on Multimedia Computing, Communications, and Applications
            ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 17, Issue 1
            February 2021
            392 pages
            ISSN:1551-6857
            EISSN:1551-6865
            DOI:10.1145/3453992
            Issue’s Table of Contents

            Copyright © 2021 Association for Computing Machinery.

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 16 April 2021
            • Accepted: 1 July 2020
            • Revised: 1 June 2020
            • Received: 1 September 2019
            Published in tomm Volume 17, Issue 1

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Refereed

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!