Abstract
With the rising incidence of some diseases, such as obesity and diabetes, the healthy diet is arousing increasing attention. However, most existing food-related research efforts focus on recipe retrieval, user-preference-based food recommendation, cooking assistance, or the nutrition and calorie estimation of dishes, ignoring the personalized health-aware food recommendation. Therefore, in this work, we present a personalized health-aware food recommendation scheme, namely, Market2Dish, mapping the ingredients displayed in the market to the healthy dishes eaten at home. The proposed scheme comprises three components, namely, recipe retrieval, user health profiling, and health-aware food recommendation. In particular, recipe retrieval aims to acquire the ingredients available to the users and then retrieve recipe candidates from a large-scale recipe dataset. User health profiling is to characterize the health conditions of users by capturing the textual health-related information crawled from social networks. Specifically, to solve the issue that the health-related information is extremely sparse, we incorporate a word-class interaction mechanism into the proposed deep model to learn the fine-grained correlations between the textual tweets and pre-defined health concepts. For the health-aware food recommendation, we present a novel category-aware hierarchical memory network–based recommender to learn the health-aware user-recipe interactions for better food recommendation. Moreover, extensive experiments demonstrate the effectiveness of the health-aware food recommendation scheme.
- Sofiane Abbar, Yelena Mejova, and Ingmar Weber. 2015. You tweet what you eat: Studying food consumption through Twitter. In Proceedings of the Conference on Human Factors in Computing Systems. ACM, 3197–3206.Google Scholar
Digital Library
- Yongsheng An, Yu Cao, Jingjing Chen, Chong-Wah Ngo, Jia Jia, Huanbo Luan, and Tat-Seng Chua. 2017. PIC2DISH: A customized cooking assistant system. In Proceedings of the International Conference on Multimedia. ACM, 1269–1273.Google Scholar
Digital Library
- Oren Barkan and Noam Koenigstein. 2016. Item2Vec: Neural item embedding for collaborative filtering. In Proceedings of the International Workshop on Machine Learning for Signal Processing. IEEE, 1–6.Google Scholar
Cross Ref
- Immanuel Bayer, Xiangnan He, Bhargav Kanagal, and Steffen Rendle. 2017. A generic coordinate descent framework for learning from implicit feedback. In Proceedings of the International Conference on World Wide Web. ACM, 1341–1350.Google Scholar
Digital Library
- Alexandre Blansché, Julien Cojan, Valmi Dufour-Lussier, Jean Lieber, Pascal Molli, Emmanuel Nauer, Hala Skaf-Molli, and Yannick Toussaint. 2010. Taaable 3: Adaptation of ingredient quantities and of textual preparations. In Proceedings of the International Conference on Case-based Reasoning Workshop. 189–198.Google Scholar
- Micael Carvalho, Rémi Cadène, David Picard, Laure Soulier, Nicolas Thome, and Matthieu Cord. 2018. Cross-modal retrieval in the cooking context: Learning semantic text-image embeddings. In Proceedings of the International SIGIR Conference on Research & Development in Information Retrieval. ACM, 35–44.Google Scholar
Digital Library
- Jingjing Chen, Lei Pang, and Chong-Wah Ngo. 2017. Cross-modal recipe retrieval: How to cook this dish? In Proceedings of the International Conference on Multimedia Modeling. Springer, 588–600.Google Scholar
Cross Ref
- Jing-jing Chen, Chong-Wah Ngo, and Tat-Seng Chua. 2017. Cross-modal recipe retrieval with rich food attributes. In Proceedings of the International Conference on Multimedia. ACM, 1771–1779.Google Scholar
- Jing-Jing Chen, Chong-Wah Ngo, Fu-Li Feng, and Tat-Seng Chua. 2018. Deep understanding of cooking procedure for cross-modal recipe retrieval. In Proceedings of the International Conference on Multimedia. ACM, 1020–1028.Google Scholar
Digital Library
- Zhiyong Cheng and Jialie Shen. 2014. Just-for-Me: An adaptive personalization system for location-aware social music recommendation. In Proceedings of the International Conference on Multimedia Retrieval. ACM, 185–192.Google Scholar
Digital Library
- Zhiyong Cheng and Jialie Shen. 2016. On effective location-aware music recommendation. ACM Trans. Inf. Syst. 34, 2 (2016).Google Scholar
Digital Library
- Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12, Aug. (2011), 2493–2537.Google Scholar
Digital Library
- Du Cunxiao, Chen Zhaozheng, Feng Fuli, Zhu Lei, Gan Tian, and Nie Liqiang. 2019. Explicit interaction model towards text classification. In Proceedings of the International AAAI Conference on Artificial Intelligence. AAAI Press, 6359–6366.Google Scholar
- Munmun De Choudhury, Sanket Sharma, and Emre Kiciman. 2016. Characterizing dietary choices, nutrition, and language in food deserts via social media. In Proceedings of the Conference on Computer-supported Cooperative Work & Social Computing. ACM, 1157–1170.Google Scholar
Digital Library
- David Elsweiler, Christoph Trattner, and Morgan Harvey. 2017. Exploiting food choice biases for healthier recipe recommendation. In Proceedings of the International SIGIR Conference on Research and Development in Information Retrieval. ACM, 575–584.Google Scholar
Digital Library
- Aleksandr Farseev and Tat-Seng Chua. 2017. Tweet can be fit: Integrating data from wearable sensors and multiple social networks for wellness profile learning. Trans. Inf. Syst. 35, 4 (2017), 42.Google Scholar
Digital Library
- Fuli Feng, Xiangnan He, Yiqun Liu, Liqiang Nie, and Tat-Seng Chua. 2018. Learning on partial-order hypergraphs. In Proceedings of the World Wide Web Conference. ACM, 1523–1532.Google Scholar
Digital Library
- Jill Freyne and Shlomo Berkovsky. 2010. Intelligent food planning: Personalized recipe recommendation. In Proceedings of the International Conference on Intelligent User Interfaces. ACM, 321–324.Google Scholar
Digital Library
- Mouzhi Ge, Mehdi Elahi, Ignacio Fernaández-Tobías, Francesco Ricci, and David Massimo. 2015. Using tags and latent factors in a food recommender system. In Proceedings of the International Conference on Digital Health. ACM, 105–112.Google Scholar
Digital Library
- Mouzhi Ge, Francesco Ricci, and David Massimo. 2015. Health-aware food recommender system. In Proceedings of the Conference on Recommender Systems. ACM, 333–334.Google Scholar
Digital Library
- Jun Harashima, Yuichiro Someya, and Yohei Kikuta. 2017. Cookpad image dataset: An image collection as infrastructure for food research. In Proceedings of the International SIGIR Conference on Research and Development in Information Retrieval. ACM, 1229–1232.Google Scholar
Digital Library
- Morgan Harvey, Bernd Ludwig, and David Elsweiler. 2013. You are what you eat: Learning user tastes for rating prediction. In Proceedings of the International Symposium on String Processing and Information Retrieval. Springer, 153–164.Google Scholar
Digital Library
- Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the International Conference on World Wide Web. ACM, 173–182.Google Scholar
Digital Library
- Xiangnan He, Hanwang Zhang, Min-Yen Kan, and Tat-Seng Chua. 2016. Fast matrix factorization for online recommendation with implicit feedback. In Proceedings of the International SIGIR Conference on Research and Development in Information Retrieval. ACM, 549–558.Google Scholar
Digital Library
- Jonathan L. Herlocker, Joseph A. Konstan, Al Borchers, and John Riedl. 1999. An algorithmic framework for performing collaborative filtering. In Proceedings of the International SIGIR Conference on Research and Development in Information Retrieval. ACM, 230–237.Google Scholar
Digital Library
- Jose Antonio Iglesias, Plamen Angelov, Agapito Ledezma, and Araceli Sanchis. 2011. Creating evolving user behavior profiles automatically. Trans. Knowl. Data Eng. 24, 5 (2011), 854–867.Google Scholar
Digital Library
- Hao Jiang, Wenjie Wang, Meng Liu, Liqiang Nie, Ling-Yu Duan, and Changsheng Xu. 2019. Market2Dish: A health-aware food recommendation system. In Proceedings of the International Conference on Multimedia. ACM, 2188–2190.Google Scholar
Digital Library
- Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics. ACL, 427–431.Google Scholar
Cross Ref
- Hokuto Kagaya, Kiyoharu Aizawa, and Makoto Ogawa. 2014. Food detection and recognition using convolutional neural network. In Proceedings of the International Conference on Multimedia. ACM, 1085–1088.Google Scholar
Digital Library
- Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. ACL, 1746–1751.Google Scholar
Cross Ref
- Keigo Kitamura, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2008. Food log by analyzing food images. In Proceedings of the International Conference on Multimedia. ACM, 999–1000.Google Scholar
Digital Library
- Tomasz Kusmierczyk, Christoph Trattner, and Kjetil Nørvåg. 2015. Temporality in online food recipe consumption and production. In Proceedings of the International Conference on World Wide Web. ACM, 55–56.Google Scholar
Digital Library
- Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In Proceedings of the International AAAI Conference on Artificial Intelligence. AAAI Press, 2267–2273.Google Scholar
- Yongqi Li, Meng Liu, Jianhua Yin, Chaoran Cui, Xin-Shun Xu, and Liqiang Nie. 2019. Routing micro-videos via a temporal graph-guided recommendation system. In Proceedings of the International Conference on Multimedia. ACM, 1464–1472.Google Scholar
Digital Library
- Yelena Mejova, Hamed Haddadi, Anastasios Noulas, and Ingmar Weber. 2015. # foodporn: Obesity patterns in culinary interactions. In Proceedings of the International Conference on Digital Health. ACM, 51–58.Google Scholar
Digital Library
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. The MIT Press, 3111–3119.Google Scholar
- Weiqing Min, Bing-Kun Bao, Shuhuan Mei, Yaohui Zhu, Yong Rui, and Shuqiang Jiang. 2017. You are what you eat: Exploring rich recipe information for cross-region food analysis. Transactions on Multimedia 20, 4 (2017), 950--964.Google Scholar
Digital Library
- Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain. 2018. A survey on food computing. arXiv preprint arXiv:1808.07202 (2018).Google Scholar
- Weiqing Min, Shuqiang Jiang, Shuhui Wang, Jitao Sang, and Shuhuan Mei. 2017. A delicious recipe analysis framework for exploring multi-modal recipes with various attributes. In Proceedings of the International Conference on Multimedia. ACM, 402–410.Google Scholar
Digital Library
- Weiqing Min, Linhu Liu, Zhengdong Luo, and Shuqiang Jiang. 2019. Ingredient-guided cascaded multi-attention network for food recognition. In Proceedings of the International Conference on Multimedia. 1331--1339.Google Scholar
Digital Library
- Liqiang Nie, Wenjie Wang, Richang Hong, Meng Wang, and Qi Tian. 2019. Multimodal dialog system: Generating responses via adaptive decoders. In Proceedings of the International Conference on Multimedia. ACM, 1098–1106.Google Scholar
Digital Library
- Ferda Ofli, Yusuf Aytar, Ingmar Weber, Raggi Al Hammouri, and Antonio Torralba. 2017. The food perception gap on Instagram and its relation to health. In Proceedings of the International Conference on World Wide Web. ACM, 509–518.Google Scholar
- Parisa Pouladzadeh and Shervin Shirmohammadi. 2017. Mobile multi-food recognition using deep learning. Trans. Multimedia Comput. Commun. Applic. 13, 3s (Aug. 2017), 36:1–36:21.Google Scholar
- Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics. ACL, 157–163.Google Scholar
Cross Ref
- Markus Rokicki, Christoph Trattner, and Eelco Herder. 2018. The impact of recipe features, social cues and demographics on estimating the healthiness of online recipes. In Proceedings of the International Conference on Web and Social Media. AAAI Press, 310–319.Google Scholar
- Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. 2017. Learning cross-modal embeddings for cooking recipes and food images. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, 3020–3028.Google Scholar
Cross Ref
- Satoshi Sanjo and Marie Katsurai. 2017. Recipe popularity prediction with deep visual-semantic fusion. In Proceedings of the Conference on Information and Knowledge Management. ACM, 2279–2282.Google Scholar
Digital Library
- Yuka Shidochi, Tomokazu Takahashi, Ichiro Ide, and Hiroshi Murase. 2009. Finding replaceable materials in cooking recipe texts considering characteristic cooking actions. In Proceedings of the Multimedia Workshop on Multimedia for Cooking and Eating Activities. ACM, 9–14.Google Scholar
Digital Library
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, 2818–2826.Google Scholar
Cross Ref
- Christoph Trattner and David Elsweiler. 2017. Food recommender systems: Important contributions, challenges and future research directions. arXiv preprint arXiv:1711.02760 (2017).Google Scholar
- Christoph Trattner and David Elsweiler. 2017. Investigating the healthiness of internet-sourced recipes: Implications for meal planning and recommender systems. In Proceedings of the International Conference on World Wide Web. ACM, 489–498.Google Scholar
Digital Library
- Liping Wang, Qing Li, Na Li, Guozhu Dong, and Yu Yang. 2008. Substructure similarity measurement in Chinese recipes. In Proceedings of the International Conference on World Wide Web. ACM, 979–988.Google Scholar
Digital Library
- Shuohang Wang and Jing Jiang. 2016. Learning natural language inference with LSTM. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. ACL, 1442–1451.Google Scholar
Cross Ref
- Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2020. Denoising implicit feedback for recommendation. arXiv preprint arXiv:2006.04153 (2020).Google Scholar
- Wenjie Wang, Minlie Huang, Xin-Shun Xu, Fumin Shen, and Liqiang Nie. 2018. Chat more: Deepening and widening the chatting topic via a deep model. In Proceedings of the International SIGIR Conference on Research & Development in Information Retrieval. ACM, 255–264.Google Scholar
Digital Library
- Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural graph collaborative filtering. In Proceedings of the International SIGIR Conference on Research & Development in Information Retrieval. ACM, 165–174.Google Scholar
Digital Library
- Hui Wu, Michele Merler, Rosario Uceda-Sosa, and John R. Smith. 2016. Learning to make better mistakes: Semantics-aware visual food recognition. In Proceedings of the International Conference on Multimedia. ACM, 172–176.Google Scholar
- Shulin Yang, Mei Chen, Dean Pomerleau, and Rahul Sukthankar. 2010. Food recognition using statistics of pairwise local features. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, 2249–2256.Google Scholar
- Yiming Yang. 1999. An evaluation of statistical approaches to text categorization. Inf. Retr. 1, 1–2 (1999), 69–90.Google Scholar
Digital Library
- Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. ACL, 1480–1489.Google Scholar
Cross Ref
- Xun Yi, Elisa Bertino, Fang-Yu Rao, Kwok-Yan Lam, Surya Nepal, and Athman Bouguettaya. 2019. Privacy-preserving user profile matching in social networks. Trans. Knowl. Data Eng. 32, 8 (2019), 1572--1585.Google Scholar
Cross Ref
- Zhou Zhao, Hanqing Lu, Deng Cai, Xiaofei He, and Yueting Zhuang. 2016. User preference learning for online social recommendation. Trans. Knowl. Data Eng. 28, 9 (2016), 2522–2534.Google Scholar
Digital Library
Index Terms
Market2Dish: Health-aware Food Recommendation
Recommendations
Personalized Food Recommendation as Constrained Question Answering over a Large-scale Food Knowledge Graph
WSDM '21: Proceedings of the 14th ACM International Conference on Web Search and Data MiningFood recommendation has become an important means to help guide users to adopt healthy dietary habits. Previous works on food recommendation either i) fail to consider users' explicit requirements, ii) ignore crucial health factors (e.g., allergies and ...
Market2Dish: A Health-aware Food Recommendation System
MM '19: Proceedings of the 27th ACM International Conference on MultimediaIn order to help people develop healthy eating habits, we present a personalized health-aware food recommendation system, calledMarket2Dish. Market2Dish could recognize the ingredients in the micro-videos taken from the market, characterize the health ...
Health-aware Food Recommender System
RecSys '15: Proceedings of the 9th ACM Conference on Recommender SystemsWith the rapid changes in the food variety and lifestyles, many people are facing the problem of making healthier food decisions to reduce the risk of chronic diseases such as obesity and diabetes. To this end, our recommender system not only offers ...






Comments