skip to main content
research-article

Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion

Authors Info & Claims
Published:06 February 2023Publication History
Skip Abstract Section

Abstract

Knowledge graphs often suffer from incompleteness, and knowledge graph completion (KGC) aims at inferring the missing triplets through knowledge graph embedding from known factual triplets. However, most existing knowledge graph embedding methods only use the relational information of knowledge graph and treat the entities and relations as IDs with simple embedding layer, ignoring the multi-modal information among triplets, such as text descriptions, images, etc. In this work, we propose a novel network to incorporate different modal information with graph structure information for more precise representation of multi-modal knowledge graph, termed as hyper-node relational graph attention (HRGAT) network. In HRGAT, we use low-rank multi-modal fusion to model the intra-modality and inter-modality dynamics, which transforms the original knowledge graph to a hyper-node graph. Then, relational graph attention (RGAT) network is used, which contains relation-specific attention and entity-relation fusion operation to capture the graph structure information. Finally, we aggregate the updated multi-modal information and graph structure information to generate the final embeddings of knowledge graph to achieve KGC. By exploring multi-modal information and graph structure information, HRGAT embraces faster convergence speed and achieves the state-of-the-art for KGC on the standard datasets. Implementation code is available at https://github.com/broliang/HRGAT.

REFERENCES

  1. [1] An Bo, Chen Bo, Han Xianpei, and Sun Le. 2018. Accurate text-enhanced knowledge graph representation learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, Volume 1 (Long Papers). 745755.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Auer Sören, Bizer Christian, Kobilarov Georgi, Lehmann Jens, Cyganiak Richard, and Ives Zachary G.. 2007. DBpedia: A nucleus for a web of open data. In The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007. 722735.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Balazevic Ivana, Allen Carl, and Hospedales Timothy M.. 2019. Hypernetwork knowledge graph embeddings. In Artificial Neural Networks and Machine Learning - ICANN 2019-28th International Conference on Artificial Neural Networks. 553565.Google ScholarGoogle Scholar
  4. [4] Balazevic Ivana, Allen Carl, and Hospedales Timothy M.. 2019. TuckER: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019. 51845193.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Bollacker Kurt D., Evans Colin, Paritosh Praveen, Sturge Tim, and Taylor Jamie. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008. 12471250.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. [6] Bordes Antoine, Usunier Nicolas, García-Durán Alberto, Weston Jason, and Yakhnenko Oksana. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. 27872795.Google ScholarGoogle Scholar
  7. [7] Bordes Antoine, Weston Jason, Collobert Ronan, and Bengio Yoshua. 2011. Learning structured embeddings of knowledge bases. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Chaudhary Chandramani, Goyal Poonam, Goyal Navneet, and Chen Yi-Ping Phoebe. 2020. Image retrieval for complex queries using knowledge embedding. ACM Trans. Multim. Comput. Commun. Appl. 16, 1 (2020), 13:1–13:23.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Chen Liyi, Li Zhi, Wang Yijun, Xu Tong, Wang Zhefeng, and Chen Enhong. 2020. MMEA: Entity alignment for multi-modal knowledge graph. In Knowledge Science, Engineering and Management - 13th International Conference, KSEM 2020. 134147.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Chen Yao, Liu Jiangang, Zhang Zhe, Wen Shiping, and Xiong Wenjun. 2021. MöbiusE: Knowledge graph embedding on möbius ring. Knowl. Based Syst. 227 (2021), 107181.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Chen Zhenfang, Mao Jiayuan, Wu Jiajun, Wong Kwan-Yee Kenneth, Tenenbaum Joshua B., and Gan Chuang. 2021. Grounding physical concepts of objects and events through dynamic visual reasoning. In 9th International Conference on Learning Representations, ICLR 2021.Google ScholarGoogle Scholar
  12. [12] Chen Zhenfang, Yi Kexin, Li Yunzhu, Ding Mingyu, Tenenbaum Joshua B., Torralba Antonio, and Gan Chuang. 2022. ComPhy: Compositional physical reasoning of objects and events from videos. In 10th International Conference on Learning Representations, ICLR 2022.Google ScholarGoogle Scholar
  13. [13] Dettmers Tim, Minervini Pasquale, Stenetorp Pontus, and Riedel Sebastian. 2018. Convolutional 2D knowledge graph embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI 2018. 18111818.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Devlin Jacob, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Volume 1 (Long and Short Papers). 41714186.Google ScholarGoogle Scholar
  15. [15] Dong Xin, Gabrilovich Evgeniy, Heitz Geremy, Horn Wilko, Lao Ni, Murphy Kevin, Strohmann Thomas, Sun Shaohua, and Zhang Wei. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’14. 601610.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Ebisu Takuma and Ichise Ryutaro. 2020. Generalized translation-based embedding of knowledge graph. IEEE Trans. Knowl. Data Eng. 32, 5 (2020), 941951.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] García-Durán Alberto and Niepert Mathias. 2018. KBlrn: End-to-end learning of knowledge base representations with latent, relational, and numerical features. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018. 372381.Google ScholarGoogle Scholar
  18. [18] Huang Xuqian, Tang Jiuyang, Tan Zhen, Zeng Weixin, Wang Ji, and Zhao Xiang. 2021. Knowledge graph embedding by relational and entity rotation. Knowl. Based Syst. 229 (2021), 107310.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Jiang Dan, Wang Ronggui, Yang Juan, and Xue Lixia. 2021. Kernel multi-attention neural network for knowledge graph embedding. Knowl. Based Syst. 227 (2021), 107188.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. [20] Kipf Thomas N., Fetaya Ethan, Wang Kuan-Chieh, Welling Max, and Zemel Richard S.. 2018. Neural relational inference for interacting systems. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018. 26932702.Google ScholarGoogle Scholar
  21. [21] Kipf Thomas N. and Welling Max. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017.Google ScholarGoogle Scholar
  22. [22] Lin Yankai, Liu Zhiyuan, Sun Maosong, Liu Yang, and Zhu Xuan. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI 2015. 21812187.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Liu Hanxiao, Wu Yuexin, and Yang Yiming. 2017. Analogical inference for multi-relational embeddings. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017. 21682178.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Liu Ye, Li Hui, García-Durán Alberto, Niepert Mathias, Oñoro-Rubio Daniel, and Rosenblum David S.. 2019. MMKG: Multi-modal knowledge graphs. In The Semantic Web - 16th International Conference, ESWC 2019. 459474.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Liu Zhun, Shen Ying, Lakshminarasimhan Varun Bharadhwaj, Liang Paul Pu, Zadeh Amir, and Morency Louis-Philippe. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Volume 1: Long Papers. 22472256.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Nathani Deepak, Chauhan Jatin, Sharma Charu, and Kaul Manohar. 2019. Learning attention-based embeddings for relation prediction in knowledge graphs. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019. 47104723.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Nguyen Dai Quoc, Nguyen Tu Dinh, Nguyen Dat Quoc, and Phung Dinh Q.. 2018. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, Volume 2 (Short Papers). 327333.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Nickel Maximilian, Tresp Volker, and Kriegel Hans-Peter. 2011. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011. 809816.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Pezeshkpour Pouya, Chen Liyan, and Singh Sameer. 2018. Embedding multimodal relational data for knowledge base completion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, EMNLP 2018. 32083218.Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Qian Shengsheng, Hu Jun, Fang Quan, and Xu Changsheng. 2021. Knowledge-aware multi-modal adaptive graph convolutional networks for fake news detection. ACM Trans. Multim. Comput. Commun. Appl. 17, 3 (2021), 98:1–98:23.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Reimers Nils and Gurevych Iryna. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019. 39803990.Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Schlichtkrull Michael Sejr, Kipf Thomas N., Bloem Peter, Berg Rianne van den, Titov Ivan, and Welling Max. 2018. Modeling relational data with graph convolutional networks. In The Semantic Web - 15th International Conference, ESWC 2018. 593607.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Shang Chao, Tang Yun, Huang Jing, Bi Jinbo, He Xiaodong, and Zhou Bowen. 2019. End-to-end structure-aware convolutional networks for knowledge base completion. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019. 30603067.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Simonyan Karen and Zisserman Andrew. 2015. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015.Google ScholarGoogle Scholar
  35. [35] Suchanek Fabian M., Kasneci Gjergji, and Weikum Gerhard. 2007. YAGO: A core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web, WWW 2007. 697706.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Sun Rui, Cao Xuezhi, Zhao Yan, Wan Junchen, Zhou Kun, Zhang Fuzheng, Wang Zhongyuan, and Zheng Kai. 2020. Multi-modal knowledge graphs for recommender systems. In CIKM’20: The 29th ACM International Conference on Information and Knowledge Management. 14051414.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Sun Zhiqing, Deng Zhihong, Nie Jianyun, and Tang Jian. 2019. RotatE: Knowledge graph embedding by relational rotation in complex space. In 7th International Conference on Learning Representations, ICLR 2019.Google ScholarGoogle Scholar
  38. [38] Toutanova Kristina and Chen Danqi. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, CVSC 2015. 5766.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Trouillon Théo, Welbl Johannes, Riedel Sebastian, Gaussier Éric, and Bouchard Guillaume. 2016. Complex embeddings for simple link prediction. In Proceedings of the 33rd International Conference on Machine Learning, ICML 2016. 20712080.Google ScholarGoogle Scholar
  40. [40] Vashishth Shikhar, Sanyal Soumya, Nitin Vikram, and Talukdar Partha P.. 2020. Composition-based multi-relational graph convolutional networks. In 8th International Conference on Learning Representations, ICLR 2020.Google ScholarGoogle Scholar
  41. [41] Vrandecic Denny and Krötzsch Markus. 2014. Wikidata: A free collaborative knowledgebase. Commun. ACM 57, 10 (2014), 7885.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Wu Bo, Liu Sijia, Chen Pin-Yu, Zhang Gaoyuan, and Gan Chuang. 2021. STAR: A benchmark for situated reasoning in real-world videos. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021).Google ScholarGoogle Scholar
  43. [43] Xiao Han, Huang Minlie, and Zhu Xiaoyan. 2016. TransG : A generative model for knowledge graph embedding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Volume 1: Long Papers.Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Xie Ruobing, Liu Zhiyuan, Luan Huanbo, and Sun Maosong. 2017. Image-embodied knowledge representation learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017. 31403146.Google ScholarGoogle ScholarCross RefCross Ref
  45. [45] Yang Bishan, Yih Wen-tau, He Xiaodong, Gao Jianfeng, and Deng Li. 2015. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations, ICLR 2015.Google ScholarGoogle Scholar
  46. [46] Yao Liang, Mao Chengsheng, and Luo Yuan. 2019. KG-BERT: BERT for knowledge graph completion. CoRR abs/1909.03193 (2019).Google ScholarGoogle Scholar
  47. [47] Yi Kexin, Gan Chuang, Li Yunzhu, Kohli Pushmeet, Wu Jiajun, Torralba Antonio, and Tenenbaum Joshua B.. 2020. CLEVRER: Collision events for video representation and reasoning. In 8th International Conference on Learning Representations, ICLR 2020.Google ScholarGoogle Scholar
  48. [48] Zadeh Amir, Chen Minghai, Poria Soujanya, Cambria Erik, and Morency Louis-Philippe. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017. 11031114.Google ScholarGoogle ScholarCross RefCross Ref
  49. [49] Zeb Adnan, Haqg Anwar Ul, Chen Junde, Lei Zhenfeng, and Zhang Defu. 2021. Learning hyperbolic attention-based embeddings for link prediction in knowledge graphs. Knowl. Based Syst. 229 (2021), 107369.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. [50] Zhang Yingying, Fang Quan, Qian Shengsheng, and Xu Changsheng. 2020. Multi-modal multi-relational feature aggregation network for medical knowledge representation learning. In MM’20: The 28th ACM International Conference on Multimedia. 39563965.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. [51] Zhang Yingying, Qian Shengsheng, Fang Quan, and Xu Changsheng. 2019. Multi-modal knowledge-aware hierarchical attention network for explainable medical question answering. In Proceedings of the 27th ACM International Conference on Multimedia, MM 2019. 10891097.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. [52] Zhang Zhaoli, Li Zhifei, Liu Hai, and Xiong Neal N.. 2022. Multi-scale dynamic convolutional network for knowledge graph embedding. IEEE Trans. Knowl. Data Eng. 34, 5 (2022), 23352347.Google ScholarGoogle ScholarCross RefCross Ref
  53. [53] Zheng Da, Song Xiang, Ma Chao, Tan Zeyuan, Ye Zihao, Dong Jin, Xiong Hao, Zhang Zheng, and Karypis George. 2020. DGL-KE: Training knowledge graph embeddings at scale. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020. 739748.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. [54] Zhou Zhehui, Wang Can, Feng Yan, and Chen Defang. 2022. JointE: Jointly utilizing 1D and 2D convolution for knowledge graph embedding. Knowl. Based Syst. 240 (2022), 108100.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 2
      March 2023
      540 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3572860
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 6 February 2023
      • Online AM: 30 June 2022
      • Accepted: 23 June 2022
      • Revised: 25 April 2022
      • Received: 26 January 2022
      Published in tomm Volume 19, Issue 2

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed
    • Article Metrics

      • Downloads (Last 12 months)775
      • Downloads (Last 6 weeks)140

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!