Abstract
Knowledge graphs often suffer from incompleteness, and knowledge graph completion (KGC) aims at inferring the missing triplets through knowledge graph embedding from known factual triplets. However, most existing knowledge graph embedding methods only use the relational information of knowledge graph and treat the entities and relations as IDs with simple embedding layer, ignoring the multi-modal information among triplets, such as text descriptions, images, etc. In this work, we propose a novel network to incorporate different modal information with graph structure information for more precise representation of multi-modal knowledge graph, termed as hyper-node relational graph attention (HRGAT) network. In HRGAT, we use low-rank multi-modal fusion to model the intra-modality and inter-modality dynamics, which transforms the original knowledge graph to a hyper-node graph. Then, relational graph attention (RGAT) network is used, which contains relation-specific attention and entity-relation fusion operation to capture the graph structure information. Finally, we aggregate the updated multi-modal information and graph structure information to generate the final embeddings of knowledge graph to achieve KGC. By exploring multi-modal information and graph structure information, HRGAT embraces faster convergence speed and achieves the state-of-the-art for KGC on the standard datasets. Implementation code is available at https://github.com/broliang/HRGAT.
- [1] . 2018. Accurate text-enhanced knowledge graph representation learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, Volume 1 (Long Papers). 745–755.Google Scholar
Cross Ref
- [2] . 2007. DBpedia: A nucleus for a web of open data. In The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007. 722–735.Google Scholar
Cross Ref
- [3] . 2019. Hypernetwork knowledge graph embeddings. In Artificial Neural Networks and Machine Learning - ICANN 2019-28th International Conference on Artificial Neural Networks. 553–565.Google Scholar
- [4] . 2019. TuckER: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019. 5184–5193.Google Scholar
Cross Ref
- [5] . 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008. 1247–1250.Google Scholar
Digital Library
- [6] . 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. 2787–2795.Google Scholar
- [7] . 2011. Learning structured embeddings of knowledge bases. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011.Google Scholar
Digital Library
- [8] . 2020. Image retrieval for complex queries using knowledge embedding. ACM Trans. Multim. Comput. Commun. Appl. 16, 1 (2020), 13:1–13:23.Google Scholar
Digital Library
- [9] . 2020. MMEA: Entity alignment for multi-modal knowledge graph. In Knowledge Science, Engineering and Management - 13th International Conference, KSEM 2020. 134–147.Google Scholar
Digital Library
- [10] . 2021. MöbiusE: Knowledge graph embedding on möbius ring. Knowl. Based Syst. 227 (2021), 107181.Google Scholar
Digital Library
- [11] . 2021. Grounding physical concepts of objects and events through dynamic visual reasoning. In 9th International Conference on Learning Representations, ICLR 2021.Google Scholar
- [12] . 2022. ComPhy: Compositional physical reasoning of objects and events from videos. In 10th International Conference on Learning Representations, ICLR 2022.Google Scholar
- [13] . 2018. Convolutional 2D knowledge graph embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI 2018. 1811–1818.Google Scholar
Cross Ref
- [14] . 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Volume 1 (Long and Short Papers). 4171–4186.Google Scholar
- [15] . 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’14. 601–610.Google Scholar
Digital Library
- [16] . 2020. Generalized translation-based embedding of knowledge graph. IEEE Trans. Knowl. Data Eng. 32, 5 (2020), 941–951.Google Scholar
Cross Ref
- [17] . 2018. KBlrn: End-to-end learning of knowledge base representations with latent, relational, and numerical features. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018. 372–381.Google Scholar
- [18] . 2021. Knowledge graph embedding by relational and entity rotation. Knowl. Based Syst. 229 (2021), 107310.Google Scholar
Digital Library
- [19] . 2021. Kernel multi-attention neural network for knowledge graph embedding. Knowl. Based Syst. 227 (2021), 107188.Google Scholar
Digital Library
- [20] . 2018. Neural relational inference for interacting systems. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018. 2693–2702.Google Scholar
- [21] . 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017.Google Scholar
- [22] . 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI 2015. 2181–2187.Google Scholar
Cross Ref
- [23] . 2017. Analogical inference for multi-relational embeddings. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017. 2168–2178.Google Scholar
Digital Library
- [24] . 2019. MMKG: Multi-modal knowledge graphs. In The Semantic Web - 16th International Conference, ESWC 2019. 459–474.Google Scholar
Cross Ref
- [25] . 2018. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Volume 1: Long Papers. 2247–2256.Google Scholar
Cross Ref
- [26] . 2019. Learning attention-based embeddings for relation prediction in knowledge graphs. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019. 4710–4723.Google Scholar
Cross Ref
- [27] . 2018. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, Volume 2 (Short Papers). 327–333.Google Scholar
Cross Ref
- [28] . 2011. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011. 809–816.Google Scholar
Digital Library
- [29] . 2018. Embedding multimodal relational data for knowledge base completion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, EMNLP 2018. 3208–3218.Google Scholar
Cross Ref
- [30] . 2021. Knowledge-aware multi-modal adaptive graph convolutional networks for fake news detection. ACM Trans. Multim. Comput. Commun. Appl. 17, 3 (2021), 98:1–98:23.Google Scholar
Digital Library
- [31] . 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019. 3980–3990.Google Scholar
Cross Ref
- [32] . 2018. Modeling relational data with graph convolutional networks. In The Semantic Web - 15th International Conference, ESWC 2018. 593–607.Google Scholar
Cross Ref
- [33] . 2019. End-to-end structure-aware convolutional networks for knowledge base completion. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019. 3060–3067.Google Scholar
Digital Library
- [34] . 2015. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015.Google Scholar
- [35] . 2007. YAGO: A core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web, WWW 2007. 697–706.Google Scholar
Digital Library
- [36] . 2020. Multi-modal knowledge graphs for recommender systems. In CIKM’20: The 29th ACM International Conference on Information and Knowledge Management. 1405–1414.Google Scholar
Digital Library
- [37] . 2019. RotatE: Knowledge graph embedding by relational rotation in complex space. In 7th International Conference on Learning Representations, ICLR 2019.Google Scholar
- [38] . 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, CVSC 2015. 57–66.Google Scholar
Cross Ref
- [39] . 2016. Complex embeddings for simple link prediction. In Proceedings of the 33rd International Conference on Machine Learning, ICML 2016. 2071–2080.Google Scholar
- [40] . 2020. Composition-based multi-relational graph convolutional networks. In 8th International Conference on Learning Representations, ICLR 2020.Google Scholar
- [41] . 2014. Wikidata: A free collaborative knowledgebase. Commun. ACM 57, 10 (2014), 78–85.Google Scholar
Digital Library
- [42] . 2021. STAR: A benchmark for situated reasoning in real-world videos. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021).Google Scholar
- [43] . 2016. TransG : A generative model for knowledge graph embedding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Volume 1: Long Papers.Google Scholar
Cross Ref
- [44] . 2017. Image-embodied knowledge representation learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017. 3140–3146.Google Scholar
Cross Ref
- [45] . 2015. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations, ICLR 2015.Google Scholar
- [46] . 2019. KG-BERT: BERT for knowledge graph completion. CoRR abs/1909.03193 (2019).Google Scholar
- [47] . 2020. CLEVRER: Collision events for video representation and reasoning. In 8th International Conference on Learning Representations, ICLR 2020.Google Scholar
- [48] . 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017. 1103–1114.Google Scholar
Cross Ref
- [49] . 2021. Learning hyperbolic attention-based embeddings for link prediction in knowledge graphs. Knowl. Based Syst. 229 (2021), 107369.Google Scholar
Digital Library
- [50] . 2020. Multi-modal multi-relational feature aggregation network for medical knowledge representation learning. In MM’20: The 28th ACM International Conference on Multimedia. 3956–3965.Google Scholar
Digital Library
- [51] . 2019. Multi-modal knowledge-aware hierarchical attention network for explainable medical question answering. In Proceedings of the 27th ACM International Conference on Multimedia, MM 2019. 1089–1097.Google Scholar
Digital Library
- [52] . 2022. Multi-scale dynamic convolutional network for knowledge graph embedding. IEEE Trans. Knowl. Data Eng. 34, 5 (2022), 2335–2347.Google Scholar
Cross Ref
- [53] . 2020. DGL-KE: Training knowledge graph embeddings at scale. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020. 739–748.Google Scholar
Digital Library
- [54] . 2022. JointE: Jointly utilizing 1D and 2D convolution for knowledge graph embedding. Knowl. Based Syst. 240 (2022), 108100.Google Scholar
Digital Library
Index Terms
Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion
Recommendations
Knowledge Graph Completion Based on Multi-Relation Graph Attention Network
ICBDE '22: Proceedings of the 5th International Conference on Big Data and EducationKnowledge Graph Completion (KGC), can be performed mainly by inferring missing facts from entities and relations already in the knowledge graphs. However, most methods for KGC only focus on modeling undirected or single relational graph data, ignoring ...
Multi-relational knowledge graph completion method with local information fusion
AbstractKnowledge graph completion(KGC) has attracted increasing attention in recent years, aiming at complementing missing relationships between entities in a Knowledge Graph(KG). While the existing KGC approaches utilizing the knowledge within KG could ...
Multi-relational graph attention networks for knowledge graph completion
AbstractKnowledge graphs are multi-relational data that contain massive entities and relations. As an effective graph representation technique based on deep learning, graph neural network has reported outstanding performance for modeling ...






Comments