Abstract
Predicting what happens next in text plays a critical role in building NLP applications. Many methods including count-based and neural-network-based have been proposed to tackle the task called script event prediction: predicting the most suitable subsequent event from a candidate list given a chain of narrative events (context). However, two problems including event ambiguity and evidence bias hinder the performance of these monolingual approaches. The former means that some events in the event chain are ambiguous. The latter means that both the wrong and correct candidate events can obtain sufficient support from the event context. In this article, we propose a novel multilingual approach to address two issues simultaneously. Specifically, to alleviate the event ambiguity problem, we project the monolingual event chains to parallel cross-lingual event chains, which can provide complementary information for monolingual event disambiguation. To deal with the evidence bias problem, we construct two monolingual event graphs and a cross-lingual event aligned graph to fully explore connections between events. What’s more, we design a graph attention mechanism to model the confidence of the complement clues, which controls the information integration from various languages. By modeling the events with graphs instead of pairs or chains, the model can compare the candidate subsequent events simultaneously and choose the more suitable subsequent event as the final answer. Extensive experiments were conducted on the widely used New York Times corpus for script event prediction task and experimental results show that our approach outperforms previous models.
- [1] . 2019. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In Proceedings of the International Conference on Machine Learning. PMLR, 21–29.Google Scholar
- [2] . 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations.Google Scholar
- [3] . 2021. Mulan: Multilingual label propagation for word sense disambiguation. In Proceedings of the 29th International Conference on International Joint Conferences on Artificial Intelligence. 3837–3844.Google Scholar
- [4] . 2014. Spectral networks and deep locally connected networks on graphs. In Proceedings of the 2nd International Conference on Learning Representations.Google Scholar
- [5] . 2008. Unsupervised learning of narrative event chains. In Proceedings of the ACL-08: HLT. 789–797.Google Scholar
- [6] . 2007. Linguistically motivated large-scale NLP with C&C and Boxer. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions. 33–36.Google Scholar
Cross Ref
- [7] . 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the NIPS.Google Scholar
- [8] . 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171–4186.Google Scholar
- [9] . 2016. Knowledge-driven event embedding for stock prediction. In Proceedings of the Coling 2016, the 26th International Conference on Computational Linguistics: Technical Papers. 2133–2142.Google Scholar
- [10] . 2022. Improving event representation via simultaneous weakly supervised contrastive learning and clustering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 3036–3049.Google Scholar
Cross Ref
- [11] . 2020. Non-linear instance-based cross-lingual mapping for non-isomorphic embedding spaces. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 7548–7555.Google Scholar
Cross Ref
- [12] . 2016. What happens next? Event prediction using a compositional neural network model. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [13] . 2017. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 1025–1035.Google Scholar
Digital Library
- [14] . 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 3509–3514.Google Scholar
Cross Ref
- [15] . 2012. Skip n-grams and ranking functions for predicting script events. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. 336–344.Google Scholar
Digital Library
- [16] . 2015. Adam: A method for stochastic optimization. In Proceedings of the ICLR.Google Scholar
- [17] . 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR’17).Google Scholar
- [18] . 2018. Constructing narrative event evolutionary graph for script event prediction. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 4201–4207.Google Scholar
Digital Library
- [19] . 2017. Neural relation extraction with multi-lingual attention. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 34–43.Google Scholar
Cross Ref
- [20] . 2019. SAM-Net: Integrating event-level and chain-level attentions to predict what happens next. In Proceedings of the AAAI Conference on Artificial Intelligence. 6802–6809.Google Scholar
Digital Library
- [21] . 2020. Integrating external event knowledge for script learning. In Proceedings of the 28th International Conference on Computational Linguistics. 306–315.Google Scholar
Cross Ref
- [22] . 2021. Event prediction based on evolutionary event ontology knowledge. Future Generation Computer Systems 115 (2021), 76–89.Google Scholar
Cross Ref
- [23] . 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems. 3111–3119.Google Scholar
Digital Library
- [24] . 2019. DisSent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 4497–4510.Google Scholar
Cross Ref
- [25] . 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 701–710.Google Scholar
Digital Library
- [26] . 2014. Statistical script learning with multi-argument events. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. 220–229.Google Scholar
Cross Ref
- [27] . 2016. Learning statistical scripts with LSTM recurrent neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [28] . 2017. Graph attention networks[J]. stat 1050 (2017), 20.Google Scholar
- [29] . 2019. Sense vocabulary compression through the semantic knowledge of WordNet for neural word sense disambiguation. In Proceedings of the 10th Global Wordnet Conference. 108–117.Google Scholar
- [30] . 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. ACL Anthology, 5797–5808.Google Scholar
Cross Ref
- [31] . 2021. CLEVE: Contrastive pre-training for event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 6283–6297.Google Scholar
Cross Ref
- [32] . 2017. Integrating order information and event relation for script event prediction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 57–67.Google Scholar
Cross Ref
- [33] . 2019. DEMO-Net: Degree-specific graph neural networks for node and graph classification. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2019. Association for Computing Machinery, 406–415.Google Scholar
Digital Library
- [34] . 2018. Word attention for sequence to sequence text understanding. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [35] . 2018. Dialog generation using multi-turn reasoning neural networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2049–2059.Google Scholar
Cross Ref
- [36] . 2022. CoCoLM: Complex commonsense enhanced language model with discourse relations. In Proceedings of the Findings of the Association for Computational Linguistics: ACL 2022. 1175–1187.Google Scholar
Cross Ref
- [37] . 2018. Adaptive co-attention network for named entity recognition in tweets. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [38] . 2018. Neural coreference resolution with deep biaffine attention by joint mention detection and mention clustering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 102–107.Google Scholar
Cross Ref
- [39] . 2022. Eventbert: A pre-trained model for event correlation reasoning. In Proceedings of the ACM Web Conference 2022. 850–859.Google Scholar
Digital Library
- [40] . 2021. Modeling event-pair relations in external knowledge graphs for script reasoning. In Proceedings of the Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 4586–4596.Google Scholar
Cross Ref
- [41] . 2022. ClarET: Pre-training a correlation-aware context-to-event transformer for event-centric generation and classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2559–2575.Google Scholar
Cross Ref
- [42] . 2018. Hierarchical attention flow for multiple-choice reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
Index Terms
Script Event Prediction via Multilingual Event Graph Networks
Recommendations
Script event prediction based on pre-trained model with tail event enhancement
CSAI '21: Proceedings of the 2021 5th International Conference on Computer Science and Artificial IntelligenceScript event prediction is a big challenge and its goal is to predict the subsequent event based on the observed events. Since an event is described by text, the pre-trained models have been applied for event representation. However, the embedding based ...
Multi-level Connection Enhanced Representation Learning for Script Event Prediction
WWW '21: Proceedings of the Web Conference 2021Script event prediction (SEP) aims to choose a correct subsequent event from a candidate list, given a chain of ordered context events. Event representation learning has been proposed and successfully applied to this task. Most previous methods ...
Constructing narrative event evolutionary graph for script event prediction
IJCAI'18: Proceedings of the 27th International Joint Conference on Artificial IntelligenceScript event prediction requires a model to predict the subsequent event given an existing event context. Previous models based on event pairs or event chains cannot make full use of dense event connections, which may limit their capability of event ...






Comments