Abstract
The recent success of neural networks in NLP applications has provided a strong impetus to develop supervised models for semantic role labeling (SRL) that forego the requirement for extensive feature engineering. Recent state-of-the-art approaches require high-quality annotated datasets that are costly to obtain and almost unavailable for low-resource languages. We present a semi-supervised approach that utilizes both labeled and unlabeled data to provide performance improvement over a mere supervised SRL model. We show that our proposed semi-supervised SRL model provides larger improvement over a supervised model in the scenario where labeled training data size is small. Our SRL system leverages unlabeled data under the language modeling paradigm. We demonstrate that the incorporation of a self pre-trained bidirectional language model (S-PrLM) into a SRL system can help in SRL performance improvement by learning composition functions from the unlabeled data. Previous researches have concluded that syntax information is very useful for high-performing SRL systems, so we incorporate syntax information by employing an unsupervised approach to leverage dependency path information to connect argument candidates in vector space, which helps in distinguishing arguments with similar contexts but different syntactic functions. The basic idea is to connect predicate (wp) with argument candidate (wa) with the dependency path (r) between them in the embedding space. Experiments on the CoNLL-2008 and CoNLL-2009 datasets confirm that our full SRL model outperforms previous best models in terms of F1 score.
- . 2018. A combination of objective functions and hybrid krill herd algorithm for text document clustering analysis. Engineering Applications of Artificial Intelligence 73 (2018), 111–125.Google Scholar
Cross Ref
- . 2019. Feature Selection and Enhanced Krill Herd Algorithm for Text Document Clustering. Springer.Google Scholar
Cross Ref
- . 2009. A complex network approach to text summarization. Information Sciences 179, 5 (2009), 584–599. Google Scholar
Digital Library
- . 2003. A neural probabilistic language model. Journal of Machine Learning Research 3, (Feb. 2003), 1137–1155. Google Scholar
Cross Ref
- . 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP’13). 1533–1544.Google Scholar
- . 2010. A high-performance syntactic and semantic dependency parser. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING’10). 33–36.Google Scholar
- . 2009. Multilingual semantic role labeling. In Proceedings of the 13th Conference on Computational Natural Language Learning - Shared Task (CoNLL’09). 43–48.Google Scholar
Digital Library
- . 2011. Learning structured embeddings of knowledge bases. In Proceedings of the 25th AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- . 2016. Neural word segmentation learning for Chinese. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16) (Volume 1: Long Papers). 409–420.Google Scholar
Cross Ref
- . 2018. A full end-to-end semantic role labeler, syntax-agnostic over syntax-aware?. In Proceedings of the 27th International Conference on Computational Linguistics (COLING’18). 2753–2765.Google Scholar
- . 2019. Semi-supervised semantic role labeling with cross-view training. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 1017–1026.Google Scholar
Cross Ref
- . 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv:1312.3005. https://arxiv.org/abs/1312.3005Google Scholar
- . 2019. Gated recurrent neural network with sentimental relations for sentiment classification. Information Sciences 502 (2019), 268–278. Google Scholar
Digital Library
- . 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics 4 (2016), 357–370.Google Scholar
Cross Ref
- . 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP’18).Google Scholar
Cross Ref
- . 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. https://arxiv.org/abs/1810.04805.Google Scholar
- . 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL’14). 1370–1380.Google Scholar
Cross Ref
- . 2015. Domain adaptation in semantic role labeling using a neural language model and linguistic resources. IEEE/ACM Transactions on Audio, Speech, and Language Processing 23, 11 (2015), 1812–1823.Google Scholar
Digital Library
- . 2016. Deep biaffine attention for neural dependency parsing. arXiv:1611.01734. https://arxiv.org/abs/1611.01734.Google Scholar
- . 2015. Semantic role labeling with neural network factors. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP’15). 960–970.Google Scholar
Cross Ref
- . 2015. Dependency-based semantic role labeling using convolutional neural networks. In Proceedings of the 4th Joint Conference on Lexical and Computational Semantics. 279–288.Google Scholar
Cross Ref
- . 2012. Bidirectional language model for handwriting recognition. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). Springer, 611–619.Google Scholar
- . 2012. Semi-supervised semantic role labeling via structural alignment. Computational Linguistics 38, 1 (2012), 135–171. Google Scholar
Digital Library
- . 2020. Semantic relation extraction using sequential and tree-structured LSTM with attention. Information Sciences 509 (2020), 183–192. Google Scholar
Digital Library
- . 2002. Automatic labeling of semantic roles. Computational Linguistics 28, 3 (2002), 245–288.Google Scholar
Digital Library
- . 2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the 13th Conference on Computational Natural Language Learning - Shared Task (CoNLL’09). 1–18.Google Scholar
Cross Ref
- . 2017. Deep semantic role labeling: What works and what’s next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL’17). 473–483.Google Scholar
Cross Ref
- . 2018. Syntax for semantic role labeling, to be, or not to be. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL’18) (Volume 1: Long Papers). 2061–2071.Google Scholar
Cross Ref
- . 1997. Long short-term memory. Neural Computation 9, 8 (1997), 1735–1780. Google Scholar
Digital Library
- . 2008. Dependency-based syntactic–semantic analysis with PropBank and NomBank. In Proceedings of the 12th Conference on Computational Natural Language Learning (CoNLL’08). 183–187.Google Scholar
Cross Ref
- . 2016. Exploring the limits of language modeling. arXiv:1602.02410. https://arxiv.org/abs/1602.02410.Google Scholar
- . 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP’13). 1700–1709.Google Scholar
- . 2009. Statistical Machine Translation. Cambridge University Press. Google Scholar
Cross Ref
- . 2011. Recurrent neural network based language modeling in meeting recognition. In Proceedings of the 12th Annual Conference of the International Speech Communication Association (ISCA’11).Google Scholar
Cross Ref
- . 2018. Deep neural networks for bot detection. Information Sciences 467 (2018), 312–322. Google Scholar
Cross Ref
- . 2011a. Unsupervised semantic role induction via split-merge clustering. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL:HLT). 1117–1126.Google Scholar
Digital Library
- . 2011b. Unsupervised semantic role induction with graph partitioning. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP’11). 1320–1331.Google Scholar
- . 2015. High-order low-rank tensors for semantic role labeling. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL:HLT). 1150–1160.Google Scholar
Cross Ref
- . 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL’14) (Volume 2: Short Papers). 302–308.Google Scholar
Cross Ref
- . 2009. Improving nominal SRL in Chinese language with verbal SRL information and automatic predicate recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP’09). 1280–1288.Google Scholar
Cross Ref
- . 2018a. Seq2seq dependency parsing. In Proceedings of the 27th International Conference on Computational Linguistics (CoNLL’18). 3203–3214.Google Scholar
- . 2018b. A unified syntax-aware framework for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP’18). 2401–2411.Google Scholar
Cross Ref
- . 2019. Dependency or span, end-to-end uniform semantic role labeling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 6730–6737.Google Scholar
Digital Library
- . 2016. Multiplicative representations for unsupervised semantic role induction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16) (Volume 2: Short Papers). 118–123.Google Scholar
Cross Ref
- . 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL’17).Google Scholar
Cross Ref
- . 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP’17). 1506–1515.Google Scholar
Cross Ref
- . 2018. Towards semi-supervised learning for deep semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP’18). 4958–4963.Google Scholar
Cross Ref
- . 2013a. Efficient estimation of word representations in vector space. arXiv:1301.3781. https://arxiv.org/abs/1301.3781.Google Scholar
- . 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of Advances in Neural Information Processing Systems (NIPS’13). 3111–3119.Google Scholar
- . 2015. Dependency recurrent neural language models for sentence completion. arXiv:1507.01193. https://arxiv.org/abs/1507.01193.Google Scholar
- . 2021a. Adaptive convolution for semantic role labeling. In IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 29. IEEE, 782–791.Google Scholar
- . 2021b. Neural unsupervised semantic role labeling. Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 20, 6 (2021), 1–16.Google Scholar
Digital Library
- . 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th Annual Meeting of the International Conference on Machine Learning (ICML’10).Google Scholar
- . 2007. Semantic passage segmentation based on sentence topics for question answering. Information Sciences 177, 18 (2007), 3696–3717. Google Scholar
Digital Library
- . 2015. A bidirectional recurrent neural language model for machine translation. Procesamiento del Lenguaje Natural55 (2015), 109–116.Google Scholar
- . 2017. Semi-supervised sequence tagging with bidirectional language models. arXiv:1705.00108. https://arxiv.org/abs/1705.00108.Google Scholar
- . 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL: HLT).Google Scholar
Cross Ref
- . 2005. Semantic role labeling using different syntactic views. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL’05). 581–588.Google Scholar
Digital Library
- . 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics 34, 2 (2008), 257–287. Google Scholar
Digital Library
- . 2017. Syntax aware LSTM model for semantic role labeling. In Proceedings of the 2nd Workshop on Structured Prediction for Natural Language Processing. 27–32.Google Scholar
Cross Ref
- . 2016. Implicit discourse relation recognition with context-aware character-enhanced embeddings. In Proceedings of the 26th International Conference on Computational Linguistics (COLING’16). 1914–1924.Google Scholar
- . 2016. Neural semantic role labeling with dependency path embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16). 1192–1202.Google Scholar
Cross Ref
- . 2016. Knowledge-based semantic embedding for machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16) (Volume 1: Long Papers). 2245–2254.Google Scholar
Cross Ref
- . 2019. Simple BERT models for relation extraction and semantic role labeling. arXiv:1904.05255. https://arxiv.org/abs/1904.05255.Google Scholar
- . 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 2: Short Papers). 231–235.Google Scholar
Cross Ref
- . 2003. Using predicate-argument structures for information extraction. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL’03). 8–15.Google Scholar
Digital Library
- . 2008. The CoNLL 2008 shared task on joint parsing of syntactic and semantic dependencies. In Proceedings of the 12th Conference on Computational Natural Language Learning - Shared Task (CoNLL’08). 159–177.Google Scholar
Cross Ref
- . 2012. Semi-supervised semantic role labeling: Approaching from an unsupervised perspective. In Proceedings of the 24th International Conference on Computational Linguistics (COLING’12). 2635–2652.Google Scholar
- . 2016. Connecting phrase based statistical machine translation adaptation. In Proceedings of the 26th International Conference on Computational Linguistics (COLING’16). 3135–3145.Google Scholar
- . 2010. A short text modeling method combining semantic and statistical information. Information Sciences 180, 20 (2010), 4031–4041. Google Scholar
Digital Library
- . 2006. Emotion recognition from text using semantic labels and separable mixture models. ACM Transactions on Asian Language Information Pprocessing (TALIP) 5, 2 (2006), 165–183. Google Scholar
Digital Library
- . 2014. SRRank: Leveraging semantic roles for extractive multi-document summarization. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22, 12 (2014), 2048–2058. Google Scholar
Digital Library
- . 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Proceedings of the Advances in Neural Information Processing Systems (NIPS’19). 5753–5763.Google Scholar
- . 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. arXiv:1703.06345. https://arxiv.org/abs/1703.06345.Google Scholar
- . 2016. Unsupervised word and dependency path embeddings for aspect term extraction. arXiv:1605.07843. https://arxiv.org/abs/1605.07843.Google Scholar
- . 2019. Three-way enhanced convolutional neural networks for sentence-level sentiment classification. Information Sciences 477 (2019), 55–64. Google Scholar
Cross Ref
- . 2018. Subword-augmented embedding for cloze reading comprehension. In Proceedings of the 27th International Conference on Computational Linguistics (COLING’18). 1802–1814.Google Scholar
- . 2016. Probabilistic graph-based dependency parsing with convolutional neural network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16) (Volume 1: Long Papers). 1382–1392.Google Scholar
Cross Ref
- . 2009a. Semantic dependency parsing of NomBank and PropBank: An efficient integrated approach via a large-scale feature selection. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP’09). 30–39.Google Scholar
Cross Ref
- . 2009b. Multilingual dependency learning: A huge feature engineering method to semantic dependency parsing. In Proceedings of the 13th Conference on Computational Natural Language Learning - Shared Task (CoNLL’09). 55–60.Google Scholar
Cross Ref
- Hai Zhao, Wenliang Chen, Jun’ichi Kazama, Kiyotaka Uchimoto, and Kentaro Torisawa. 2009. Multilingual dependency learning: Exploiting rich features for tagging syntactic and semantic dependencies. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task, Association for Computational Linguistics, Boulder, CO, 61–66. https://aclanthology.org/W09-1209.Google Scholar
- . 2013. Integrative semantic dependency parsing via efficient large-scale feature selection. Journal of Artificial Intelligence Research 46 (2013), 203–233. Google Scholar
Cross Ref
- . 2020. Parsing all: Syntax and semantics, dependencies and spans. In Findings of the Association for Computational Linguistics: EMNLP 2020. 4438–4449.Google Scholar
- . 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL’15). 1127–1137.Google Scholar
Cross Ref
Index Terms
Semi-Supervised Semantic Role Labeling with Bidirectional Language Models
Recommendations
Neural Unsupervised Semantic Role Labeling
The task of semantic role labeling (SRL) is dedicated to finding the predicate-argument structure. Previous works on SRL are mostly supervised and do not consider the difficulty in labeling each example which can be very expensive and time-consuming. In ...
Semi-supervised multi-label classification using incomplete label information
Highlights- An inductive semi-supervised method called Smile is proposed for multi-label classification using incomplete label information.
AbstractClassifying multi-label instances using incompletely labeled instances is one of the fundamental tasks in multi-label learning. Most existing methods regard this task as supervised weak-label learning problem and assume sufficient ...
Semi-supervised partial label learning algorithm via reliable label propagation
AbstractPartial label learning (PLL) is a weakly supervised learning method that is able to predict one label as the correct answer from a given candidate label set. In PLL, when all possible candidate labels are as signed to real-world training examples, ...






Comments