skip to main content
research-article

Contrastive Learning between Classical and Modern Chinese for Classical Chinese Machine Reading Comprehension

Published:27 December 2022Publication History
Skip Abstract Section

Abstract

By leveraging self-supervised tasks, pre-trained language model (PLM) has made significant progress in the field of machine reading comprehension (MRC). However, in classical Chinese MRC (CCMRC), the passage is typically in classical style, but the question and options are given in modern style. Existing pre-trained methods seldom model the relationship between classical and modern styles, resulting in overall misunderstanding of the passage. In this paper, we propose a contrastive learning method between classical and modern Chinese in order to reach a deep understanding of the two different styles. In particular, a novel pre-training task and an enhanced co-matching network have been defined: (1) The synonym discrimination (SD) task is used to identify whether modern meaning corresponds to classical Chinese. (2) The enhanced dual co-matching (EDCM) network is employed for a more interactive understanding of the classical passage and the modern options. The experimental results show that our proposed method improves language understanding ability and outperforms existing PLMs on the Haihua, CCLUE, and ChID datasets.

REFERENCES

  1. [1] Clark Kevin, Luong Minh-Thang, Le Quoc V., and Manning Christopher D.. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In Proceedings of the 8th International Conference on Learning Representations (ICLR’20). https://openreview.net/forum?id=r1xMH1BtvB.Google ScholarGoogle Scholar
  2. [2] Cui Yiming, Che Wanxiang, Liu Ting, Qin Bing, and Yang Ziqing. 2021. Pre-training with whole word masking for Chinese BERT. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021), 35043514. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Cui Yiming, Che Wanxiang, Yang Ziqing, Liu Ting, Qin Bing, Wang Shijin, and Hu Guoping. 2022. Interactive gated decoder for machine reading comprehension. ACM Transactions on Asian and Low-Resource Language Information Processing 21, 4, Article 72 (2022), 19 pages. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. [4] Cui Yiming, Liu Ting, Che Wanxiang, Xiao Li, Chen Zhipeng, Ma Wentao, Wang Shijin, and Hu Guoping. 2019. A span-extraction dataset for Chinese machine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 58835889. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Dahlkamp Hendrik, Kaehler Adrian, Stavens David, Thrun Sebastian, and Bradski Gary R.. 2006. Self-supervised monocular road detection in desert terrain. In Robotics: Science and Systems, Vol. 38.Google ScholarGoogle Scholar
  6. [6] Dawson Raymond and Dawson Raymond Stanley. 1984. A new introduction to Classical Chinese. Oxford University Press.Google ScholarGoogle Scholar
  7. [7] Devlin Jacob, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Minneapolis, Minnesota, 41714186. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Dong Li, Yang Nan, Wang Wenhui, Wei Furu, Liu Xiaodong, Wang Yu, Gao Jianfeng, Zhou Ming, and Hon Hsiao-Wuen. 2019. Unified language model pre-training for natural language understanding and generation. Advances in Neural Information Processing Systems 32 (2019).Google ScholarGoogle Scholar
  9. [9] He Wei, Liu Kai, Liu Jing, Lyu Yajuan, Zhao Shiqi, Xiao Xinyan, Liu Yuan, Wang Yizhong, Wu Hua, She Qiaoqiao, Liu Xuan, Wu Tian, and Wang Haifeng. 2018. DuReader: A Chinese machine reading comprehension dataset from real-world applications. In Proceedings of the Workshop on Machine Reading for Question Answering. Association for Computational Linguistics, Melbourne, Australia, 3746. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Howard Jeremy and Ruder Sebastian. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Melbourne, Australia, 328339. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Joshi Mandar, Chen Danqi, Liu Yinhan, Weld Daniel S., Zettlemoyer Luke, and Levy Omer. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics 8 (2020), 6477. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Lai Guokun, Xie Qizhe, Liu Hanxiao, Yang Yiming, and Hovy Eduard. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, 785794. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Lan Zhenzhong, Chen Mingda, Goodman Sebastian, Gimpel Kevin, Sharma Piyush, and Soricut Radu. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In Proceedings of the 8th International Conference on Learning Representations (ICLR’20). https://openreview.net/forum?id=H1eA7AEtvS.Google ScholarGoogle Scholar
  14. [14] Lewis Mike, Liu Yinhan, Goyal Naman, Ghazvininejad Marjan, Mohamed Abdelrahman, Levy Omer, Stoyanov Veselin, and Zettlemoyer Luke. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 78717880. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Lin Xuming, Liu Ruifang, and Li Yiwei. 2018. An option gate module for sentence inference on machine reading comprehension. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM’18). Association for Computing Machinery, 17431746. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Liu Kai, Liu Xin, Yang An, Liu Jing, Su Jinsong, Li Sujian, and She Qiaoqiao. 2020. A robust adversarial training approach to machine reading comprehension. Proceedings of the 34th AAAI Conference on Artificial Intelligence 34, 5 (2020), 83928400. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Liu Shanshan, Zhang Xin, Zhang Sheng, Wang Hui, and Zhang Weiming. 2019. Neural machine reading comprehension: Methods and trends. Applied Sciences 9, 18 (2019), 3698. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Liu Weijie, Zhou Peng, Zhao Zhe, Wang Zhiruo, Ju Qi, Deng Haotang, and Wang Ping. 2020. K-BERT: Enabling language representation with knowledge graph. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. 29012908.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Liu Xin, Liu Kai, Li Xiang, Su Jinsong, Ge Yubin, Wang Bin, and Luo Jiebo. 2020. An iterative multi-source mutual knowledge transfer framework for machine reading comprehension. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI’20). 37943800. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Liu Yinhan, Ott Myle, Goyal Naman, Du Jingfei, Joshi Mandar, Chen Danqi, Levy Omer, Lewis Mike, Zettlemoyer Luke, and Stoyanov Veselin. 2019. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv preprint abs/1907.11692 (2019).Google ScholarGoogle Scholar
  21. [21] McCloskey Michael and Cohen Neal J.. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, Vol. 24. Academic Press, 109165. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Miao Hang, Liu Ruifang, and Gao Sheng. 2019. A multiple granularity co-reasoning model for multi-choice reading comprehension. In 2019 International Joint Conference on Neural Networks (IJCNN). 17. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Nguyen Tri, Rosenberg Mir, Song Xia, Gao Jianfeng, Tiwary Saurabh, Majumder Rangan, and Deng Li. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPS.Google ScholarGoogle Scholar
  24. [24] Pan Xiao, Wang Mingxuan, Wu Liwei, and Li Lei. 2021. Contrastive learning for many-to-many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, Online, 244258. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Pang Liang, Lan Yanyan, Guo Jiafeng, Xu Jun, Wan Shengxian, and Cheng Xueqi. 2016. Text matching as image recognition. In Proceedings of the 30th AAAI Conference on Artificial Intelligence. 27932799.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Peng Jiajia and Han Kaixu. 2021. Survey of pre-trained models for natural language processing. In 2021 International Conference on Electronic Communications, Internet of Things and Big Data (ICEIB). 277280. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Qin Yujia, Lin Yankai, Takanobu Ryuichi, Liu Zhiyuan, Li Peng, Ji Heng, Huang Minlie, Sun Maosong, and Zhou Jie. 2021. ERICA: Improving entity and relation understanding for pre-trained language models via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, Online, 33503363. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Qiu Xipeng, Sun Tianxiang, Xu Yige, Shao Yunfan, Dai Ning, and Huang Xuanjing. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences 63, 10 (2020), 18721897.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Rajpurkar Pranav, Jia Robin, and Liang Percy. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Melbourne, Australia, 784789. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Ren Feiliang, Liu Yongkang, Li Bochao, Wang Zhibo, Guo Yu, Liu Shilei, Wu Huimin, Wang Jiaqi, Liu Chunchao, and Wang Bingchao. 2022. Deep understanding based multi-document machine reading comprehension. ACM Transactions on Asian and Low-Resource Language Information Processing 21, 5, Article 108 (2022), 21 pages. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Song Kaitao, Tan Xu, Qin Tao, Lu Jianfeng, and Liu Tie-Yan. 2019. MASS: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning (ICML). 59265936.Google ScholarGoogle Scholar
  32. [32] Sun Kai, Yu Dian, Yu Dong, and Cardie Claire. 2020. Investigating prior knowledge for challenging Chinese machine reading comprehension. Transactions of the Association for Computational Linguistics 8 (2020), 141155. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Sun Yu, Wang Shuohuan, Li Yukun, Feng Shikun, Chen Xuyi, Zhang Han, Tian Xin, Zhu Danxiang, Tian Hao, and Wu Hua. 2019. ERNIE: Enhanced representation through knowledge integration. ArXiv preprint abs/1904.09223 (2019).Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] Taylor Wilson L.. 1953. “Cloze procedure”: A new tool for measuring readability. Journalism Quarterly 30, 4 (1953), 415433.Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Kaiser Ł ukasz, and Polosukhin Illia. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, I. Guyon, Luxburg U. Von, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc., 59986008.Google ScholarGoogle Scholar
  36. [36] Wang Dong, Ding Ning, Li Piji, and Zheng Haitao. 2021. CLINE: Contrastive learning with semantic negative examples for natural language understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, Online, 23322342. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Wang Dongbo, Liu Chang, Zhu Zihe, Liu Jiangfeng, Hu Haotian, Shen Si, and Bin Li. 2021. SikuBERT and SikuRoBERTa: Research on the construction and application of the pre-training model of Sikuquanshu for digital humanities. Library Forum (2021), 114.Google ScholarGoogle Scholar
  38. [38] Wang Shuohang, Yu Mo, Jiang Jing, and Chang Shiyu. 2018. A co-matching model for multi-choice reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Melbourne, Australia, 746751. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Wang Ziqi, Wang Xiaozhi, Han Xu, Lin Yankai, Hou Lei, Liu Zhiyuan, Li Peng, Li Juanzi, and Zhou Jie. 2021. CLEVE: Contrastive pre-training for event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, Online, 62836297. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Xie Qizhe, Lai Guokun, Dai Zihang, and Hovy Eduard. 2018. Large-scale Cloze test dataset created by teachers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 23442356. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Xu Liang, Hu Hai, Zhang Xuanwei, Li Lu, Cao Chenjie, Li Yudong, Xu Yechen, Sun Kai, Yu Dian, Yu Cong, Tian Yin, Dong Qianqian, Liu Weitang, Shi Bo, Cui Yiming, Li Junyi, Zeng Jun, Wang Rongzhao, Xie Weijian, Li Yanting, Patterson Yina, Tian Zuoyu, Zhang Yiwen, Zhou He, Liu Shaoweihua, Zhao Zhe, Zhao Qipeng, Yue Cong, Zhang Xinrui, Yang Zhengliang, Richardson Kyle, and Lan Zhenzhong. 2020. CLUE: A Chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Barcelona, Spain (Online), 47624772. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Xu Shusheng, Liu Yichen, Yi Xiaoyu, Zhou Siyuan, Li Huizi, and Wu Yi. 2021. Native Chinese reader: A dataset towards native-level Chinese machine reading comprehension. arXiv preprint arXiv:2112.06494 (2021).Google ScholarGoogle Scholar
  43. [43] Xu Yichong, Liu Jingjing, Gao Jianfeng, Shen Yelong, and Liu Xiaodong. 2017. Dynamic fusion networks for machine reading comprehension. ArXiv preprint abs/1711.04964 (2017). https://arxiv.org/abs/1711.04964.Google ScholarGoogle Scholar
  44. [44] Yang Zhilin, Dai Zihang, Yang Yiming, Carbonell Jaime, Salakhutdinov Russ R., and Le Quoc V.. 2019. XLNet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems 32 (2019).Google ScholarGoogle Scholar
  45. [45] Zhang Shuailiang, Zhao Hai, Wu Yuwei, Zhang Zhuosheng, Zhou Xi, and Zhou Xiang. 2020. DCMN+: Dual co-matching network for multi-choice reading comprehension. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, Vol. 34. 95639570. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Zheng Chujie, Huang Minlie, and Sun Aixin. 2019. ChID: A large-scale Chinese IDiom dataset for Cloze test. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 778787. DOI:Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Contrastive Learning between Classical and Modern Chinese for Classical Chinese Machine Reading Comprehension

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Asian and Low-Resource Language Information Processing
          ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 22, Issue 2
          February 2023
          624 pages
          ISSN:2375-4699
          EISSN:2375-4702
          DOI:10.1145/3572719
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 27 December 2022
          • Online AM: 5 August 2022
          • Accepted: 21 July 2022
          • Received: 14 June 2022
          • Revised: 27 December 2021
          Published in tallip Volume 22, Issue 2

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed
        • Article Metrics

          • Downloads (Last 12 months)158
          • Downloads (Last 6 weeks)15

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!