skip to main content
research-article

Deep Understanding Based Multi-Document Machine Reading Comprehension

Authors Info & Claims
Published:29 April 2022Publication History
Skip Abstract Section

Abstract

Most existing multi-document machine reading comprehension models mainly focus on understanding the interactions between the input question and documents, but ignore the following two kinds of understandings. First, to understand the semantic meaning of words in the input question and documents from the perspective of each other. Second, to understand the supporting cues for a correct answer from the perspective of intra-document and inter-documents. Ignoring these two kinds of important understandings would make the models overlook some important information that may be helpful for finding correct answers. To overcome this deficiency, we propose a deep understanding based model for multi-document machine reading comprehension. It has three cascaded deep understanding modules which are designed to understand the accurate semantic meaning of words, the interactions between the input question and documents, and the supporting cues for the correct answer. We evaluate our model on two large scale benchmark datasets, namely TriviaQA Web and DuReader. Extensive experiments show that our model achieves state-of-the-art results on both datasets.

REFERENCES

  1. Bahdanau Dzmitry, Cho Kyunghyun, and Bengio Yoshua. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015: International Conference on Learning Representations 2015.Google ScholarGoogle Scholar
  2. Banerjee Pratyay, Gokhale Tejas, and Baral Chitta. 2021. Self-supervised test-time learning for reading comprehension.arXiv preprint arXiv:2103.11263 (2021).Google ScholarGoogle Scholar
  3. Chen Wuya, Quan Xiaojun, Kit Chunyu, Min Zhengcheng, and Wang Jiahai. 2020. Multi-choice relational reasoning for machine reading comprehension. In Proceedings of the 28th International Conference on Computational Linguistics. 64486458.Google ScholarGoogle ScholarCross RefCross Ref
  4. Chen Zheng and Wu Kangjian. 2020. ForceReader: A BERT-based interactive machine reading comprehension model with attention separation. In Proceedings of the 28th International Conference on Computational Linguistics. 26762686.Google ScholarGoogle ScholarCross RefCross Ref
  5. Chen Zheqian, Yang Rongqin, Cao Bin, Zhao Zhou, Cai Deng, and He Xiaofei. 2017. Smarnet: Teaching machines to read and comprehend like human.arXiv preprint arXiv:1710.02772 (2017).Google ScholarGoogle Scholar
  6. Clark Christopher and Gardner Matt. 2018. Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 845855.Google ScholarGoogle ScholarCross RefCross Ref
  7. Cui Yiming, Chen Zhipeng, Wei Si, Wang Shijin, Liu Ting, and Hu Guoping. 2017. Attention-over-attention neural networks for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 593602.Google ScholarGoogle ScholarCross RefCross Ref
  8. Dai Yilin, Ji Qian, Liu Gongshen, and Su Bo. 2020. Multi-paragraph reading comprehension with token-level dynamic reader and hybrid verifier. In 2020 International Joint Conference on Neural Networks (IJCNN). 18.Google ScholarGoogle ScholarCross RefCross Ref
  9. Devlin Jacob, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina N.. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 41714186.Google ScholarGoogle Scholar
  10. Gong Chen, Li Zhenghua, Xia Qingrong, Chen Wenliang, and Zhang Min. 2020a. Hierarchical LSTM with char-subword-word tree-structure representation for Chinese named entity recognition. Science in China Series F: Information Sciences 63, 10 (2020), 202102.Google ScholarGoogle ScholarCross RefCross Ref
  11. Gong Hongyu, Shen Yelong, Yu Dian, Chen Jianshu, and Yu Dong. 2020b. Recurrent chunking mechanisms for long-text machine reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 67516761.Google ScholarGoogle ScholarCross RefCross Ref
  12. Guo Shaoru, Guan Yong, Li Ru, Li Xiaoli, and Tan Hongye. 2020a. Incorporating syntax and frame semantics in neural network for machine reading comprehension. In Proceedings of the 28th International Conference on Computational Linguistics. 26352641.Google ScholarGoogle ScholarCross RefCross Ref
  13. Guo Shaoru, Li Ru, Tan Hongye, Li Xiaoli, Guan Yong, Zhao Hongyan, and Zhang Yueping. 2020b. A frame-based sentence representation for machine reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 891896.Google ScholarGoogle ScholarCross RefCross Ref
  14. He Wei, Liu Kai, Liu Jing, Lyu Yajuan, Zhao Shiqi, Xiao Xinyan, Liu Yuan, Wang Yizhong, Wu Hua, She Qiaoqiao, Liu Xuan, Wu Tian, and Wang Haifeng. 2018. DuReader: A Chinese machine reading comprehension dataset from real-world applications. In Proceedings of the Workshop on Machine Reading for Question Answering. 3746.Google ScholarGoogle ScholarCross RefCross Ref
  15. Hu Minghao, Wei Furu, Peng Yuxing, Huang Zhen, Yang Nan, and Li Dongsheng. 2019. Read + verify: Machine reading comprehension with unanswerable questions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 65296537.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Huang Rongtao, Zou Bowei, Hong Yu, Zhang Wei, Aw Ai Ti, and Zhou Guodong. 2020. NUT-RC: Noisy user-generated text-oriented reading comprehension. In Proceedings of the 28th International Conference on Computational Linguistics. 26872698.Google ScholarGoogle ScholarCross RefCross Ref
  17. Joshi Mandar, Choi Eunsol, Weld Daniel S., and Zettlemoyer Luke. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 16011611.Google ScholarGoogle ScholarCross RefCross Ref
  18. Lan Zhenzhong, Chen Mingda, Goodman Sebastian, Gimpel Kevin, Sharma Piyush, and Soricut Radu. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR 2020: Eighth International Conference on Learning Representations.Google ScholarGoogle Scholar
  19. Li Dongfang, Hu Baotian, Chen Qingcai, Peng Weihua, and Wang Anqi. 2020b. Towards medical machine reading comprehension with structural knowledge and plain text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 14271438.Google ScholarGoogle ScholarCross RefCross Ref
  20. Li Hongyu, Chen Tengyang, Bai Shuting, Utsuro Takehito, and Kawada Yasuhide. 2020a. MRC examples answerable by BERT without a question are less effective in MRC model training. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop. 146152.Google ScholarGoogle Scholar
  21. Liu Xiaodong, Shen Yelong, Duh Kevin, and Gao Jianfeng. 2018. Stochastic answer networks for machine reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 16941704.Google ScholarGoogle ScholarCross RefCross Ref
  22. Liu Yinhan, Ott Myle, Goyal Naman, Du Jingfei, Joshi Mandar, Chen Danqi, Levy Omer, Lewis Mike, Zettlemoyer Luke, and Stoyanov Veselin. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019).Google ScholarGoogle Scholar
  23. Long Siyu, Wang Ran, Tao Kun, Zeng Jiali, and Dai Xinyu. 2020. Synonym knowledge enhanced reader for Chinese idiom reading comprehension. In Proceedings of the 28th International Conference on Computational Linguistics. 36843695.Google ScholarGoogle ScholarCross RefCross Ref
  24. Luo Huaishao, Shi Yu, Gong Ming, Shou Linjun, and Li Tianrui. 2020. MaP: A matrix-based prediction approach to improve span extraction in machine reading comprehension. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing. 687695.Google ScholarGoogle Scholar
  25. Malmaud Jonathan, Levy Roger, and Berzak Yevgeni. 2020. Bridging information-seeking human gaze and machine reading comprehension. In Proceedings of the 24th Conference on Computational Natural Language Learning. 142152.Google ScholarGoogle ScholarCross RefCross Ref
  26. Mihaylov Todor and Frank Anette. 2019. Discourse-aware semantic self-attention for narrative reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 25412552.Google ScholarGoogle ScholarCross RefCross Ref
  27. Nguyen Tri, Rosenberg Mir, Song Xia, Gao Jianfeng, Tiwary Saurabh, Majumder Rangan, and Deng Li. 2016. MS MARCO: A human generated MAchine reading COmprehension dataset. In [email protected].Google ScholarGoogle Scholar
  28. Nishida Kyosuke, Saito Itsumi, Nishida Kosuke, Shinoda Kazutoshi, Otsuka Atsushi, Asano Hisako, and Tomita Junji. 2019. Multi-style generative reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 22732284.Google ScholarGoogle ScholarCross RefCross Ref
  29. Peng Wei, Hu Yue, Xing Luxi, Xie Yuqiang, Yu Jing, Sun Yajing, and Wei Xiangpeng. 2020. Bi-directional cognitive thinking network for machine reading comprehension. In Proceedings of the 28th International Conference on Computational Linguistics.Google ScholarGoogle ScholarCross RefCross Ref
  30. Pennington Jeffrey, Socher Richard, and Manning Christopher. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 15321543.Google ScholarGoogle ScholarCross RefCross Ref
  31. Peters Matthew E., Neumann Mark, Iyyer Mohit, Gardner Matt, Clark Christopher, Lee Kenton, and Zettlemoyer Luke. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Vol. 1. 22272237.Google ScholarGoogle ScholarCross RefCross Ref
  32. Rajpurkar Pranav, Jia Robin, and Liang Percy. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vol. 2. 784789.Google ScholarGoogle ScholarCross RefCross Ref
  33. Seo Min Joon, Kembhavi Aniruddha, Farhadi Ali, and Hajishirzi Hannaneh. 2016. Bidirectional attention flow for machine comprehension. In ICLR (Poster).Google ScholarGoogle Scholar
  34. Sun Kai, Yu Dian, Yu Dong, and Cardie Claire. 2019. Improving machine reading comprehension with general reading strategies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 26332643.Google ScholarGoogle ScholarCross RefCross Ref
  35. Tian Zhixing, Zhang Yuanzhe, Liu Kang, Zhao Jun, Jia Yantao, and Sheng Zhicheng. 2020. Scene restoring for narrative machine reading comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 30633073.Google ScholarGoogle ScholarCross RefCross Ref
  36. Vinyals Oriol, Fortunato Meire, and Jaitly Navdeep. 2015. Pointer networks. In NIPS’15 Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, Vol. 28. 26922700.Google ScholarGoogle Scholar
  37. Wang Shuohang and Jiang Jing. 2017. Machine comprehension using match-LSTM and answer pointer. In ICLR (Poster). 1.Google ScholarGoogle Scholar
  38. Wang Wei, Yan Ming, and Wu Chen. 2018c. Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 17051714.Google ScholarGoogle ScholarCross RefCross Ref
  39. Wang Wenhui, Yang Nan, Wei Furu, Chang Baobao, and Zhou Ming. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 189198.Google ScholarGoogle ScholarCross RefCross Ref
  40. Wang Yizhong, Liu Kai, Liu Jing, He Wei, Lyu Yajuan, Wu Hua, Li Sujian, and Wang Haifeng. 2018a. Multi-passage machine reading comprehension with cross-passage answer verification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 19181927.Google ScholarGoogle ScholarCross RefCross Ref
  41. Wang Yucheng, Yu Bowen, Zhang Yueyang, Liu Tingwen, Zhu Hongsong, and Sun Limin. 2020. TPLinker: Single-stage joint extraction of entities and relations through token pair linking. In Proceedings of the 28th International Conference on Computational Linguistics. Barcelona, Spain (Online), 15721582.Google ScholarGoogle ScholarCross RefCross Ref
  42. Wang Zhen, Liu Jiachen, Xiao Xinyan, Lyu Yajuan, and Wu Tian. 2018b. Joint training of candidate extraction and answer selection for reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 17151724.Google ScholarGoogle ScholarCross RefCross Ref
  43. Wu Felix, Lao Ni, Blitzer John, Yang Guandao, and Weinberger Kilian Q.. 2018. Fast Reading Comprehension With ConvNets. arXiv preprint arXiv:1711.04352 (2018).Google ScholarGoogle Scholar
  44. Xu Yichong, Liu Xiaodong, Shen Yelong, Liu Jingjing, and Gao Jianfeng. 2019. Multi-task learning with sample re-weighting for machine reading comprehension. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 26442655.Google ScholarGoogle ScholarCross RefCross Ref
  45. Yan Ming, Xia Jiangnan, Wu Chen, Bi Bin, Zhao Zhongzhou, Zhang Ji, Si Luo, Wang Rui, Wang Wei, and Chen Haiqing. 2019. A deep cascade model for multi-document reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 73547361.Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Yang An, Wang Quan, Liu Jing, Liu Kai, Lyu Yajuan, Wu Hua, She Qiaoqiao, and Li Sujian. 2019b. Enhancing pre-trained language representations with rich knowledge for machine reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 23462357.Google ScholarGoogle ScholarCross RefCross Ref
  47. Yang Wei, Xie Yuqing, Lin Aileen, Li Xingyu, Tan Luchen, Xiong Kun, Li Ming, and Lin Jimmy. 2019c. End-to-end open-domain question answering with BERTserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). 7277.Google ScholarGoogle Scholar
  48. Yang Zhilin, Dai Zihang, Yang Yiming, Carbonell Jaime G., Salakhutdinov Ruslan, and Le Quoc V.. 2019a. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, Vol. 32. 57535763.Google ScholarGoogle Scholar
  49. Yu Adams Wei, Dohan David, Luong Minh-Thang, Zhao Rui, Chen Kai, Norouzi Mohammad, and Le Quoc V.. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In International Conference on Learning Representations.Google ScholarGoogle Scholar
  50. Zemlyanskiy Yury, Ainslie Joshua, Jong Michiel de, Pham Philip, Eckstein Ilya, and Sha Fei. 2021. ReadTwice: Reading very large documents with memories. arXiv preprint arXiv:2105.04241 (2021).Google ScholarGoogle Scholar
  51. Zhang Xuemiao, Zhou Kun, Wang Sirui, Zhang Fuzheng, Wang Zhongyuan, and Liu Junfei. 2020c. Learn with noisy data via unsupervised loss correction for weakly supervised reading comprehension. In Proceedings of the 28th International Conference on Computational Linguistics. 26242634.Google ScholarGoogle ScholarCross RefCross Ref
  52. Zhang Zhuosheng, Wu Yuwei, Zhao Hai, Li Zuchao, Zhang Shuailiang, Zhou Xi, and Zhou Xiang. 2020a. Semantics-aware BERT for language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 96289635.Google ScholarGoogle ScholarCross RefCross Ref
  53. Zhang Zhuosheng, Wu Yuwei, Zhou Junru, Duan Sufeng, Zhao Hai, and Wang Rui. 2020b. SG-Net: Syntax-guided machine reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 96369643.Google ScholarGoogle ScholarCross RefCross Ref
  54. Zhang Zhuosheng, Yang Junjie, and Zhao Hai. 2021. Retrospective reader for machine reading comprehension. In AAAI 2021.Google ScholarGoogle Scholar
  55. Zheng Bo, Wen Haoyang, Liang Yaobo, Duan Nan, Che Wanxiang, Jiang Daxin, Zhou Ming, and Liu Ting. 2020. Document modeling with graph attention networks for multi-grained machine reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 67086718.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Deep Understanding Based Multi-Document Machine Reading Comprehension

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Asian and Low-Resource Language Information Processing
      ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 21, Issue 5
      September 2022
      486 pages
      ISSN:2375-4699
      EISSN:2375-4702
      DOI:10.1145/3533669
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 29 April 2022
      • Online AM: 24 February 2022
      • Accepted: 1 February 2022
      • Revised: 1 November 2021
      • Received: 1 December 2020
      Published in tallip Volume 21, Issue 5

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!