Abstract
The Chinese radiology report summarization is a crucial component in smart healthcare that employs language models to summarize key findings in radiology reports and communicate these findings to physicians. However, most language models for radiology report summarization utilize a softmax transformation in their output layer, leading to dense alignments and strictly positive output probabilities. This density is inefficient, reducing model interpretability and giving probability mass to many unrealistic outputs. To tackle this issue, we propose a novel approach named nucleusmax. Nucleusmax is able to mitigate dense outputs and improve model interpretability by truncating the unreliable tail of the probability distribution. In addition, we incorporate nucleusmax with a copy mechanism, a useful technique to avoid professional errors in the generated diagnostic opinions. To further promote the research of radiology report summarization, we also have created a Chinese radiology report summarization dataset, which is freely available. Experimental results showed via both automatic and human evaluation that the proposed approach substantially improves the sparsity and overall quality of outputs over competitive softmax models, producing radiology summaries that approach the quality of those authored by physicians. In general, our work demonstrates the feasibility and prospect of the language model to the domain of radiology and smart healthcare.
- [1] . 2021. Overview of the MEDIQA 2021 shared task on summarization in the medical domain. In Proceedings of the 20th Workshop on Biomedical Language Processing.Google Scholar
Cross Ref
- [2] . 2021. What’s in a summary? Laying the groundwork for advances in hospital-course summarization. In Proc. of NAACL.Google Scholar
- [3] . 2003. Latent Dirichlet allocation. The Journal of Machine Learning Research (2003).Google Scholar
Digital Library
- [4] . 2020. Learning with Fenchel-Young losses. J. Mach. Learn. Res. (2020).Google Scholar
- [5] . 1990. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing.Google Scholar
- [6] . 2020. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical Image Analysis (2020).Google Scholar
Cross Ref
- [7] . 2021. ChestXRayBERT: A pretrained language model for chest radiology report summarization. IEEE Transactions on Multimedia (2021).Google Scholar
- [8] . 2015. Microsoft COCO captions: Data collection and evaluation server. IEEE Conference on Computer Vision and Pattern Recognition (2015).Google Scholar
- [9] . 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of ACL.Google Scholar
Cross Ref
- [10] . 2020. Generating radiology reports via memory-driven transformer. In Proc. of EMNLP.Google Scholar
- [11] . 2020. Meshed-memory transformer for image captioning. In Proc. of CVPR.Google Scholar
- [12] . 2020. Revisiting pre-trained models for Chinese natural language processing. In Proc. of EMNLP.Google Scholar
- [13] . 2020. A dataset of chest X-ray reports annotated with spatial role labeling annotations. Data in Brief (2020).Google Scholar
Cross Ref
- [14] . 2016. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association (2016).Google Scholar
Cross Ref
- [15] . 2009. ImageNet: A large-scale hierarchical image database. In Proc. of CVPR.Google Scholar
- [16] . 2019. Arcface: Additive angular margin loss for deep face recognition. In Proc. of CVPR.Google Scholar
- [17] . 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL.Google Scholar
- [18] . 2019. Unified language model pre-training for natural language understanding and generation. In Proc. of ICONIP.Google Scholar
- [19] . 2022. Use of natural language processing (NLP) in evaluation of radiology reports: An update on applications and technology advances. In Seminars in Ultrasound, CT and MRI.Google Scholar
- [20] . 2022. UGDAS: Unsupervised graph-network based denoiser for abstractive summarization in biomedical domain. (2022).Google Scholar
- [21] . 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization.Google Scholar
Cross Ref
- [22] . 2019. The curious case of neural text degeneration. In Proc. of ICLR.Google Scholar
- [23] . 2015. LCSTS: A large scale Chinese short text summarization dataset. In Proc. of EMNLP.Google Scholar
- [24] . 2022. Graph enhanced contrastive learning for radiology findings summarization. In Proc. of ACL.Google Scholar
- [25] . 2020. PAN: Pipeline assisted neural networks model for data-to-text generation in social Internet of Things. Information Sciences (2020).Google Scholar
Cross Ref
- [26] . 2019. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data (2019).Google Scholar
Cross Ref
- [27] . 2022. Differentiable multi-agent actor-critic for multi-step radiology report summarization. In Proc. of ACL.Google Scholar
- [28] . 2022. Methods for automatic generation of radiological reports of chest radiographs: A comprehensive survey. (2022).Google Scholar
- [29] . 2015. Adam: A method for stochastic optimization. In ICLR (Poster).Google Scholar
- [30] . 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proc. of NAACL.Google Scholar
- [31] . 2021. Auto-encoding knowledge graph for unsupervised medical report generation. Proc. of NeurIPS (2021).Google Scholar
- [32] . 2017. Sphereface: Deep hypersphere embedding for face recognition. In Proc. of CVPR.Google Scholar
- [33] . 2016. Large-margin softmax loss for convolutional neural networks. In Proc. of ICML.Google Scholar
- [34] . 2021. On the copying behaviors of pre-training for neural machine translation. In Proc. of ACL Findings.Google Scholar
- [35] . 2019. Fine-tune BERT for extractive summarization. arXiv preprint arXiv:1903.10318 (2019).Google Scholar
- [36] . 2017. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In Proc. of CVPR.Google Scholar
- [37] . 2020. Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles. In Proc. of EMNLP.Google Scholar
- [38] . 2019. Ontology-aware clinical abstractive summarization. In Proc. of SIGIR.Google Scholar
- [39] . 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proc. of ICML.Google Scholar
- [40] . 2020. Sparse text generation. In Proc. of EMNLP.Google Scholar
- [41] . 2004. TextRank: Bringing order into text. In Proc. of EMNLP.Google Scholar
- [42] . 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proc. of CoNLL.Google Scholar
- [43] . 2017. A regularized framework for sparse and structured neural attention. In Proc. of ICONIP.Google Scholar
- [44] . 2018. SparseMAP: Differentiable sparse structured inference. In Proc. of ICML.Google Scholar
- [45] . 2013. Reporting guidance for oncologic 18F-FDG PET/CT imaging. Journal of Nuclear Medicine (2013).Google Scholar
Cross Ref
- [46] . 2002. Bleu: A method for automatic evaluation of machine translation. In Proc. of ACL.Google Scholar
- [47] . 2019. Sparse sequence-to-sequence models. In Proc. of ACL.Google Scholar
- [48] . 2014. Zipf’s word frequency law in natural language: A critical review and future directions. (2014).Google Scholar
- [49] . 2020. On extractive and abstractive neural document summarization with transformer language models. In Proc. of EMNLP.Google Scholar
- [50] . 2017. Self-critical sequence training for image captioning. In Proc. of CVPR.Google Scholar
- [51] . 2017. Get to the point: Summarization with pointer-generator networks. In Proc. of ACL.Google Scholar
- [52] . 2020. SPACES: “Extractive-abstractive” Long Text Summaries.
Technical Report .Google Scholar - [53] . 2021. Sparse-softmax: A simpler and faster alternative softmax transformation. (2021).Google Scholar
- [54] . 2017. Attention is all you need. In Proc. of NeurIPS.Google Scholar
- [55] . 2015. Pointer networks. In Proc. of ICONIP.Google Scholar
- [56] . 2015. Show and tell: A neural image caption generator. In Proc. of CVPR.Google Scholar
- [57] . 2018. CosFace: Large margin cosine loss for deep face recognition. In Proc. of CVPR.Google Scholar
- [58] . 2018. TieNet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In Proc. of CVPR.Google Scholar
- [59] . 2019. NEZHA: Neural contextualized representation for Chinese language understanding. arXiv preprint arXiv:1909.00204 (2019).Google Scholar
- [60] . 2018. Pay less attention with lightweight and dynamic convolutions. In Proc. of ICLR.Google Scholar
- [61] . 2020. Copy or rewrite: Hybrid summarization with hierarchical reinforcement learning. In Proc. of AAAI.Google Scholar
- [62] . 2021. ChicHealth@ MEDIQA 2021: Exploring the limits of pre-trained seq2seq models for medical summarization. In Proceedings of the 20th Workshop on Biomedical Language Processing.Google Scholar
Cross Ref
- [63] . 2022. Question-aware transformer models for consumer health question summarization. Journal of Biomedical Informatics (2022).Google Scholar
Digital Library
- [64] . 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In Proc. of ICLR.Google Scholar
- [65] . 2021. Sparse attention with linear units. (2021).Google Scholar
- [66] . 2020. Conceptualized representation learning for Chinese biomedical text mining. (2020).Google Scholar
- [67] . 2018. Learning to summarize radiology findings. In Proceedings of the 9th International Workshop on Health Text Mining and Information Analysis.Google Scholar
Cross Ref
- [68] . 2020. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In Proc. of ACL.Google Scholar
- [69] . 2020. Bridging the structural gap between encoding and decoding for data-to-text generation. In Proc. of ACL.Google Scholar
- [70] . 2022. Sparsing and smoothing for the seq2seq models. IEEE Transactions on Artificial Intelligence (2022).Google Scholar
- [71] . 2022. AP-BERT: Enhanced pre-trained model through average pooling. Applied Intelligence (2022).Google Scholar
- [72] . 2019. The standards for PET/CT diagnostic reports: Setting and exploring. Labeled Immunoassays and Clinical Medicine (2019).Google Scholar
Index Terms
From Softmax to Nucleusmax: A Novel Sparse Language Model for Chinese Radiology Report Summarization
Recommendations
ChestXRayBERT: A Pretrained Language Model for Chest Radiology Report Summarization
Automatically generating the “impression” section of a radiology report given the “findings” section can summarize as much salient information of the “findings” section as possible, thus promoting more effective ...
Hybrid multi-document summarization using pre-trained language models
AbstractAbstractive multi-document summarization is a type of automatic text summarization. It obtains information from multiple documents and generates a human-like summary from them. In this paper, we propose an abstractive multi-document ...
Highlights- Introducing a multi-document summarizer, called HMSumm, based on pre-trained methods.
Methodical Systematic Review of Abstractive Summarization and Natural Language Processing Models for Biomedical Health Informatics: Approaches, Metrics and Challenges
Text summarization tasks are primarily very useful for decision support systems and provide a source for useful data for training of bots as they can reduce and retain the useful information from the large corpus. This review article is for studying the ...






Comments