skip to main content
research-article

Radial Basis Function Attention for Named Entity Recognition

Published:30 November 2022Publication History
Skip Abstract Section

Abstract

Attention mechanism is an increasingly important approach in the field of natural language processing (NLP). In the attention-based named entity recognition (NER) model, most attention mechanisms can calculate attention coefficient to express the importance of sentence semantic information but cannot adjust the position distribution of contextual feature vectors in the semantic space. To address this issue, a radial basis function attention (RBF-attention) layer is proposed to adaptively regulate the position distribution of sequence contextual feature vectors, which can minimize the relative distance of within-category named entities and maximize the relative distance of between-category named entities in the semantic space. The experimental results on CoNLL2003 English and MSRA Chinese NER datasets indicate that the proposed model performs better than other baseline approaches without relying on any external feature engineering.

REFERENCES

  1. [1] Nadeau D. and Sekine S.. 2007. A survey of named entity recognition and classification. In Lingvisticae Investigationes. 30, 1 (2007), 326.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Bastianelli E., Castellucci G., Croce D., and Basili R.. 2013. Textual inference and meaning representation in human robot interaction. In Joint Symposium on Semantic Processing Textual Inference and Structures in Corpora. (2013), 6569.Google ScholarGoogle Scholar
  3. [3] Mendsaikhan O., Hasegawa H., Yukiko Y., and Shimada H.. 2020. Quantifying the significance of cybersecurity text through semantic similarity and named entity recognition. In 6th International Conference on Information Systems Security and Privacy (ICISP’20). 325332.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Yao X. and Durme B. V.. 2014. Information extraction over structured data: Question answering with freebase. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL’14). 956966.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Kuru O., Can O. A., and Yuret D., 2016. CharNER: Character-level named entity recognition. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (COLING’16). 911921.Google ScholarGoogle Scholar
  6. [6] Lample G., Ballesteros M., Subramanian S., Kawakami, K. and Dyer C.. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’16). 260270.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Zhao D., Huang J., and Jia Y.. 2018. Chinese name entity recognition using highway-LSTM-CRF. In Proceedings of the 2018 International Conference on Algorithms, Computing and Artificial Intelligence (ACAI’2018). Association for Computing Machinery, 15. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Chiu J. P. and Nichols E.. 2016. Named entity recognition with bidirectional LSTM-CNNs. In Transactions of the Association for Computational Linguistics (2016), 4, 2016, 357370.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Tan Z., Wang M., Xie J., Chen Y., and Shi X.. 2018. Deep semantic role labeling with self-attention. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI'18). 49294936.Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Wei W., Wang Z., Mao X., Zhou G., and Jiang S.. 2021. Position-aware self-attention based neural sequence labeling. Pattern Recognition 110, 1 (2021), 107636.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Liu M., Zhang Y., Li W., and Ji D.. 2020. Joint model of entity recognition and relation extraction with self-attention mechanism. ACM Trans. Asian Low-Resour. Lang. (2020). DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Li J., Shang S., and Chen L.. 2020. Domain generalization for named entity boundary detection via meta-learning. In IEEE Transactions on Neural Networks and Learning Systems 32, 9 (2020), 38193830.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Gu L., Han Y., Wang C., Wei C., Jun J., and Yuan X.. 2018. Module overlapping structure detection in PPI using an improved link similarity-based Markov clustering algorithm. In Neural Computing & Applications. 31, 5 (2019), 14811490.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Alokaili A. and Menai M. E. B.. 2020. SVM ensembles for named entity disambiguation. In Computing. 102, 4 (2020), 10511076.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Lu Y., Zhang Y., and Ji D.. 2016. Multi-prototype Chinese character embedding. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). 855859.Google ScholarGoogle Scholar
  16. [16] Cheng J., Dong L., and Lapata M.. 2016. Long short-term memory-networks for machine reading. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP’16). 551561.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A. N., Kaiser L., and Polosukhin I.. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). 59996009.Google ScholarGoogle Scholar
  18. [18] Lin Z., Feng M., Santos C. N. D., Yu M., Xiang, B. and Zhou B.. 2017. A structured self-attentive sentence embedding. In Proceedings of the 6th International Conference on Learning Representations (ICLR’17).Google ScholarGoogle Scholar
  19. [19] Shen T., Zhou T., Long G., Jiang J., Pan S., and Zhang C.. 2018. DiSAN: Directional Self-attention network for RNN/CNN-free language understanding. In Proceeding of the 32nd AAAI Conference on Artificial Intelligence (AAAI’18). 54465455.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Phan M. C., Sun A., Tay Y., Han J., and Li C.. 2017. NeuPL: Attention-based semantic matching and pair-linking for entity disambiguation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM’17). 16671676.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Liu L., Shang J., Ren X., F. Xu F., Gui H., Peng J., and Han J.. 2018. Empower sequence labeling with task-aware neural language model. In Proceeding of the 32nd AAAI Conference on Artificial Intelligence (AAAI’18).Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Peters M. E., Ammar W., Bhagavatula C., and Power R.. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL’17).Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Wang J., Xu W., Fu X., Xu G., and Wu Y.. 2020. ASTRAL: Adversarial trained LSTM-CNN for named entity recognition. Knowledge-Based Systems. (2020) 105842.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Li J., Han P., Ren X., Hu L., Chen L., and Shang S.. 2021. Sequence labeling with meta-learning. In IEEE Transactions on Knowledge and Data Engineering.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Dong C., Zhang J., Zong C., Hattori M., and Di H.. 2016. Character-based LSTM-CRF with radical-level features for Chinese named entity recognition. In Natural Language Understanding and Intelligent Applications (2016). 239250.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Xu C., Wang F., Han, J. and Li C.. 2019. Exploiting multiple embeddings for Chinese named entity recognition. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM’19). 22692272.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] Li J., Chiu B., Feng S., and Wang H.. 2020. Few-shot named entity recognition via meta-learning. In IEEE Transactions on Knowledge and Data Engineering.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Wu Y., Wei X., Qin Y., and Chen Y.. 2019. A radical-based method for Chinese named entity recognition. In Proceedings of the 2nd International Conference on Big Data Technologies (ICBDT2019). Association for Computing Machinery, 125130. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Zhang X., Zhao R., Qiao Y., and Li H.. 2020. RBF-Softmax: Learning deep representative prototypes with radial basis function softmax. In Vedaldi A., Bischof H., Brox T., Frahm J. M. (eds). Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science, vol 12371. Springer, Cham. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Abdulhameed T. Z., Zitouni I., and Abdel-Qader I.. 2018. Enhancement of the Word2vec class-based language modeling by optimizing the features vector using PCA. IEEE International Conference on Electro/Information Technology.Google ScholarGoogle Scholar
  31. [31] Mikolov T., Sutskever I., Chen K., Corrado G., and Dean J.. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NIPS’13). 31113119.Google ScholarGoogle Scholar
  32. [32] Murtaza M., Sharif M., Raza M., and Shah J.. 2014. Face recognition using adaptive margin Fisher's criterion and linear discriminant analysis. International Arab Journal of Information Technology 11, 2 (2014), 111.Google ScholarGoogle Scholar
  33. [33] Zhao Z., Chen Z., Liu J., Huang Y., and Ji X.. 2019. Chinese named entity recognition in power domain based on Bi-LSTM-CRF. In Proceedings of the 2nd International Conference on Artificial Intelligence and Pattern Recognition (AIPR’19). Association for Computing Machinery, 176180. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Ma X. and Hovy E.. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16). 10641074.Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Yadav V., Sharp R., and Bethard S.. 2018. Deep affix features improve neural named entity recognizers. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. 167172.Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Bai Y., Wang Y., Xia B., Li Y., and Zhu Z.. 2020. Adversarial named entity recognition with POS label embedding. In 2020 International Joint Conference on Neural Networks (IJCNN'20). IEEE, 18.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Ronran C., Lee S., and Jang H. J.. 2020. Delayed combination of feature embedding in bidirectional LSTM CRF for NER. In Applied Sciences (2020).Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Lester B., Pressel D., Hemmeter A., and Choudhury S. R.. 2020. Computationally efficient NER taggers with combined embeddings and constrained decoding. (2020). arXiv preprint arXiv:2001.01167v1.Google ScholarGoogle Scholar
  39. [39] Cao P., Chen Y., Liu K., Zhao J., and Liu S.. 2018. Adversarial transfer learning for Chinese named entity recognition with self-attention mechanism. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP’18). 182192.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Zhang N., Li F., Xu G., Zhang W., and Yu H.. 2019. Chinese NER using dynamic meta-e mbeddings. In IEEE Access 7 (2019), 6445064459.Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Wang C., Chen W., and Xu B.. 2017. Named entity recognition with gated convolutional neural networks. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data. 110121.Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Zhu Y., Wang G., and Karlsson B. F.. 2019. CAN-NER: Convolutional attention network for Chinese named entity recognition. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL-HLT'19).Google ScholarGoogle Scholar

Index Terms

  1. Radial Basis Function Attention for Named Entity Recognition

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Asian and Low-Resource Language Information Processing
      ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 22, Issue 1
      January 2023
      340 pages
      ISSN:2375-4699
      EISSN:2375-4702
      DOI:10.1145/3572718
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 30 November 2022
      • Online AM: 31 May 2022
      • Accepted: 11 May 2022
      • Revised: 9 May 2022
      • Received: 28 April 2021
      Published in tallip Volume 22, Issue 1

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed
    • Article Metrics

      • Downloads (Last 12 months)130
      • Downloads (Last 6 weeks)7

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!