skip to main content
research-article

Building Arabic Paraphrasing Benchmark based on Transformation Rules

Published:09 June 2021Publication History
Skip Abstract Section

Abstract

Measuring semantic similarity between short texts is an important task in many applications of natural language processing, such as paraphrasing identification. This process requires a benchmark of sentence pairs that are labeled by Arab linguists and considered a standard that can be used by researchers when evaluating their results. This research describes an Arabic paraphrasing benchmark to be a good standard for evaluation algorithms that are developed to measure semantic similarity for Arabic sentences to detect paraphrasing in the same language. The transformed sentences are in accordance with a set of rules for Arabic paraphrasing. These sentences are constructed from the words in the Arabic word semantic similarity dataset and from different Arabic books, educational texts, and lexicons. The proposed benchmark consists of 1,010 sentence pairs wherein each pair is tagged with scores determining semantic similarity and paraphrasing. The quality of the data is assessed using statistical analysis for the distribution of the sentences over the Arabic transformation rules and exploration through hierarchical clustering (HCL). Our exploration using HCL shows that the sentences in the proposed benchmark are grouped into 27 clusters representing different subjects. The inter-annotator agreement measures show a moderate agreement for the annotations of the graduate students and a poor reliability for the annotations of the undergraduate students.

References

  1. V. Vaishnavi, Madhesh Saritha, and S. Milton Rajendram. 2013. Paraphrase identification in short texts using grammar patterns. In 2013 International Conference on Recent Trends in Information Technology (ICRTIT). 472–477.Google ScholarGoogle Scholar
  2. Samuel Fernando and Mark Stevenson. 2008. A semantic similarity approach to paraphrase detection. In 11th Annual Research Colloquium of the UK Special Interest Group for Computational Linguistics.Google ScholarGoogle Scholar
  3. Peter W. Culicover. 1968. Paraphrase generation and information retrieval from stored text. Mechanical Translation and Computational Linguistics 11, 1 and 2 (1968), 78–88.Google ScholarGoogle Scholar
  4. Ngoc Phuoc An Vo, Simone Magnolini, and Octavian Popescu. 2015. Paraphrase identification and semantic similarity in Twitter with simple features. In International Workshop on Natural Language Processing for Social Media (SocialNLP’15), 10–19.Google ScholarGoogle Scholar
  5. Salha Alzahrani. 2016. Cross-language semantic similarity of Arabic-English short phrases and sentences. Journal of Computer Sciences 12, 1 (2016), 1–18.Google ScholarGoogle Scholar
  6. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 Task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In the 9th International Workshop on Semantic Evaluation (SemEval’15) (Denver, CO 2015). 252–263. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Marwah Alian and Arafat Awajan. 2018. Semantic similarity approaches- review. In 2018 International Arab Conference on Information Technology (ACIT’18) (Werdanye, Lebanon, 2018), 1–6.Google ScholarGoogle ScholarCross RefCross Ref
  8. James O'Shea, Zuhair Bandar, Keeley Crockett, and David McLean. 2008. Benchmarking short text semantic similarity. International Journal of Intelligent Information and Database Systems 4, 2 (2008), 103–120. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Bill Dolan, Chris Brockett, and Chris Quirk. 2005. Microsoft Research Paraphrase Corpus. (March 2005). Microsoft Research.Google ScholarGoogle Scholar
  10. Wafa Wali, Bilel Gargouri, and Abdelmajid Ben Hamadou. 2017. Enhancing the sentence similarity measure by semantic and syntactico-semantic knowledge. Vietnam Journal of Computer Science 4 (2017). 51–60. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Daniel Cera, Mona Diabb, Eneko Agirrec, Iñigo Lopez-Gazpio, and Lucia Speciad. 2017. SemEval-2017 Task 1: Semantic textual similarity multilingual and cross-lingual focused evaluation. (Canada 2017). 11th International Workshop on Semantic Evaluation (SemEval-2017).Google ScholarGoogle ScholarCross RefCross Ref
  12. Ali AlJarem and Mustafa Ameen. 2004. Clear grammer of Arabic language—AlnHw AlwADH fy qwAEd AllgAh AlErbyAh. Al-Dar Almysria Alsuadia for Publishing.Google ScholarGoogle Scholar
  13. Ahmad Mukhtar Omar. 1998. Semantics. Elm AldlAlAh. Book World. Qairo.Google ScholarGoogle Scholar
  14. Mohammad AlKholi. 2001. Semantics. Elm AldlAlAh (Elm AlmEnY). Dar Al-falah. Amman.Google ScholarGoogle Scholar
  15. Ahmad M. Omar and others. 1999. Language and grammar exercises. AltdrybAt AllgwyAh wAlqwAEd. Kuwait University—Art Collage.Google ScholarGoogle Scholar
  16. Faaza A. Almarsoomi, James D. O'shea, Zuhair Bandar, and Keeley Crockett. 2013. AWSS: An algorithm for measuring Arabic word semantic similarity. In 2013 IEEE International Conference on Systems, Man, and Cybernetics. 504–509. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Mohammad AlKholi. 1999. Transformation rules for Arabic language. qwAEd tHwylyAh llgAh AlErbyAh. Dar Al-Falah. Amman.Google ScholarGoogle Scholar
  18. Noam Chomsky. 1957. Syntactic Structure. Mouton Publishers, The Hague, Paris.Google ScholarGoogle Scholar
  19. Abdel Haleem Benaissa. 2011. Transfer Grammar in Arabic Phrase. Dar Al-Kotob Al-Ilmiyah, Lebanon.Google ScholarGoogle Scholar
  20. Abu Bakr Soliman Mohammad, Kareem Eissa, and Samhaa R. El-Beltagy. 2017. AraVec: A set of Arabic word embedding models for use in Arabic NLP. Procedia Computer Science 117, (2017) 256–265.Google ScholarGoogle Scholar
  21. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 5 (1971), 378.Google ScholarGoogle ScholarCross RefCross Ref
  22. Andrew F. Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 1 (2007), 77–89.Google ScholarGoogle ScholarCross RefCross Ref
  23. Jinyuan Liu, Wan Tang, Guanqin Chen, Yin Lu, Changyong Feng, and Xin M Tu. 2016. Correlation and agreement: Overview and clarification of competing concepts and measures. Shanghai Arch Psychiatry 28, 2 (2016), 115–120.Google ScholarGoogle Scholar
  24. Adrian Sanborn and Jacek Skryzalin. 2015. Deep learning for semantic similarity. CS224d: Deep Learning for Natural Language Processing. Stanford, CA: Stanford University.Google ScholarGoogle Scholar
  25. Yuhua Li, David McLean, Zuhair Bandar, James Dominic O'Shea, and Keeley Crockett. 2006. Sentence similarity based on semantic nets and corpus statistics. IEEE Transactions on Knowledge and Data Engineering 18, 8 (2006), 1138–1150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Marwah Alian, Arafat Awajan, Ahmad Al-Hasan, and Raeda Akuzhia. 2019. Towards building Arabic paraphrasing benchmark. In Proceedings of the 2nd International Conference on Data Science, E-Learning and Information Systems. (2019). Article No. 17. 1–5. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Joel R. Brandt, Jiayi Chong, and Sean Rosenbaum. 2006. Interactive Clustering for Data Exploration. Stanford University, Stanford, CA.Google ScholarGoogle Scholar
  28. Amir Ben-Dor, Ron Shamir, and Zohar Yakhini. 1999. Clustering gene expression patterns. Journal of Computational Biology. 6 (3/4). 281–297.Google ScholarGoogle Scholar
  29. Marwah Alian and Arafat Awajan. 2020. Factors affecting sentence similarity and paraphrasing identification. International Journal of Speech Technology 23, 851–859. https://doi.org/10.1007/s10772-020-09753-4Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Building Arabic Paraphrasing Benchmark based on Transformation Rules

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!