Abstract
Measuring semantic similarity between short texts is an important task in many applications of natural language processing, such as paraphrasing identification. This process requires a benchmark of sentence pairs that are labeled by Arab linguists and considered a standard that can be used by researchers when evaluating their results. This research describes an Arabic paraphrasing benchmark to be a good standard for evaluation algorithms that are developed to measure semantic similarity for Arabic sentences to detect paraphrasing in the same language. The transformed sentences are in accordance with a set of rules for Arabic paraphrasing. These sentences are constructed from the words in the Arabic word semantic similarity dataset and from different Arabic books, educational texts, and lexicons. The proposed benchmark consists of 1,010 sentence pairs wherein each pair is tagged with scores determining semantic similarity and paraphrasing. The quality of the data is assessed using statistical analysis for the distribution of the sentences over the Arabic transformation rules and exploration through hierarchical clustering (HCL). Our exploration using HCL shows that the sentences in the proposed benchmark are grouped into 27 clusters representing different subjects. The inter-annotator agreement measures show a moderate agreement for the annotations of the graduate students and a poor reliability for the annotations of the undergraduate students.
- V. Vaishnavi, Madhesh Saritha, and S. Milton Rajendram. 2013. Paraphrase identification in short texts using grammar patterns. In 2013 International Conference on Recent Trends in Information Technology (ICRTIT). 472–477.Google Scholar
- Samuel Fernando and Mark Stevenson. 2008. A semantic similarity approach to paraphrase detection. In 11th Annual Research Colloquium of the UK Special Interest Group for Computational Linguistics.Google Scholar
- Peter W. Culicover. 1968. Paraphrase generation and information retrieval from stored text. Mechanical Translation and Computational Linguistics 11, 1 and 2 (1968), 78–88.Google Scholar
- Ngoc Phuoc An Vo, Simone Magnolini, and Octavian Popescu. 2015. Paraphrase identification and semantic similarity in Twitter with simple features. In International Workshop on Natural Language Processing for Social Media (SocialNLP’15), 10–19.Google Scholar
- Salha Alzahrani. 2016. Cross-language semantic similarity of Arabic-English short phrases and sentences. Journal of Computer Sciences 12, 1 (2016), 1–18.Google Scholar
- Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 Task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In the 9th International Workshop on Semantic Evaluation (SemEval’15) (Denver, CO 2015). 252–263. Google Scholar
Digital Library
- Marwah Alian and Arafat Awajan. 2018. Semantic similarity approaches- review. In 2018 International Arab Conference on Information Technology (ACIT’18) (Werdanye, Lebanon, 2018), 1–6.Google Scholar
Cross Ref
- James O'Shea, Zuhair Bandar, Keeley Crockett, and David McLean. 2008. Benchmarking short text semantic similarity. International Journal of Intelligent Information and Database Systems 4, 2 (2008), 103–120. Google Scholar
Digital Library
- Bill Dolan, Chris Brockett, and Chris Quirk. 2005. Microsoft Research Paraphrase Corpus. (March 2005). Microsoft Research.Google Scholar
- Wafa Wali, Bilel Gargouri, and Abdelmajid Ben Hamadou. 2017. Enhancing the sentence similarity measure by semantic and syntactico-semantic knowledge. Vietnam Journal of Computer Science 4 (2017). 51–60. Google Scholar
Digital Library
- Daniel Cera, Mona Diabb, Eneko Agirrec, Iñigo Lopez-Gazpio, and Lucia Speciad. 2017. SemEval-2017 Task 1: Semantic textual similarity multilingual and cross-lingual focused evaluation. (Canada 2017). 11th International Workshop on Semantic Evaluation (SemEval-2017).Google Scholar
Cross Ref
- Ali AlJarem and Mustafa Ameen. 2004. Clear grammer of Arabic language—AlnHw AlwADH fy qwAEd AllgAh AlErbyAh. Al-Dar Almysria Alsuadia for Publishing.Google Scholar
- Ahmad Mukhtar Omar. 1998. Semantics. Elm AldlAlAh. Book World. Qairo.Google Scholar
- Mohammad AlKholi. 2001. Semantics. Elm AldlAlAh (Elm AlmEnY). Dar Al-falah. Amman.Google Scholar
- Ahmad M. Omar and others. 1999. Language and grammar exercises. AltdrybAt AllgwyAh wAlqwAEd. Kuwait University—Art Collage.Google Scholar
- Faaza A. Almarsoomi, James D. O'shea, Zuhair Bandar, and Keeley Crockett. 2013. AWSS: An algorithm for measuring Arabic word semantic similarity. In 2013 IEEE International Conference on Systems, Man, and Cybernetics. 504–509. Google Scholar
Digital Library
- Mohammad AlKholi. 1999. Transformation rules for Arabic language. qwAEd tHwylyAh llgAh AlErbyAh. Dar Al-Falah. Amman.Google Scholar
- Noam Chomsky. 1957. Syntactic Structure. Mouton Publishers, The Hague, Paris.Google Scholar
- Abdel Haleem Benaissa. 2011. Transfer Grammar in Arabic Phrase. Dar Al-Kotob Al-Ilmiyah, Lebanon.Google Scholar
- Abu Bakr Soliman Mohammad, Kareem Eissa, and Samhaa R. El-Beltagy. 2017. AraVec: A set of Arabic word embedding models for use in Arabic NLP. Procedia Computer Science 117, (2017) 256–265.Google Scholar
- Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 5 (1971), 378.Google Scholar
Cross Ref
- Andrew F. Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 1 (2007), 77–89.Google Scholar
Cross Ref
- Jinyuan Liu, Wan Tang, Guanqin Chen, Yin Lu, Changyong Feng, and Xin M Tu. 2016. Correlation and agreement: Overview and clarification of competing concepts and measures. Shanghai Arch Psychiatry 28, 2 (2016), 115–120.Google Scholar
- Adrian Sanborn and Jacek Skryzalin. 2015. Deep learning for semantic similarity. CS224d: Deep Learning for Natural Language Processing. Stanford, CA: Stanford University.Google Scholar
- Yuhua Li, David McLean, Zuhair Bandar, James Dominic O'Shea, and Keeley Crockett. 2006. Sentence similarity based on semantic nets and corpus statistics. IEEE Transactions on Knowledge and Data Engineering 18, 8 (2006), 1138–1150. Google Scholar
Digital Library
- Marwah Alian, Arafat Awajan, Ahmad Al-Hasan, and Raeda Akuzhia. 2019. Towards building Arabic paraphrasing benchmark. In Proceedings of the 2nd International Conference on Data Science, E-Learning and Information Systems. (2019). Article No. 17. 1–5. Google Scholar
Digital Library
- Joel R. Brandt, Jiayi Chong, and Sean Rosenbaum. 2006. Interactive Clustering for Data Exploration. Stanford University, Stanford, CA.Google Scholar
- Amir Ben-Dor, Ron Shamir, and Zohar Yakhini. 1999. Clustering gene expression patterns. Journal of Computational Biology. 6 (3/4). 281–297.Google Scholar
- Marwah Alian and Arafat Awajan. 2020. Factors affecting sentence similarity and paraphrasing identification. International Journal of Speech Technology 23, 851–859. https://doi.org/10.1007/s10772-020-09753-4Google Scholar
Cross Ref
Index Terms
Building Arabic Paraphrasing Benchmark based on Transformation Rules
Recommendations
Towards building arabic paraphrasing benchmark
DATA '19: Proceedings of the Second International Conference on Data Science, E-Learning and Information SystemsThis research describes a paraphrasing benchmark of Arabic sentences for evaluating algorithms developed to measure Semantic Similarity for sentences and Paraphrasing identification task. The sentences are formed based on a set of rules for Arabic ...
Corpus annotation with paraphrase types: new annotation scheme and inter-annotator agreement measures
Paraphrase corpora annotated with the types of paraphrases they contain constitute an essential resource for the understanding of the phenomenon of paraphrasing and the improvement of paraphrase-related systems in natural language processing. In this ...
Noun Compound Interpretation Using Paraphrasing Verbs: Feasibility Study
AIMSA '08: Proceedings of the 13th international conference on Artificial Intelligence: Methodology, Systems, and ApplicationsThe paper addresses an important challenge for the automatic processing of English written text: understanding noun compounds' semantics. Following Downing (1977) [1], we define noun compounds as sequences of nouns acting as a single noun, e.g., <em>bee ...






Comments