The week's pick
Reseach Article
A Rule based Stemming Method for Multilingual Urdu Text
| International Journal of Computer Applications |
| Foundation of Computer Science (FCS), NY, USA |
| Volume 134 - Number 8 |
| Year of Publication: 2016 |
| Authors: Mubashir Ali, Shehzad Khalid, M. Haneef Saleemi, Waheed Iqbal, Armughan Ali, Ghayur Naqvi |
10.5120/ijca2016907784
|
Mubashir Ali, Shehzad Khalid, M. Haneef Saleemi, Waheed Iqbal, Armughan Ali, Ghayur Naqvi . A Rule based Stemming Method for Multilingual Urdu Text. International Journal of Computer Applications. 134, 8 ( January 2016), 10-18. DOI=10.5120/ijca2016907784
Abstract
Urdu is a national language of Pakistan and spoken more than 200 million people use it as a verbal and written communication. There exists a large amount of unstructured Urdu textual data in the world; by applying data mining techniques useful information can be achieved. However it seriously lacks processing capabilities to develop innovative systems based on Urdu language. In this paper, authors present a rule based stemming method for Urdu language that has the ability to cope the challenges of Urdu infix stemming. The proposed stemming method generates the stem of Urdu words by removing prefix, infix and postfix from it. In this proposed Urdu stemming technique, authors have introduced two novel classes of Urdu infix words and a new minimum word length rule. To generate stem of Urdu word that belongs to proposed Urdu infix word classes, infix stripping rules are developed. The proposed Urdu stemming technique is competent to generate the stem of borrowed words and compound words, as well. The proposed approach is evaluated on Urdu headline news datasets. This proposed approach is compared with existing state-of-the art technique (A Light Weight Urdu Stemmer) to demonstrate the effectiveness of the proposed method. The proposed method provides 90% to 95 % accuracy and shows significant improvements comparing to the Urdu stemming technique.
References
- Bowman, Q. Akram, A. Naseer and S. Hussain. Assas-band, an affix- exception-list based Urdu stemmer. Proceedings of the 7th Workshop on Asia Language Resources. Singapore. pages 40–47. (2009).
- M. Al-Khuli. A dictionary of theoretical linguistics: English-Arabic with an Arabic- English glossary. Published by Library of Lebanon. (1991).
- K. Riaz. Challenges in Urdu Stemming (A Progress Report). BCS IRSG Symposium: Future Directions in Information Access (FDIA). (2007).
- S. Ahmad, W. Anwar, U.I. Bajwa. Challenges in Developing a Rule based Urdu Stemmer. Proceedings of the 2nd Workshop on South and Southeast Asian Natural Language Processing (WSSANLP). Chiang Mai, Thailand. pages 46–51. (2011).
- J. B. Lovins. Development of a stemming algorithm. Mechanical Translation and Computer Linguistic. vol.11, no.1/2, pp. 22-31, (1968).
- D.C. Paice. Another stemmer. ACM SIGIR Forum. Volume 24, No. 3: 56-61. (1990).
- M.F. Porter. An algorithm for suffix stripping. Program. 14: 130-137. (1980).
- M.F. Porter. Snowball: A language for stemming algorithms. (2001).
- N. Thabet, Stemming the Qur’an. In the Proceedings of the Workshop on Computational Approaches to Arabic Script-based Languages. pages 85-88. (2004).
- M. Tashakori, M. Meybodi & F. Oroumchian. Bon: first Persian stemmer. Lecture Notes on Information and Communication Technology. pages 487-494. (2002).
- S. Ahmad, W. Anwar, U.I. Bajwa, X. Wang. A Light Weight Stemmer for Urdu Language: A Scarce Resourced Language. Proceedings of the 3rd Workshop on South and Southeast Asian Natural Language Processing (SANLP). Mumbai. pages 69–78. (2012).
- Mayfield James and McNamee Paul. “Single Ngram stemming”. Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval. 415-416. (2003).
- Melucci Massimo and Orio Nicola. “A novel method for stemmer generation based on hidden Markov models”. Proceedings of the twelfth international conference on Information and knowledge management. 131-138. (2003).
- Prasenjit Majumder, Mandar Mitra, Swapan K. Parui, Gobinda Kole, Pabitra Mitra and Kalyankumar Datta. “YASS: Yet another suffix stripper”. ACM Transactions on Information Systems. Volume 25, Issue 4. Article No. 18. (2007).
- Hussain, Sara. Finite-State Morphological Analyzer for Urdu. Unpublished MS thesis, Center for Research in Urdu Language Processing, National University of Computer and Emerging Sciences, Pakistan. (2004).
- Sabzwari, S. Urdu Quwaid. Sang-e-Meel Publication. (2002).
- M. Ali, S. Khalid, M.H. Saleemi. A Novel Stemming Approach for Urdu language. Journal of Applied Environmental and Biological Sciences. 4(7S) 436-443, (2014).
Index Terms
Keywords