The week's pick
Reseach Article
Enhancing NameNode Fault Tolerance in Hadoop Distributed File System
| International Journal of Computer Applications |
| Foundation of Computer Science (FCS), NY, USA |
| Volume 87 - Number 12 |
| Year of Publication: 2014 |
| Authors: Ohnmar Aung, Thandar Thein |
10.5120/15264-4020
|
Ohnmar Aung, Thandar Thein . Enhancing NameNode Fault Tolerance in Hadoop Distributed File System. International Journal of Computer Applications. 87, 12 ( February 2014), 41-47. DOI=10.5120/15264-4020
Abstract
In today's cloud computing environment, Hadoop is applied for handling huge data, tens of terabytes to petabytes, with commodity hardware (HDFS) for storage and software (MapReduce) for parallel data processing. In Hadoop version 1. 0. 3, there is a single metadata server called NameNode which stores the entire file system metadata in main memory and most of I/O operations are associated with those credential metadata. Hadoop is out of commission if NameNode is crashed because it works on memory which becomes exhausted due to multiple concurrent accesses [3]. Therefore, NameNode is a single point of failure (SPOF) in Hadoop and it has to tolerate faults. To solve this issue, a proactive predictive solution is proposed for enhancing NameNode fault tolerance. The solution is designed to proactively calculate the predicted time to crash of NameNode due to resource exhaustion by evaluating the use of powerful Back Propagation Algorithm Neural Network. The proposed approach can give prediction accuracy with minimal error compared to the actual result. Therefore, NameNode's single point of failure can overcome through proposed proactively predicting the time to crash of NameNode caused by memory resource exhaustion.
References
- Anil K. Jain, "Artificial Neural Networks: A Tutorial", in Proceedings of Neural Computing: Companion issue to Engineering, Vol. 29 Issue 3, March 1996, pp. 31-44
- Cristina L. Abad, Huong Luu, Nathan Roberts, Kihwal Lee, Yi Lu and Roy H. Campbell, "Metadata Traces and Workload Models for Evaluating Big Storage Systems", in Proceedings of IEEE 5th International Conference on Utility and Cloud Computing (UCC), Chicago, IL, November 5-8, 2012, pp. 125-132.
- Chuck Lam, "Hadoop in Action", Manning Publications Co. 180 Broad St. Suite 1323, Stamfor, CT 06901, December 22, 2010.
- Diane Hatcher, "Considerations for Implementing a Highly Available or Disaster Recovery Environment," SAS Institute Inc, Cary, NC, USA, 2011.
- Dhruba Borthakur, "Apache Hadoop and Its Usage in Facebook", UC Berkeley, April 4, 2011. Online Available : http://www. facebook. com/hadoopfs
- Eric Sammer, "Hadoop Operations", O'Reilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472, United States of America, September 9, 2012.
- Feng Wang, Jie Qiu, Jie Yang, "Hadoop High Availability through Metadata Replication", IBM Research, China, 2009.
- Javier Alonso and Jordi Torres, "Predicting Web Server Crashes: A Case Study in Comparing Prediction Algorithms", in Proceedings of 5th IEEE International Conference on Autonomic and Autonomous Systems (ICAS'09), Valencia, April 20-25, 2009, pp. 264-269.
- Javier Alonso Lopez, "Proactive Software Rejuvenation Solution for Web Environment on Virtualized Platforms," Doctoral thesis, Barcelona, Spain 2011.
- Jimmy Lin and Chris Dyer, "Data-Intensive Text Processing with MapReduce", University of Maryland, College Park, April 11, 2010.
- Roman Dudko, Abhishek Sharma, Jon Tedsco, "Effective Failure Prediction in Hadoop Clusters", March, 2012. Online Available: http://www. techrepublic. com/resource-library/whitepapers/effective-failure-prediction-in-hadoop-clusters/
- Simon Haykin, "Neural Network: A Comprehensive Foundation," Prentice Hall, Delhi, India, 1999.
- Tom White, "Hadoop: The Definitive Guide", O'Reilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472, United States of America, May 2012.
- Xiaojuan Ren, Seyong Lee, Rudolf Eigenmann, Saurabh Bagchi, "Prediction of Resource Availability in Fine-Grained Cycle Sharing Systems Empirical Evaluation", J Grid Computing (2007), Vol 5, pp 173-195.
Index Terms
Keywords