Abstract
Many big-data processing jobs use data-analytics frameworks such as Apache Hadoop (currently also known as YARN). Such frameworks have tunable configuration parameters set by experienced system administrators and/or job developers. However, tuning parameters manually can be hard and time-consuming because it requires domain-specific knowledge and understanding of complex inter-dependencies among parameters. Most of the frameworks seek efficient resource management by assigning resource units to jobs, the maximum number of units allowed in a system being part of the static configuration of the system. This static resource management has limited effectiveness in coping with job diversity and workload dynamics, even in the case of a single job. The work reported in this article seeks to improve performance (e.g., multiple-jobs makespan and job completion time) without modification of either the framework or the applications and avoiding problems of previous self-tuning approaches based on performance models or resource usage. These problems include (1) the need for time-consuming training, typically offline and (2) unsuitability for multi-jobs/tenant environments. This article proposes a hierarchical self-tuning approach using (1) a fuzzy-logic controller to dynamically adjust the maximum number of concurrent jobs and (2) additional controllers (one for each cluster node) to adjust the maximum number of resource units assigned to jobs on each node. The fuzzy-logic controller uses fuzzy rules based on a concave-downward relationship between aggregate CPU usage and the number of concurrent jobs. The other controllers use a heuristic algorithm to adjust the number of resource units on the basis of both CPU and disk IO usage by jobs. To manage the maximum number of available resource units in each node, the controllers also take resource usage by other processes (e.g., system processes) into account. A prototype of our approach was implemented for Apache Hadoop on a cluster running at CloudLab. The proposed approach was demonstrated and evaluated with workloads composed of jobs with similar resource usage patterns as well as other realistic mixed-pattern workloads synthesized by SWIM, a statistical workload injector for MapReduce. The evaluation shows that the proposed approach yields up to a 48% reduction of the jobs makespan that results from using Hadoop-default settings.
- Yanpei Chen, Archana Ganapathi, Rean Griffith, and Randy Katz. 2011. The case for evaluating mapreduce performance using workload suites. In Proceedings of the 2011 IEEE 19th International Symposium on Modeling, Analysis 8 Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 390--399. Google Scholar
Digital Library
- Dazhao Cheng, Jia Rao, Yanfei Guo, and Xiaobo Zhou. 2014. Improving mapreduce performance in heterogeneous environments with adaptive task tuning. In Proceedings of the 15th International Middleware Conference. ACM, 97--108. Google Scholar
Digital Library
- Jeffrey Dean and Sanjay Ghemawat. 2008. Mapreduce: Simplified data processing on large clusters. Communications of the ACM 51, 1 (2008), 107--113. Google Scholar
Digital Library
- Xiaoan Ding, Yi Liu, and Depei Qian. 2015. Jellyfish: Online performance tuning with adaptive configuration and elastic container in Hadoop yarn. In Proceedings of the 2015 IEEE 21st International Conference on Parallel and Distributed Systems (ICPADS’15). IEEE, 831--836. Google Scholar
Digital Library
- Zhenhua Guo, Geoffrey Fox, Mo Zhou, and Yang Ruan. 2012. Improving resource utilization in mapreduce. In Proceedings of the 2012 IEEE International Conference on Cluster Computing (CLUSTER’12). IEEE, 402--410. Google Scholar
Digital Library
- Mohammad Hammoud and Majd F. Sakr. 2013. Mc2: Map concurrency characterization for mapreduce on the cloud. In Proceedings of the 2013 IEEE 6th International Conference on Cloud Computing (CLOUD’13). IEEE, 17--26. Google Scholar
Digital Library
- Herodotos Herodotou, Harold Lim, Gang Luo, Nedyalko Borisov, Liang Dong, Fatma Bilgen Cetin, and Shivnath Babu. 2011. Starfish: A self-tuning system for big data analytics. In CIDR. 261--272.Google Scholar
- Impetus. 2009. Advanced Hadoop Tuning and Optimizations. Retrieved on January 26, 2019 from http://www.slideshare.net/impetusinfo/ppt-on-advancedhadoop-tuning-n-optimisation.Google Scholar
- Selvi Kadirvel and José A. B. Fortes. 2012. Grey-box approach for performance prediction in map-reduce based platforms. In Proceedings of the 2012 21st International Conference on Computer Communications and Networks (ICCCN’12). IEEE, 1--9.Google Scholar
- Kamal Kc and Vincent W. Freeh. 2015. Dynamically controlling node-level parallelism in Hadoop. In Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing (CLOUD’15). IEEE, 309--316. Google Scholar
Digital Library
- Palden Lama and Xiaobo Zhou. 2012. Aroma: Automated resource allocation and configuration of mapreduce environment in the cloud. In Proceedings of the 9th International Conference on Autonomic Computing. ACM, 63--72. Google Scholar
Digital Library
- Gil Jae Lee and José A. B. Fortes. 2016. Hadoop performance self-tuning using a fuzzy-prediction approach. In Proceedings of the 2016 IEEE International Conference on Autonomic Computing (ICAC’16). IEEE, 55--64.Google Scholar
- Min Li, Liangzhao Zeng, Shicong Meng, Jian Tan, Li Zhang, Ali R. Butt, and Nicholas Fuller. 2014. Mronline: Mapreduce online performance tuning. In Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing. ACM, 165--176. Google Scholar
Digital Library
- Guangdeng Liao, Kushal Datta, and Theodore L. Willke. 2013. Gunther: Search-based auto-tuning of mapreduce. In Proceedings of the Euro-Par 2013 Parallel Processing. Springer, 406--419. Google Scholar
Digital Library
- Jorda Polo, David Carrera, Yolanda Becerra, Malgorzata Steinder, and Ian Whalley. 2010. Performance-driven task co-scheduling for mapreduce environments. In Proceedings of Network Operations and Management Symposium (NOMS’10). IEEE, 373--380.Google Scholar
Cross Ref
- Albert Reuther, Chansup Byun, William Arcand, David Bestor, Bill Bergeron, Matthew Hubbell, Michael Jones, Peter Michaleas, Andrew Prout, Antonio Rosa, and Jeremy Kepner. 2016. Scheduler technologies in support of high performance data analysis. In Proceedings of the High Performance Extreme Computing Conference (HPEC’16). IEEE, 1--6.Google Scholar
Cross Ref
- Robert Ricci, Eric Eide, and Cloud Lab Team. 2014. Introducing CloudLab: Scientific infrastructure for advancing cloud architectures and applications. USENIX 8 SAGE 39, 6 (2014), 36--38.Google Scholar
- Hadoop: Capacity Scheduler. Retrieved from http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/capacityscheduler.html.Google Scholar
- Hadoop: Fair Scheduler. Retrieved from http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/fairscheduler.html.Google Scholar
- Bikash Sharma, Ramya Prabhakar, S. Lim, Mahmut T. Kandemir, and Chita R. Das. 2012. Mrorchestrator: A Fine-Grained Resource Orchestration Framework for Hadoop Mapreduce. Technical Report CSE-12-001, Pennsylvania State University.Google Scholar
- Shanjiang Tang, Bu-Sung Lee, and Bingsheng He. 2014. Dynamicmr: A dynamic slot allocation optimization framework for mapreduce clusters. IEEE Transactions on Cloud Computing 2, 3 (2014), 333--347.Google Scholar
Cross Ref
- Fengguang Tian and Keke Chen. 2011. Towards optimal resource provisioning for running mapreduce programs in public clouds. In Proceedings of the 2011 IEEE International Conference on Cloud Computing (CLOUD’11). IEEE, 155--162. Google Scholar
Digital Library
- Werner Van Leekwijck and Etienne E. Kerre. 1999. Defuzzification: Criteria and classification. Fuzzy Sets and Systems 108, 2 (1999), 159--178. Google Scholar
Digital Library
- Vinod Kumar Vavilapalli, Arun C. Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, Jason Lowe, Hitesh Shah, Siddharth Seth, Bikas Saha, Carlo Curino, Owen O'Malley, Sanjay Radia, Benjamin Reed, and Eric Baldeschwieler. 2013. Apache Hadoop yarn: Yet another resource negotiator. In Proceedings of the 4th Annual Symposium on Cloud Computing. ACM, 5. Google Scholar
Digital Library
- Abhishek Verma, Ludmila Cherkasova, and Roy H. Campbell. 2011. Aria: Automatic resource inference and allocation for mapreduce environments. In Proceedings of the 8th ACM International Conference on Autonomic Computing. ACM, 235--244. Google Scholar
Digital Library
- Tom White. 2012. Hadoop: The Definitive Guide. O'Reilly Media, Inc. Google Scholar
Digital Library
- Nezih Yigitbasi, Theodore L. Willke, Guangdeng Liao, and Dick Epema. 2013. Towards machine learning-based auto-tuning of mapreduce. In Proceedings of the 2013 IEEE 21st International Symposium on Modeling, Analysis 8 Simulation of Computer and Telecommunication Systems (MASCOTS’13). IEEE, 11--20. Google Scholar
Digital Library
- Zhibin Yu, Zhendong Bei, Huiling Zhang, Wen Xiong, Lieven Eeckhout, Chengzhong Xu, and Shenzhong Feng. 2016. RFHOC: A random-forest approach to auto-tuning Hadoop's configuration. IEEE Transactions on Parallel and Distributed Systems 27, 5 (2016), 1470--1483. Google Scholar
Digital Library
- Lotfi A. Zadeh. 1965. Fuzzy sets. Information and control 8, 3 (1965), 338--353.Google Scholar
- Bo Zhang, F. Krikava, R. Rouvoy, and L. Seinturier. 2015. Self-configuration of the number of concurrently running mapreduce jobs in a Hadoop cluster. In Proceedings of the 2015 IEEE International Conference on Autonomic Computing (ICAC’15). 149--150. Google Scholar
Digital Library
- F. Yan, L. Cherkasova, Z. Zhang, and E. Smirni. 2017. Dyscale: A mapreduce job scheduler for heterogeneous multicore processors. IEEE Transactions on Cloud Computing 5, 2 (2017), 317--330.Google Scholar
Cross Ref
Index Terms
Improving Data-Analytics Performance Via Autonomic Control of Concurrency and Resource Units
Recommendations
Evaluation and Analysis of Capacity Scheduler and Fair Scheduler in Hadoop Framework on Big Data Technology
AIVR 2018: Proceedings of the 2018 International Conference on Artificial Intelligence and Virtual RealityApache Hadoop is an open source framework that implements MapReduce. It is scalable, reliable, and fault tolerant. Scheduling is an important process in Hadoop MapReduce. It is because scheduling has responsibility to allocate resources for running ...
A classification of hadoop job schedulers based on performance optimization approaches
AbstractJob scheduling in MapReduce plays a vital role in Hadoop performance. In recent years, many researchers have presented job scheduler algorithms to improve Hadoop performance. Designing a job scheduler that minimizes job execution time with maximum ...
Multi-policy-aware MapReduce resource allocation and scheduling for smart computing cluster
When a user submit a MapReduce job in the smart computing cluster, we first need to allocate cluster resource for the job. It is widely concerned that how to save time and resource costs to provide users with computing capacity and services. Here, we ...






Comments