skip to main content
research-article

Efficient shuffle management with SCache for DAG computing frameworks

Published:10 February 2018Publication History
Skip Abstract Section

Abstract

In large-scale data-parallel analytics, shuffle, or the cross-network read and aggregation of partitioned data between tasks with data dependencies, usually brings in large overhead. To reduce shuffle overhead, we present SCache, an open source plug-in system that particularly focuses on shuffle optimization. By extracting and analyzing shuffle dependencies prior to the actual task execution, SCache can adopt heuristic pre-scheduling combining with shuffle size prediction to pre-fetch shuffle data and balance load on each node. Meanwhile, SCache takes full advantage of the system memory to accelerate the shuffle process. We have implemented SCache and customized Spark to use it as the external shuffle service and co-scheduler. The performance of SCache is evaluated with both simulations and testbed experiments on a 50-node Amazon EC2 cluster. Those evaluations have demonstrated that, by incorporating SCache, the shuffle overhead of Spark can be reduced by nearly 89%, and the overall completion time of TPC-DS queries improves 40% on average.

References

  1. Faraz Ahmad, Srimat T Chakradhar, Anand Raghunathan, and TN Vijaykumar. 2014. ShuffleWatcher: Shuffle-aware Scheduling in Multi-tenant MapReduce Clusters.. In USENIX Annual Technical Conference. 1--12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Ganesh Ananthanarayanan, Ali Ghodsi, Andrew Wang, Dhruba Borthakur, Srikanth Kandula, Scott Shenker, and Ion Stoica. 2012. PAC-Man: Coordinated memory caching for parallel jobs. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation. USENIX Association, 20--20. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Ganesh Ananthanarayanan, Srikanth Kandula, Albert G Greenberg, Ion Stoica, Yi Lu, Bikas Saha, and Edward Harris. 2010. Reining in the Outliers in Map-Reduce Clusters using Mantri.. In OSDI, Vol. 10. 24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Apache. 2017. Apache Hadoop. (2017). http://hadoop.apache.com/Google ScholarGoogle Scholar
  5. Apache. 2017. Apache Hadoop Tutorial. (2017). http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.htmlGoogle ScholarGoogle Scholar
  6. Apache. 2017. Apache Spark 1.6.2 Configuration. (2017). http://spark.apache.org/docs/1.6.2/configuration.htmlGoogle ScholarGoogle Scholar
  7. Shivnath Babu. 2010. Towards Automatic Optimization of MapReduce Programs. In Proceedings of the 1st ACM Symposium on Cloud Computing (SoCC '10). ACM, New York, NY, USA, 137--142. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Laszlo A. Belady. 1966. A study of replacement algorithms for a virtual-storage computer. IBM Systems journal 5, 2 (1966), 78--101. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Dazhao Cheng, Jia Rao, Yanfei Guo, and Xiaobo Zhou. 2014. Improving MapReduce performance in heterogeneous environments with adaptive task tuning. In Proceedings of the 15th International Middleware Conference. ACM, 97--108. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Mosharaf Chowdhury and Ion Stoica. 2015. Efficient Coflow Scheduling Without Prior Knowledge. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication (SIGCOMM '15). ACM, New York, NY, USA, 393--406. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Mosharaf Chowdhury, Matei Zaharia, Justin Ma, Michael I Jordan, and Ion Stoica. 2011. Managing data transfers in computer clusters with orchestra. In ACM SIGCOMM Computer Communication Review, Vol. 41. ACM, 98--109. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Mosharaf Chowdhury, Yuan Zhong, and Ion Stoica. 2014. Efficient coflow scheduling with varys. In ACM SIGCOMM Computer Communication Review, Vol. 44. ACM, 443--454. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Eslam Elnikety, Tamer Elsayed, and Hany E Ramadan. 2011. iHadoop: asynchronous iterations for MapReduce. In Cloud Computing Technology and Science (CloudCom), 2011 IEEE Third International Conference on. IEEE, 81--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Brad Fitzpatrick. 2004. Distributed Caching with Memcached. Linux J. 2004, 124 (Aug. 2004), 5-. http://dl.acm.org/citation.cfm?id=1012889.1012894 Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. 2003. The Google file system. In ACM SIGOPS operating systems review, Vol. 37. ACM, 29--43. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Benjamin Gufler, Nikolaus Augsten, Angelika Reiser, and Alfons Kemper. 2012. Load balancing in mapreduce based on scalable cardinality estimates. In Data Engineering (ICDE), 2012 IEEE 28th International Conference on. IEEE, 522--533. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Herodotos Herodotou, Harold Lim, Gang Luo, Nedyalko Borisov, Liang Dong, Fatma Bilgen Cetin, and Shivnath Babu. 2011. Starfish: A Self-tuning System for Big Data Analytics.. In Cidr, Vol. 11. 261--272.Google ScholarGoogle Scholar
  18. Ewan Higgs, Animesh Trivedi, and Jeff Zhang. 2017. Spark Terasort. (2017). https://github.com/ehiggs/spark-terasortGoogle ScholarGoogle Scholar
  19. Patrick Hunt, Mahadev Konar, Flavio Paiva Junqueira, and Benjamin Reed. 2010. ZooKeeper: Wait-free Coordination for Internet-scale Systems.. In USENIX Annual Technical Conference, Vol. 8. 9. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly. 2007. Dryad: distributed data-parallel programs from sequential building blocks. In ACM SIGOPS operating systems review, Vol. 41. ACM, 59--72. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Michael Isard, Vijayan Prabhakaran, Jon Currey, Udi Wieder, Kunal Talwar, and Andrew Goldberg. 2009. Quincy: fair scheduling for distributed computing clusters. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. ACM, 261--276. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Steven Y Ko, Imranul Hoque, Brian Cho, and Indranil Gupta. 2009. On Availability of Intermediate Data in Cloud Computations.. In HotOS. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. YongChul Kwon, Magdalena Balazinska, Bill Howe, and Jerome Rolia. 2012. Skewtune: mitigating skew in mapreduce applications. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. ACM, 25--36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Haoyuan Li, Ali Ghodsi, Matei Zaharia, Scott Shenker, and Ion Stoica. 2014. Tachyon: Reliable, memory speed storage for cluster computing frameworks. In Proceedings of the ACM Symposium on Cloud Computing. ACM, 1--15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Diego Ongaro, Stephen M Rumble, Ryan Stutsman, John Ousterhout, and Mendel Rosenblum. 2011. Fast crash recovery in RAMCloud. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles. ACM, 29--41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Kay Ousterhout, Ryan Rasti, Sylvia Ratnasamy, Scott Shenker, Byung-Gon Chun, and V ICSI. 2015. Making Sense of Performance in Data Analytics Frameworks.. In NSDI, Vol. 15. 293--307. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Bikas Saha, Hitesh Shah, Siddharth Seth, Gopal Vijayaraghavan, Arun Murthy, and Carlo Curino. 2015. Apache tez: A unifying framework for modeling and building data processing applications. In Proceedings of the 2015 ACM SIGMOD international conference on Management of Data. ACM, 1357--1369. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Jian Tan, Alicia Chin, Zane Zhenhua Hu, Yonggang Hu, Shicong Meng, Xiaoqiao Meng, and Li Zhang. 2014. Dynmr: Dynamic mapreduce with reducetask interleaving and maptask backfilling. In Proceedings of the Ninth European Conference on Computer Systems. ACM, 2. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. TPC. 2017. TPC Benchmark DS (TPC-DS): The Benchmark Standard for decision support solutions including Big Data. (2017). http://www.tpc.org/tpcds/Google ScholarGoogle Scholar
  30. Abhishek Verma, Ludmila Cherkasova, and Roy H Campbell. 2011. Resource provisioning framework for mapreduce jobs with performance goals. In ACM/IFIP/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing. Springer, 165--186. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Jeffrey S Vitter. 1985. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS) 11, 1 (1985), 37--57. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Yandong Wang, Jian Tan, Weikuan Yu, Li Zhang, Xiaoqiao Meng, and Xiaobing Li. 2013. Preemptive ReduceTask Scheduling for Fair and Fast Job Completion.. In ICAC. 279--289.Google ScholarGoogle Scholar
  33. David P Williamson and David B Shmoys. 2010. The Design of Approximation Algorithms. 2010. preprint http://www.designofapproxalgs.com (2010). Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Chenning Xie, Rong Chen, Haibing Guan, Binyu Zang, and Haibo Chen. 2015. SYNC or ASYNC: Time to Fuse for Distributed Graph-parallel Computation. In Proceedings of the 20th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP 2015). ACM, New York, NY, USA, 194--204. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, and Ion Stoica. 2010. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of the 5th European conference on Computer systems. ACM, 265--278. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation. USENIX Association, 2--2. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica. 2016. Apache Spark: A Unified Engine for Big Data Processing. Commun. ACM 59, 11 (Oct. 2016), 56--65. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Efficient shuffle management with SCache for DAG computing frameworks

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM SIGPLAN Notices
      ACM SIGPLAN Notices  Volume 53, Issue 1
      PPoPP '18
      January 2018
      426 pages
      ISSN:0362-1340
      EISSN:1558-1160
      DOI:10.1145/3200691
      Issue’s Table of Contents
      • cover image ACM Conferences
        PPoPP '18: Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming
        February 2018
        442 pages
        ISBN:9781450349826
        DOI:10.1145/3178487

      Copyright © 2018 ACM

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 10 February 2018

      Check for updates

      Qualifiers

      • research-article

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!