skip to main content
research-article
Public Access

CoRAL: Confined Recovery in Distributed Asynchronous Graph Processing

Authors Info & Claims
Published:04 April 2017Publication History
Skip Abstract Section

Abstract

Existing distributed asynchronous graph processing systems employ checkpointing to capture globally consistent snapshots and rollback all machines to most recent checkpoint to recover from machine failures. In this paper we argue that recovery in distributed asynchronous graph processing does not require the entire execution state to be rolled back to a globally consistent state due to the relaxed asynchronous execution semantics. We define the properties required in the recovered state for it to be usable for correct asynchronous processing and develop CoRAL, a lightweight checkpointing and recovery algorithm. First, this algorithm carries out confined recovery that only rolls back graph execution states of the failed machines to affect recovery. Second, it relies upon lightweight checkpoints that capture locally consistent snapshots with a reduced peak network bandwidth requirement. Our experiments using real-world graphs show that our technique recovers from failures and finishes processing 1.5x to 3.2x faster compared to the traditional asynchronous checkpointing and recovery mechanism when failures impact 1 to 6 machines of a 16 machine cluster. Moreover, capturing locally consistent snapshots significantly reduces intermittent high peak bandwidth usage required to save the snapshots -- the average reduction in 99th percentile bandwidth ranges from 22% to 51% while 1 to 6 snapshot replicas are being maintained.

References

  1. L. Backstrom, D. Huttenlocher, J. Kleinberg, and X. Lan. Group formation in large social networks: Membership, growth, and evolution. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 44--54, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. P. Boldi and S. Vigna. The WebGraph framework I: Compression techniques. In WWW, pages 595--601, 2004.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. K. M. Chandy and L. Lamport. Distributed snapshots: Determining global states of distributed systems. ACM TOCS, 3(1):63--75, Feb. 1985. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. A. Ching, S. Edunov, M. Kabiljo, D. Logothetis, and S. Muthukrishnan. One trillion edges: graph processing at facebook-scale. In Proc. VLDB Endowment, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. E. N. M. Elnozahy, L. Alvisi, Y.-M. Wang, and D. B. Johnson. A survey of rollback-recovery protocols in message-passing systems. ACM Computing Surveys, 34(3):375--408, Sept. 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. A. Farahat, T. LoFaro, J. C. Miller, G. Rae, and L. A. Ward. Authority rankings from hits, pagerank, and salsa: Existence, uniqueness, and effect of initialization. SIAM Jornal of Scientific Computing, 27(4):1181--1201, Nov. 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. J. E. Gonzalez, R. S. Xin, A. Dave, D. Crankshaw, M. J. Franklin, and I. Stoica. GraphX: Graph processing in a distributed dataflow framework. In USENIX OSDI, pages 599--613, 2014.Google ScholarGoogle Scholar
  8. M. Han and K. Daudjee. Giraph unchained: Barrierless asynchronous parallel execution in pregel-like graph processing systems. Proc. VLDB Endowment, 8(9):950--961, May 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Harshvardhan, A. Fidel, N. M. Amato, and L. Rauchwerger. Kla: A new algorithmic paradigm for parallel graph computations. In PACT, pages 27--38, New York, NY, 2014.Google ScholarGoogle Scholar
  10. P. Hunt, M. Konar, F. P. Junqueira, and B. Reed. Zookeeper: Wait-free coordination for internet-scale systems. In USENIX ATC, pages 11--11, Berkeley, CA, 2010.Google ScholarGoogle Scholar
  11. H. Kwak, C. Lee, H. Park, and S. Moon. What is Twitter, a social network or a news media? In WWW, 2010.%pages 591--600, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M. Hellerstein. Distributed graphlab: A framework for machine learning and data mining in the cloud. Proc. VLDB Endowment, 5(8):716--727, Apr. 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. G. Malewicz, M. H. Austern, A. J. C. Bik, J. C. Dehnert, I. Horn, N. Leiser, G. Czajkowski, and G. Inc. Pregel: A system for large-scale graph processing. In ACM SIGMOD, pages 135--146, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. D. Manivannan and M. Singhal. Quasi-synchronous checkpointing: Models, characterization, and classification. IEEE TPDS, 10(7):703--713, 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. D. Ongaro, S. M. Rumble, R. Stutsman, J. Ousterhout, and M. Rosenblum. Fast crash recovery in ramcloud. In ACM SOSP, pages 29--41, New York, NY, USA, 2011. ACM.\newpage Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. L. Page, S. Brin, R. Motwani, and T. Winograd. The PageRank citation ranking: Bringing order to the web. Technical report, Stanford University, 1998.Google ScholarGoogle Scholar
  17. R. Power and J. Li. Piccolo: Building fast, distributed programs with partitioned tables. In USENIX OSDI, pages 293--306, Berkeley, CA, USA, 2010.Google ScholarGoogle Scholar
  18. M. Pundir, L. M. Leslie, I. Gupta, and R. H. Campbell. Zorro: Zero-cost reactive failure recovery in distributed graph processing. In ACM SoCC, pages 195--208, 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. S. Salihoglu and J. Widom. GPS: A graph processing system. In Scientific and Statistical Database Management Conference, pages 22:1--22:12, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. B. Shao, H. Wang, and Y. Li. Trinity: A distributed graph engine on a memory cloud. In ACM SIGMOD, pages 505--516, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Y. Shen, G. Chen, H. V. Jagadish, W. Lu, B. C. Ooi, and B. M. Tudor. Fast failure recovery in distributed graph processing systems. Proc. VLDB Endowment, 8(4):437--448, Dec. 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. L. G. Valiant. A bridging model for parallel computation. CACM, 33(8):103--111, Aug. 1990. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. H. Cui, J. Cipar, Q. Ho, J.K. Kim, S, Lee, A. Kumar, J. Wei, W. Dai, G.R. Ganger, P.B. Gibbons, G.A. Gibson, and E.P. Xing. Exploiting Bounded Staleness to Speed Up Big Data Analytics. In USENIX ATC, pages 37--48, 2014.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. K. Vora, G. Xu, and R. Gupta. Load the Edges You Need: A Generic I/O Optimization for Disk-based Graph Processing. In USENIX ATC, pages 507--522, 2016.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. K. Vora, S. C. Koduru, and R. Gupta. ASPIRE: Exploiting Asynchronous Parallelism in Iterative Algorithms using a Relaxed Consistency based DSM. In OOPSLA, pages 861--878, 2014.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. G. Wang, W. Xie, A. Demers, and J. Gehrke. Asynchronous large-scale graph processing made easy. In Conference on Innovative Data Systems Research (CIDR), 2013.Google ScholarGoogle Scholar
  27. P. Wang, K. Zhang, R. Chen, and H. Chen. Replication-based fault-tolerance for large-scale graph processing. In IEEE/IFIP DSN, pages 562--573, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. J. W. Young. A first order approximation to the optimum checkpoint interval. CACM, 17(9):530--531, Sept. 1974. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In USENIX NSDI, pages 2--2, 2012.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. ZeroMQ. http://zeromq.org/.Google ScholarGoogle Scholar
  31. Y. Zhang, Q. Gao, L. Gao, and C. Wang. Accelerate large-scale iterative computation through asynchronous accumulative updates. In ScienceCloud, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. X. Zhu and Z. Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical Report CALD-02--107, Carnegie Mellon University, 2002.Google ScholarGoogle Scholar

Recommendations

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Sign in

Full Access

  • Published in

    cover image ACM SIGPLAN Notices
    ACM SIGPLAN Notices  Volume 52, Issue 4
    ASPLOS '17
    April 2017
    811 pages
    ISSN:0362-1340
    EISSN:1558-1160
    DOI:10.1145/3093336
    Issue’s Table of Contents
    • cover image ACM Conferences
      ASPLOS '17: Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems
      April 2017
      856 pages
      ISBN:9781450344654
      DOI:10.1145/3037697

    Copyright © 2017 ACM

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    • Published: 4 April 2017

    Check for updates

    Qualifiers

    • research-article

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader
About Cookies On This Site

We use cookies to ensure that we give you the best experience on our website.

Learn more

Got it!