skip to main content
article

Think like a vertex, behave like a function! a functional DSL for vertex-centric big graph processing

Published:04 September 2016Publication History
Skip Abstract Section

Abstract

The vertex-centric programming model, known as “think like a vertex”, is being used more and more to support various big graph processing methods through iterative supersteps that execute in parallel a user-defined vertex program over each vertex of a graph. However, the imperative and message-passing style of existing systems makes defining a vertex program unintuitive. In this paper, we show that one can benefit more from “Thinking like a vertex” by “Behaving like a function” rather than “Acting like a procedure” with full use of side effects and explicit control of message passing, state, and termination. We propose a functional approach to vertex-centric graph processing in which the computation at every vertex is abstracted as a higher-order function and present Fregel, a new domain-specific language. Fregel has clear functional semantics, supports declarative description of vertex computation, and can be automatically translated into Pregel, an emerging imperative-style distributed graph processing framework, and thereby achieve promising performance. Experimental results for several typical examples show the promise of this functional approach.

References

  1. S.-H. Bae and B. Howe. GossipMap: A distributed community detection algorithm for billion-edge directed graphs. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’15, pages 27:1–27:12. ACM, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. B. Bahmani, R. Kumar, and S. Vassilvitskii. Densest subgraph in streaming and MapReduce. Proceedings of the VLDB Endowment, 5(5):454–465, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. P. Buneman, M. Fernandez, and D. Suciu. UnQL: A query language and algebra for semistructured data based on structural recursion. The VLDB Journal, 9(1):76–110, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. M. Erwig. Functional programming with graphs. In Proceedings of the 1997 ACM SIGPLAN International Conference on Functional Programming, ICFP ’97, pages 52–65. ACM, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. M. Erwig. Inductive graphs and functional graph algorithms. Journal of Functional Programming, 11(5):467–492, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. L. Fegaras and T. Sheard. Revisiting catamorphisms over datatypes with embedded functions (or, programs from outer space). In Proceedings of the 23rd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’96, pages 284–294. ACM, 1996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. A. F. Gates, O. Natkovich, S. Chopra, P. Kamath, S. M. Narayanamurthy, C. Olston, B. Reed, S. Srinivasan, and U. Srivastava. Building a highlevel dataflow system on top of Map-Reduce: The Pig experience. Proceedings of the VLDB Endowment, 2(2):1414–1425, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin. PowerGraph: Distributed graph-parallel computation on natural graphs. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation, OSDI’12, pages 17–30. USENIX Association, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. M. Hamana. Initial algebra semantics for cyclic sharing tree structures. Logical Methods in Computer Science, 6(3):1–23, 2010.Google ScholarGoogle ScholarCross RefCross Ref
  10. S. Hidaka, Z. Hu, K. Inaba, H. Kato, K. Matsuda, and K. Nakano. Bidirectionalizing graph transformations. In Proceedings of the 15th ACM SIGPLAN International Conference on Functional Programming, ICFP ’10, pages 205–216. ACM, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. S. Hidaka, K. Asada, Z. Hu, H. Kato, and K. Nakano. Structural recursion for querying ordered graphs. In Proceedings of the 18th ACM SIGPLAN International Conference on Functional Programming, ICFP ’13, pages 305–318. ACM, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. S. Hong, H. Chafi, E. Sedlar, and K. Olukotun. Green-Marl: A DSL for easy and efficient graph analysis. In Proceedings of the Seventeenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XVII, pages 349–362. ACM, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. S. Hong, S. Salihoglu, J. Widom, and K. Olukotun. Simplifying scalable graph processing with a domain-specific language. In Proceedings of Annual IEEE/ACM International Symposium on Code Generation and Optimization, CGO ’14, pages 208–218. ACM, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. J. Launchbury. Graph algorithms with a functional flavour. In Advanced Functional Programming, First International Spring School on Advanced Functional Programming Techniques-Tutorial Text, pages 308– 331. Springer-Verlag, 1995. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. M. Lesniak. Palovca: Describing and executing graph algorithms in haskell. In Proceedings of the 14th International Conference on Practical Aspects of Declarative Languages, PADL’12, pages 153–167. Springer-Verlag, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M. Hellerstein. Distributed GraphLab: A framework for machine learning and data mining in the cloud. Proceedings of the VLDB Endowment, 5(8): 716–727, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: A system for large-scale graph processing. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, SIGMOD ’10, pages 135–146. ACM, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. R. R. McCune, T. Weninger, and G. Madey. Thinking like a vertex: A survey of vertex-centric frameworks for large-scale distributed graph processing. ACM Computing Surveys, 48(2):25:1–25:39, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. B. C. Oliveira and W. R. Cook. Functional programming with structured graphs. In Proceedings of the 17th ACM SIGPLAN International Conference on Functional Programming, ICFP ’12, pages 77–88. ACM, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. D. Sengupta, S. L. Song, K. Agarwal, and K. Schwan. GraphReduce: Processing large-scale graphs on accelerator-based systems. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’15, pages 28:1–28:12. ACM, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive: A warehousing solution over a Map-Reduce framework. Proceedings of the VLDB Endowment, 2(2):1626– 1629, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. L. D. Tung and Z. Hu. Towards systematic parallelization of graph transformations over Pregel. In Proceedings of the 8th International Symposium on High-level Parallel Programming and Applications, HLPP ’15, 2015.Google ScholarGoogle Scholar
  23. L. G. Valiant. A bridging model for parallel computation. Communications of the ACM, 33(8):103–111, 1990. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. D. Yan, J. Cheng, K. Xing, Y. Lu, W. Ng, and Y. Bu. Pregel algorithms for graph connectivity problems with performance guarantees. Proceedings of the VLDB Endowment, 7(14):1821–1832, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Think like a vertex, behave like a function! a functional DSL for vertex-centric big graph processing

              Recommendations

              Comments

              Login options

              Check if you have access through your login credentials or your institution to get full access on this article.

              Sign in

              Full Access

              PDF Format

              View or Download as a PDF file.

              PDF

              eReader

              View online with eReader.

              eReader
              About Cookies On This Site

              We use cookies to ensure that we give you the best experience on our website.

              Learn more

              Got it!