Abstract
Combining type theory, language design, and empirical work, we present techniques for computing with large and dynamically changing datasets. Based on lambda calculus, our techniques are suitable for expressing a diverse set of algorithms on large datasets and, via self-adjusting computation, enable computations to respond automatically to changes in their data. To improve the scalability of self-adjusting computation, we present a type system for precise dependency tracking that minimizes the time and space for storing dependency metadata. The type system eliminates an important assumption of prior work that can lead to recording spurious dependencies. We present a type-directed translation algorithm that generates correct self-adjusting programs without relying on this assumption. We then show a probabilistic-chunking technique to further decrease space usage by controlling the fundamental space-time tradeoff in self-adjusting computation. We implement and evaluate these techniques, showing promising results on challenging benchmarks involving large graphs.
- U. A. Acar, G. E. Blelloch, and R. Harper. Adaptive functional programming. ACM Trans. Prog. Lang. Sys., 28(6):990--1034, 2006. Google Scholar
Digital Library
- U. A. Acar, G. E. Blelloch, K. Tangwongsan, and J. L. Vittes. Kinetic algorithms via self-adjusting computation. In Proceedings of the 14th Annual European Symposium on Algorithms, pages 636--647, Sept. 2006. Google Scholar
Digital Library
- U. A. Acar, A. Ihler, R. Mettu, and O. Sümer. Adaptive Bayesian inference. In Neural Information Processing Systems (NIPS), 2007.Google Scholar
- U. A. Acar, A. Ahmed, and M. Blume. Imperative self-adjusting computation. In Proceedings of the 25th Annual ACM Symposium on Principles of Programming Languages, 2008. Google Scholar
Digital Library
- U. A. Acar, G. E. Blelloch, K. Tangwongsan, and D. Türkoğlu. Robust kinetic convex hulls in 3D. In Proceedings of the 16th Annual European Symposium on Algorithms, Sept. 2008. Google Scholar
Digital Library
- U. A. Acar, G. E. Blelloch, M. Blume, R. Harper, and K. Tangwongsan. An experimental analysis of self-adjusting computation. ACM Trans. Prog. Lang. Sys., 32(1):3:1--53, 2009. Google Scholar
Digital Library
- U. A. Acar, A. Cotter, B. Hudson, and D. Türkoğlu. Parallelism in dynamic well-spaced point sets. In Proceedings of the 23rd ACM Symposium on Parallelism in Algorithms and Architectures, 2011. Google Scholar
Digital Library
- U. A. Acar, A. Cotter, B. Hudson, and D. Türkoğlu. Dynamic well-spaced point sets. Journal of Computational Geometry: Theory and Applications, 2013. Google Scholar
Digital Library
- G. Berry and G. Gonthier. The esterel synchronous programming language: design, semantics, implementation. Sci. Comput. Program., 19(2):87--152, Nov. 1992. ISSN 0167-6423. Google Scholar
Digital Library
- P. Bhatotia, A. Wieder, R. Rodrigues, U. A. Acar, and R. Pasquini. Incoop: MapReduce for incremental computations. In ACM Symposium on Cloud Computing, 2011. Google Scholar
Digital Library
- S. Burckhardt, D. Leijen, C. Sadowski, J. Yi, and T. Ball. Two for the price of one: A model for parallel and incremental computation. In ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, 2011. Google Scholar
Digital Library
- M. Carlsson. Monads for incremental computing. In International Conference on Functional Programming, pages 26--35, 2002. Google Scholar
Digital Library
- P. Caspi, D. Pilaud, N. Halbwachs, and J. A. Plaice. Lustre: a declarative language for real-time programming. In Proceedings of the 14th ACM SIGACT-SIGPLAN symposium on Principles of programming languages, POPL '87, pages 178--188, 1987. ISBN 0-89791-215-2. Google Scholar
Digital Library
- Y. Chen, U. A. Acar, and K. Tangwongsan. Appendix to functional programming for dynamic and large data with self-adjusting computation. URL http://www.mpi-sws.org/~chenyan/papers/icfp14-appendix.pdf.Google Scholar
- Y. Chen, J. Dunfield, M. A. Hammer, and U. A. Acar. Implicit self-adjusting computation for purely functional programs. In Int'l Conference on Functional Programming (ICFP '11), pages 129--141, Sept. 2011. Google Scholar
Digital Library
- Y. Chen, J. Dunfield, and U. A. Acar. Type-directed automatic incrementalization. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Jun 2012. Google Scholar
Digital Library
- Y. Chen, J. Dunfield, M. A. Hammer, and U. A. Acar. Implicit selfadjusting computation for purely functional programs. Journal of Functional Programming, 24:56--112, 1 2014. ISSN 1469-7653.Google Scholar
Cross Ref
- T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears. Mapreduce online. In Proc. 7th Symposium on Networked systems design and implementation (NSDI'10). Google Scholar
Digital Library
- E. Czaplicki and S. Chong. Asynchronous functional reactive programming for guis. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, PLDI '13, pages 411--422, 2013. ISBN 978-1-4503-2014-6. Google Scholar
Digital Library
- J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1):107--113, 2008. Google Scholar
Digital Library
- A. Demers, T. Reps, and T. Teitelbaum. Incremental evaluation of attribute grammars with application to syntax-directed editors. In Principles of Programming Languages, pages 105--116, 1981. Google Scholar
Digital Library
- C. Demetrescu, S. Emiliozzi, and G. F. Italiano. Experimental analysis of dynamic all pairs shortest path algorithms. In ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 369--378, 2004. Google Scholar
Digital Library
- C. Demetrescu, I. Finocchi, and G. Italiano. Handbook on Data Structures and Applications, chapter 36: Dynamic Graphs. CRC Press, 2005.Google Scholar
- C. Demetrescu, I. Finocchi, and A. Ribichini. Reactive imperative programming with dataflow constraints. In Proceedings of ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2011. Google Scholar
Digital Library
- J. Donham. Froc: a library for functional reactive programming in ocaml, 2010. URL http://jaked.github.com/froc.Google Scholar
- C. Elliott and P. Hudak. Functional reactive animation. In Proceedings of the second ACM SIGPLAN International Conference on Functional Programming, pages 263--273. ACM, 1997. Google Scholar
Digital Library
- J. Field and T. Teitelbaum. Incremental reduction in the lambda calculus. In ACM Conference on LISP and Functional Programming, pages 307--322, 1990. Google Scholar
Digital Library
- P. K. Gunda, L. Ravindranath, C. A. Thekkath, Y. Yu, and L. Zhuang. Nectar: Automatic management of data and computation in data centers. In OSDI'10. Google Scholar
Digital Library
- M. Hammer, U. A. Acar, M. Rajagopalan, and A. Ghuloum. A proposal for parallel self-adjusting computation. In DAMP '07: Declarative Aspects of Multicore Programming, 2007. Google Scholar
Digital Library
- M. A. Hammer, U. A. Acar, and Y. Chen. CEAL: a C-based language for self-adjusting computation. In ACM SIGPLAN Conference on Programming Language Design and Implementation, 2009. Google Scholar
Digital Library
- M. A. Hammer, K. Y. Phang, M. Hicks, and J. S. Foster. Adapton: Composable, demand-driven incremental computation. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '14, pages 156--166, 2014. ISBN 978-1-4503-2784-8. Google Scholar
Digital Library
- M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. SIGOPS Oper. Syst. Rev., 41(3):59--72, Mar. 2007. ISSN 0163-5980. Google Scholar
Digital Library
- A. Jeffrey. LTL types FRP: linear-time temporal logic propositions as types, proofs as functional reactive programs. In PLPV '12: Proceedings of the sixth workshop on Programming languages meets program verification, pages 49--60, 2012. ISBN 978-1-4503-1125-0. Google Scholar
Digital Library
- W. Jeltsch. Temporal logic with "until", functional reactive programming with processes, and concrete process categories. In Proceedings of the 7th Workshop on Programming Languages Meets Program Verification, PLPV '13, pages 69--78, 2013. ISBN 978-1-4503-1860-0. Google Scholar
Digital Library
- U. Kang, C. E. Tsourakakis, A. P. Appel, C. Faloutsos, and J. Leskovec. Hadi: Mining radii of large graphs. TKDD, 5(2):8, 2011. Google Scholar
Digital Library
- U. Kang, C. E. Tsourakakis, and C. Faloutsos. Pegasus: mining petascale graphs. Knowl. Inf. Syst., 27(2):303--325, 2011. Google Scholar
Digital Library
- N. R. Krishnaswami. Higher-order functional reactive programming without spacetime leaks. SIGPLAN Not., 48(9):221--232, Sept. 2013. ISSN 0362-1340. Google Scholar
Digital Library
- H. Liu, E. Cheng, and P. Hudak. Causal commutative arrows and their optimization. In Proceedings of the 14th ACM SIGPLAN international conference on Functional programming, ICFP '09, pages 35--46, 2009. ISBN 978-1-60558-332-7. Google Scholar
Digital Library
- Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M. Hellerstein. Distributed GraphLab: a framework for machine learning and data mining in the cloud. VLDB Endow., 5(8):716--727, Apr. 2012. Google Scholar
Digital Library
- G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: a system for large-scale graph processing. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, SIGMOD '10, pages 135--146, 2010. Google Scholar
Digital Library
- S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis. Dremel: interactive analysis of web-scale datasets. Commun. ACM, 54(6):114--123, June 2011. ISSN 0001-0782. Google Scholar
Digital Library
- D. G. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi. Naiad: A timely dataflow system. In Proc. of SOSP, pages 439--455, 2013. Google Scholar
Digital Library
- A. Muthitacharoen, B. Chen, and D. Mazières. A low-bandwidth network file system. In SOSP, pages 174--187, 2001. Google Scholar
Digital Library
- M. Odersky, M. Sulzmann, and M. Wehr. Type inference with constrained types. Theory and Practice of Object Systems, 5(1):35--55, 1999. Google Scholar
Digital Library
- D. Peng and F. Dabek. Large-scale incremental processing using distributed transactions and notifications. In Proc. 9th Symposium on Operating Systems Design and Implementation (OSDI'10), 2010. Google Scholar
Digital Library
- W. Pugh and T. Teitelbaum. Incremental computation via function caching. In Principles of Programming Languages, pages 315--328, 1989. Google Scholar
Digital Library
- G. Ramalingam and T. Reps. A categorized bibliography on incremental computation. In Principles of Programming Languages, pages 502--510, 1993. Google Scholar
Digital Library
- E. J. Riedy, H. Meyerhenke, D. A. Bader, D. Ediger, and T. G. Mattson. Analysis of streaming social networks and graphs on multicore architectures. In ICASSP, pages 5337--5340, 2012.Google Scholar
Cross Ref
- N. Sculthorpe and H. Nilsson. Keeping calm in the face of change. Higher Order Symbol. Comput., 23(2):227--271, June 2010. ISSN 1388-3690. Google Scholar
Digital Library
- N. Sculthorpe and H. Nilsson. Safe functional reactive programming through dependent types. SIGPLAN Not., 44(9):23--34, Aug. 2009. ISSN 0362-1340. Google Scholar
Digital Library
- O. Sümer, U. A. Acar, A. Ihler, and R. Mettu. Adaptive exact inference in graphical models. Journal of Machine Learning, 8:180--186, 2011.Google Scholar
- J.-P. Talpin and P. Jouvelot. The type and effect discipline. Inf. Comput., 111(2):245--296, June 1994. ISSN 0890-5401. Google Scholar
Digital Library
- K. Tangwongsan, H. Pucha, D. G. Andersen, and M. Kaminsky. Efficient similarity estimation for systems exploiting data redundancy. In INFOCOM, pages 1487--1495, 2010. Google Scholar
Digital Library
- Z. Wan, W. Taha, and P. Hudak. Real-time FRP. SIGPLAN Not., 36 (10):146--156, 2001. Google Scholar
Digital Library
- Z. Wan, W. Taha, and P. Hudak. Event-driven FRP. In Proceedings of the 4th International Symposium on Practical Aspects of Declarative Languages, PADL '02, pages 155--172, 2002. Google Scholar
Digital Library
- D. M. Yellin and R. E. Strom. INC: a language for incremental computations. ACM Transactions on Programming Languages and Systems, 13(2):211--236, Apr. 1991. Google Scholar
Digital Library
Index Terms
Functional programming for dynamic and large data with self-adjusting computation
Recommendations
Functional programming for dynamic and large data with self-adjusting computation
ICFP '14: Proceedings of the 19th ACM SIGPLAN international conference on Functional programmingCombining type theory, language design, and empirical work, we present techniques for computing with large and dynamically changing datasets. Based on lambda calculus, our techniques are suitable for expressing a diverse set of algorithms on large ...
An experimental analysis of self-adjusting computation
Recent work on adaptive functional programming (AFP) developed techniques for writing programs that can respond to modifications to their data by performing change propagation. To achieve this, executions of programs are represented with dynamic ...
Implicit self-adjusting computation for purely functional programs
ICFP '11: Proceedings of the 16th ACM SIGPLAN international conference on Functional programmingComputational problems that involve dynamic data, such as physics simulations and program development environments, have been an important subject of study in programming languages. Building on this work, recent advances in self-adjusting computation ...







Comments