research-article

Functional programming for dynamic and large data with self-adjusting computation

Authors:

Kanat TangwongsanAuthors Info & Claims

ICFP '14: Proceedings of the 19th ACM SIGPLAN international conference on Functional programming

Pages 227 - 240

https://doi.org/10.1145/2628136.2628150

Published: 19 August 2014 Publication History

Abstract

Combining type theory, language design, and empirical work, we present techniques for computing with large and dynamically changing datasets. Based on lambda calculus, our techniques are suitable for expressing a diverse set of algorithms on large datasets and, via self-adjusting computation, enable computations to respond automatically to changes in their data. To improve the scalability of self-adjusting computation, we present a type system for precise dependency tracking that minimizes the time and space for storing dependency metadata. The type system eliminates an important assumption of prior work that can lead to recording spurious dependencies. We present a type-directed translation algorithm that generates correct self-adjusting programs without relying on this assumption. We then show a probabilistic-chunking technique to further decrease space usage by controlling the fundamental space-time tradeoff in self-adjusting computation. We implement and evaluate these techniques, showing promising results on challenging benchmarks involving large graphs.

References

[1]

U. A. Acar, G. E. Blelloch, and R. Harper. Adaptive functional programming. ACM Trans. Prog. Lang. Sys., 28(6):990--1034, 2006.

Digital Library

[2]

U. A. Acar, G. E. Blelloch, K. Tangwongsan, and J. L. Vittes. Kinetic algorithms via self-adjusting computation. In Proceedings of the 14th Annual European Symposium on Algorithms, pages 636--647, Sept. 2006.

Digital Library

[3]

U. A. Acar, A. Ihler, R. Mettu, and O. Sümer. Adaptive Bayesian inference. In Neural Information Processing Systems (NIPS), 2007.

[4]

U. A. Acar, A. Ahmed, and M. Blume. Imperative self-adjusting computation. In Proceedings of the 25th Annual ACM Symposium on Principles of Programming Languages, 2008.

Digital Library

[5]

U. A. Acar, G. E. Blelloch, K. Tangwongsan, and D. Türkoğlu. Robust kinetic convex hulls in 3D. In Proceedings of the 16th Annual European Symposium on Algorithms, Sept. 2008.

Digital Library

[6]

U. A. Acar, G. E. Blelloch, M. Blume, R. Harper, and K. Tangwongsan. An experimental analysis of self-adjusting computation. ACM Trans. Prog. Lang. Sys., 32(1):3:1--53, 2009.

Digital Library

[7]

U. A. Acar, A. Cotter, B. Hudson, and D. Türkoğlu. Parallelism in dynamic well-spaced point sets. In Proceedings of the 23rd ACM Symposium on Parallelism in Algorithms and Architectures, 2011.

Digital Library

[8]

U. A. Acar, A. Cotter, B. Hudson, and D. Türkoğlu. Dynamic well-spaced point sets. Journal of Computational Geometry: Theory and Applications, 2013.

Digital Library

[9]

G. Berry and G. Gonthier. The esterel synchronous programming language: design, semantics, implementation. Sci. Comput. Program., 19(2):87--152, Nov. 1992. ISSN 0167-6423.

Digital Library

[10]

P. Bhatotia, A. Wieder, R. Rodrigues, U. A. Acar, and R. Pasquini. Incoop: MapReduce for incremental computations. In ACM Symposium on Cloud Computing, 2011.

Digital Library

[11]

S. Burckhardt, D. Leijen, C. Sadowski, J. Yi, and T. Ball. Two for the price of one: A model for parallel and incremental computation. In ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, 2011.

Digital Library

[12]

M. Carlsson. Monads for incremental computing. In International Conference on Functional Programming, pages 26--35, 2002.

Digital Library

[13]

P. Caspi, D. Pilaud, N. Halbwachs, and J. A. Plaice. Lustre: a declarative language for real-time programming. In Proceedings of the 14th ACM SIGACT-SIGPLAN symposium on Principles of programming languages, POPL '87, pages 178--188, 1987. ISBN 0-89791-215-2.

Digital Library

[14]

Y. Chen, U. A. Acar, and K. Tangwongsan. Appendix to functional programming for dynamic and large data with self-adjusting computation. URL http://www.mpi-sws.org/~chenyan/papers/icfp14-appendix.pdf.

[15]

Y. Chen, J. Dunfield, M. A. Hammer, and U. A. Acar. Implicit self-adjusting computation for purely functional programs. In Int'l Conference on Functional Programming (ICFP '11), pages 129--141, Sept. 2011.

Digital Library

[16]

Y. Chen, J. Dunfield, and U. A. Acar. Type-directed automatic incrementalization. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Jun 2012.

Digital Library

[17]

Y. Chen, J. Dunfield, M. A. Hammer, and U. A. Acar. Implicit selfadjusting computation for purely functional programs. Journal of Functional Programming, 24:56--112, 1 2014. ISSN 1469-7653.

[18]

T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears. Mapreduce online. In Proc. 7th Symposium on Networked systems design and implementation (NSDI'10).

Digital Library

[19]

E. Czaplicki and S. Chong. Asynchronous functional reactive programming for guis. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, PLDI '13, pages 411--422, 2013. ISBN 978-1-4503-2014-6.

Digital Library

[20]

J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1):107--113, 2008.

Digital Library

[21]

A. Demers, T. Reps, and T. Teitelbaum. Incremental evaluation of attribute grammars with application to syntax-directed editors. In Principles of Programming Languages, pages 105--116, 1981.

Digital Library

[22]

C. Demetrescu, S. Emiliozzi, and G. F. Italiano. Experimental analysis of dynamic all pairs shortest path algorithms. In ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 369--378, 2004.

Digital Library

[23]

C. Demetrescu, I. Finocchi, and G. Italiano. Handbook on Data Structures and Applications, chapter 36: Dynamic Graphs. CRC Press, 2005.

[24]

C. Demetrescu, I. Finocchi, and A. Ribichini. Reactive imperative programming with dataflow constraints. In Proceedings of ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2011.

Digital Library

[25]

J. Donham. Froc: a library for functional reactive programming in ocaml, 2010. URL http://jaked.github.com/froc.

[26]

C. Elliott and P. Hudak. Functional reactive animation. In Proceedings of the second ACM SIGPLAN International Conference on Functional Programming, pages 263--273. ACM, 1997.

Digital Library

[27]

J. Field and T. Teitelbaum. Incremental reduction in the lambda calculus. In ACM Conference on LISP and Functional Programming, pages 307--322, 1990.

Digital Library

[28]

P. K. Gunda, L. Ravindranath, C. A. Thekkath, Y. Yu, and L. Zhuang. Nectar: Automatic management of data and computation in data centers. In OSDI'10.

Digital Library

[29]

M. Hammer, U. A. Acar, M. Rajagopalan, and A. Ghuloum. A proposal for parallel self-adjusting computation. In DAMP '07: Declarative Aspects of Multicore Programming, 2007.

Digital Library

[30]

M. A. Hammer, U. A. Acar, and Y. Chen. CEAL: a C-based language for self-adjusting computation. In ACM SIGPLAN Conference on Programming Language Design and Implementation, 2009.

Digital Library

[31]

M. A. Hammer, K. Y. Phang, M. Hicks, and J. S. Foster. Adapton: Composable, demand-driven incremental computation. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '14, pages 156--166, 2014. ISBN 978-1-4503-2784-8.

Digital Library

[32]

M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. SIGOPS Oper. Syst. Rev., 41(3):59--72, Mar. 2007. ISSN 0163-5980.

Digital Library

[33]

A. Jeffrey. LTL types FRP: linear-time temporal logic propositions as types, proofs as functional reactive programs. In PLPV '12: Proceedings of the sixth workshop on Programming languages meets program verification, pages 49--60, 2012. ISBN 978-1-4503-1125-0.

Digital Library

[34]

W. Jeltsch. Temporal logic with "until", functional reactive programming with processes, and concrete process categories. In Proceedings of the 7th Workshop on Programming Languages Meets Program Verification, PLPV '13, pages 69--78, 2013. ISBN 978-1-4503-1860-0.

Digital Library

[35]

U. Kang, C. E. Tsourakakis, A. P. Appel, C. Faloutsos, and J. Leskovec. Hadi: Mining radii of large graphs. TKDD, 5(2):8, 2011.

Digital Library

[36]

U. Kang, C. E. Tsourakakis, and C. Faloutsos. Pegasus: mining petascale graphs. Knowl. Inf. Syst., 27(2):303--325, 2011.

Digital Library

[37]

N. R. Krishnaswami. Higher-order functional reactive programming without spacetime leaks. SIGPLAN Not., 48(9):221--232, Sept. 2013. ISSN 0362-1340.

Digital Library

[38]

H. Liu, E. Cheng, and P. Hudak. Causal commutative arrows and their optimization. In Proceedings of the 14th ACM SIGPLAN international conference on Functional programming, ICFP '09, pages 35--46, 2009. ISBN 978-1-60558-332-7.

Digital Library

[39]

Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M. Hellerstein. Distributed GraphLab: a framework for machine learning and data mining in the cloud. VLDB Endow., 5(8):716--727, Apr. 2012.

Digital Library

[40]

G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: a system for large-scale graph processing. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, SIGMOD '10, pages 135--146, 2010.

Digital Library

[41]

S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis. Dremel: interactive analysis of web-scale datasets. Commun. ACM, 54(6):114--123, June 2011. ISSN 0001-0782.

Digital Library

[42]

D. G. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi. Naiad: A timely dataflow system. In Proc. of SOSP, pages 439--455, 2013.

Digital Library

[43]

A. Muthitacharoen, B. Chen, and D. Mazières. A low-bandwidth network file system. In SOSP, pages 174--187, 2001.

Digital Library

[44]

M. Odersky, M. Sulzmann, and M. Wehr. Type inference with constrained types. Theory and Practice of Object Systems, 5(1):35--55, 1999.

Digital Library

[45]

D. Peng and F. Dabek. Large-scale incremental processing using distributed transactions and notifications. In Proc. 9th Symposium on Operating Systems Design and Implementation (OSDI'10), 2010.

Digital Library

[46]

W. Pugh and T. Teitelbaum. Incremental computation via function caching. In Principles of Programming Languages, pages 315--328, 1989.

Digital Library

[47]

G. Ramalingam and T. Reps. A categorized bibliography on incremental computation. In Principles of Programming Languages, pages 502--510, 1993.

Digital Library

[48]

E. J. Riedy, H. Meyerhenke, D. A. Bader, D. Ediger, and T. G. Mattson. Analysis of streaming social networks and graphs on multicore architectures. In ICASSP, pages 5337--5340, 2012.

[49]

N. Sculthorpe and H. Nilsson. Keeping calm in the face of change. Higher Order Symbol. Comput., 23(2):227--271, June 2010. ISSN 1388-3690.

Digital Library

[50]

N. Sculthorpe and H. Nilsson. Safe functional reactive programming through dependent types. SIGPLAN Not., 44(9):23--34, Aug. 2009. ISSN 0362-1340.

Digital Library

[51]

O. Sümer, U. A. Acar, A. Ihler, and R. Mettu. Adaptive exact inference in graphical models. Journal of Machine Learning, 8:180--186, 2011.

[52]

J.-P. Talpin and P. Jouvelot. The type and effect discipline. Inf. Comput., 111(2):245--296, June 1994. ISSN 0890-5401.

Digital Library

[53]

K. Tangwongsan, H. Pucha, D. G. Andersen, and M. Kaminsky. Efficient similarity estimation for systems exploiting data redundancy. In INFOCOM, pages 1487--1495, 2010.

Digital Library

[54]

Z. Wan, W. Taha, and P. Hudak. Real-time FRP. SIGPLAN Not., 36 (10):146--156, 2001.

Digital Library

[55]

Z. Wan, W. Taha, and P. Hudak. Event-driven FRP. In Proceedings of the 4th International Symposium on Practical Aspects of Declarative Languages, PADL '02, pages 155--172, 2002.

Digital Library

[56]

D. M. Yellin and R. E. Strom. INC: a language for incremental computations. ACM Transactions on Programming Languages and Systems, 13(2):211--236, Apr. 1991.

Digital Library

Cited By

Chen SWu B(2020)Efficient counter-factual type error debuggingScience of Computer Programming10.1016/j.scico.2020.102544200(102544)Online publication date: Dec-2020
https://doi.org/10.1016/j.scico.2020.102544
Acar UAksenov VWestrick SScheideler CHajiaghayi M(2017)Brief AnnouncementProceedings of the 29th ACM Symposium on Parallelism in Algorithms and Architectures10.1145/3087556.3087595(275-277)Online publication date: 24-Jul-2017
https://dl.acm.org/doi/10.1145/3087556.3087595
Hammer MDunfield JHeadley KLabich NFoster JHicks MVan Horn D(2015)Incremental computation with namesACM SIGPLAN Notices10.1145/2858965.281430550:10(748-766)Online publication date: 23-Oct-2015
https://dl.acm.org/doi/10.1145/2858965.2814305
Show More Cited By

Index Terms

Functional programming for dynamic and large data with self-adjusting computation
1. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language features
      2. Language types
        Functional languages
2. Theory of computation
  1. Semantics and reasoning
    1. Program constructs

Recommendations

Functional programming for dynamic and large data with self-adjusting computation
ICFP '14

Combining type theory, language design, and empirical work, we present techniques for computing with large and dynamically changing datasets. Based on lambda calculus, our techniques are suitable for expressing a diverse set of algorithms on large ...
Self-adjusting computation: (an overview)
PEPM '09: Proceedings of the 2009 ACM SIGPLAN workshop on Partial evaluation and program manipulation

Many applications need to respond to incremental modifications to data. Being incremental, such modification often require incremental modifications to the output, making it possible to respond to them asymptotically faster than recomputing from ...
An experimental analysis of self-adjusting computation

Recent work on adaptive functional programming (AFP) developed techniques for writing programs that can respond to modifications to their data by performing change propagation. To achieve this, executions of programs are represented with dynamic ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICFP '14: Proceedings of the 19th ACM SIGPLAN international conference on Functional programming

August 2014

390 pages

ISBN:9781450328739

DOI:10.1145/2628136

General Chair:
Johan Jeuring
Universiteit Utrecht, The Netherlands
,
Program Chair:
Manuel M.T. Chakravarty
University of New South Wales, Australia

ACM SIGPLAN Notices Volume 49, Issue 9
ICFP '14
September 2014
361 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/2692915
Editors:
Mark W. Bailey
Hamilton College, Clinton, NY
,
Rajeev Balasubramonian
University of Utah
,
Al Davis
University of Utah
,
Sarita Adve
University of Illinois at Urbana-Champ
Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 August 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ICFP'14

Sponsor:

SIGPLAN

ICFP'14: ACM SIGPLAN International Conference on Functional Programming

September 1 - 6, 2014

Gothenburg, Sweden

Acceptance Rates

ICFP '14 Paper Acceptance Rate 28 of 85 submissions, 33%;

Overall Acceptance Rate 333 of 1,064 submissions, 31%

Upcoming Conference

ICFP '25

Sponsor:
sigplan

ACM SIGPLAN International Conference on Functional Programming

October 12 - 18, 2025

Singapore , Singapore

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
388
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen SWu B(2020)Efficient counter-factual type error debuggingScience of Computer Programming10.1016/j.scico.2020.102544200(102544)Online publication date: Dec-2020
https://doi.org/10.1016/j.scico.2020.102544
Acar UAksenov VWestrick SScheideler CHajiaghayi M(2017)Brief AnnouncementProceedings of the 29th ACM Symposium on Parallelism in Algorithms and Architectures10.1145/3087556.3087595(275-277)Online publication date: 24-Jul-2017
https://dl.acm.org/doi/10.1145/3087556.3087595
Hammer MDunfield JHeadley KLabich NFoster JHicks MVan Horn D(2015)Incremental computation with namesACM SIGPLAN Notices10.1145/2858965.281430550:10(748-766)Online publication date: 23-Oct-2015
https://dl.acm.org/doi/10.1145/2858965.2814305
Hammer MDunfield JHeadley KLabich NFoster JHicks MVan Horn DAldrich JEugster P(2015)Incremental computation with namesProceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications10.1145/2814270.2814305(748-766)Online publication date: 23-Oct-2015
https://dl.acm.org/doi/10.1145/2814270.2814305
Bransen JDijkstra ASwierstra SAsai KSagonas K(2015)Incremental Evaluation of Higher Order AttributesProceedings of the 2015 Workshop on Partial Evaluation and Program Manipulation10.1145/2678015.2682541(39-48)Online publication date: 13-Jan-2015
https://dl.acm.org/doi/10.1145/2678015.2682541
Cerny DDobes JNavratil V(2017)Functional chaining mechanism allowing definable models of electronic devices2017 European Conference on Circuit Theory and Design (ECCTD)10.1109/ECCTD.2017.8093250(1-4)Online publication date: Sep-2017
https://doi.org/10.1109/ECCTD.2017.8093250

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents