Abstract
An incremental computation updates its result based on a change to its input, which is often an order of magnitude faster than a recomputation from scratch. In particular, incrementalization can make expensive computations feasible for settings that require short feedback cycles, such as interactive systems, IDEs, or (soft) real-time systems.
This paper presents i3QL, a general-purpose programming language for specifying incremental computations. i3QL provides a declarative SQL-like syntax and is based on incremental versions of operators from relational algebra, enriched with support for general recursion. We integrated i3QL into Scala as a library, which enables programmers to use regular Scala code for non-incremental subcomputations of an i3QL query and to easily integrate incremental computations into larger software projects. To improve performance, i3QL optimizes user-defined queries by applying algebraic laws and partial evaluation. We describe the design and implementation of i3QL and its optimizations, demonstrate its applicability, and evaluate its performance.
Supplemental Material
Available for Download
- U. A. Acar, G. E. Blelloch, M. Blume, R. Harper, and K. Tangwongsan. An experimental analysis of self-adjusting computation. ACM Trans. Program. Lang. Syst., 32(1):3:1--3:53, Nov. 2009. Google Scholar
Digital Library
- P. Bhatotia, A. Wieder, R. Rodrigues, U. A. Acar, and R. Pasquin. Incoop: Mapreduce for incremental computations. In Proceedings of the 2nd ACM Symposium on Cloud Computing, SOCC '11, pages 7:1--7:14, New York, NY, USA, 2011. ACM. Google Scholar
Digital Library
- J. A. Blakeley, N. Coburn, and P.-V. Larson. Updating derived relations: detecting irrelevant and autonomously computable updates. ACM Trans. Database Syst., 14(3):369--400, Sept. 1989. Google Scholar
Digital Library
- J. A. Blakeley, P.-A. Larson, and F. W. Tompa. Efficiently updating materialized views. SIGMOD Rec., 15(2):61--71, June 1986. Google Scholar
Digital Library
- S. Ceri and J. Widom. Deriving production rules for incremental view maintenance. In Proceedings of the 17th International Conference on Very Large Data Bases, VLDB '91, pages 577--589, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc. Google Scholar
Digital Library
- S. Chaudhuri, R. Krishnamurthy, S. Potamianos, and K. Shim. Optimizing queries with materialized views. In Data Engineering, 1995. Proceedings of the Eleventh International Conference on, pages 190--200, mar 1995. Google Scholar
Digital Library
- N. Chomsky. Three models for the description of language. Information Theory, IRE Transactions on, 2(3):113--124, September 1956.Google Scholar
- G. Cooper and S. Krishnamurthi. Embedding dynamic dataflow in a call-by-value language. In P. Sestoft, editor, Programming Languages and Systems, volume 3924 of Lecture Notes in Computer Science, pages 294--308. Springer Berlin Heidelberg, 2006. Google Scholar
Digital Library
- C. Demetrescu, I. Finocchi, and A. Ribichini. Reactive imperative programming with dataflow constraints. In Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications, OOPSLA '11, pages 407--426, New York, NY, USA, 2011. ACM. Google Scholar
Digital Library
- M. Eichberg, M. Kahl, D. Saha, M. Mezini, and K. Ostermann. Automatic incrementalization of prolog based static analyses. In Proceedings of the 9th International Conference on Practical Aspects of Declarative Languages, PADL'07, pages 109--123, Berlin, Heidelberg, 2007. Springer-Verlag. Google Scholar
Digital Library
- B. N. Freeman-Benson. Kaleidoscope: Mixing objects, constraints, and imperative programming. In Proceedings of the European Conference on Object-oriented Programming on Object-oriented Programming Systems, Languages, and Applications, OOPSLA/ECOOP '90, pages 77--88, New York, NY, USA, 1990. ACM. Google Scholar
Digital Library
- M. Garcia, A. Izmaylova, and S. Schupp. Extending Scala with database query capability. Journal of Object Technology, 9(4):45--68, 2010.Google Scholar
Cross Ref
- P. G. Giarrusso, K. Ostermann, M. Eichberg, R. Mitschke, T. Rendel, and C. Kästner. Reify your collection queries for modularity and speed! In Proceedings of the 12th Annual International Conference on Aspect-oriented Software Development, AOSD '13, pages 1--12, New York, NY, USA, 2013. ACM. Google Scholar
Digital Library
- P. M. D. Gray, L. Kerschberg, P. J. H. King, and A. Poulovassilis. The Functional Approach to Data Management: Modeling, Analyzing and Integrating Heterogeneous Data. Springer Publishing Company, Incorporated, 1st edition, 2010. Google Scholar
Digital Library
- T. Grust, M. Mayr, J. Rittinger, and T. Schreiber. Ferry: database-supported program execution. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, SIGMOD '09, pages 1063--1066, New York, NY, USA, 2009. ACM. Google Scholar
Digital Library
- A. Gupta and I. Mumick. Materialized Views: Techniques, Implementations, and Applications. MIT Press, 1999. Google Scholar
Digital Library
- A. Gupta, I. S. Mumick, and K. A. Ross. Adapting materialized views after redefinitions. SIGMOD Rec., 24(2):211--222, May 1995. Google Scholar
Digital Library
- A. Gupta, I. S. Mumick, and V. S. Subrahmanian. Maintaining views incrementally. In Proceedings of the 1993 ACM SIGMOD international conference on Management of data, SIGMOD '93, pages 157--166, New York, NY, USA, 1993. ACM. Google Scholar
Digital Library
- D. Hovemeyer and W. Pugh. Finding bugs is easy. SIGPLAN Not., 39(12):92--106, Dec. 2004. Google Scholar
Digital Library
- P. Hudak. Modular domain specific languages and tools. In Proceedings of International Conference on Software Reuse (ICSR), pages 134--142. IEEE, 1998. Google Scholar
Digital Library
- M. Kay. Readings in natural language processing. chapter Algorithm schemata and data structures in syntactic processing, pages 35--70. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1986. Google Scholar
Digital Library
- D. Leijen and E. Meijer. Domain specific embedded compilers. In Proceedings of the 2nd conference on Domain-specific languages, DSL '99, pages 109--122, New York, NY, USA, 1999. ACM. Google Scholar
Digital Library
- V. Lifschitz. Principles of knowledge representation. chapter Foundations of Logic Programming, pages 69--127. Center for the Study of Language and Information, Stanford, CA, USA, 1996. Google Scholar
Digital Library
- Y. A. Liu, S. D. Stoller, M. Gorbovitski, T. Rothamel, and Y. E. Liu. Incrementalization across object abstraction. In Proceedings of the 20th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications, OOPSLA '05, pages 473--486, New York, NY, USA, 2005. ACM. Google Scholar
Digital Library
- LiveLinq Web site. http://www.componentone.com/SuperProducts/LiveLinq/.Google Scholar
- I. Maier and M. Odersky. Higher-order reactive programming with incremental lists. In Proceedings of the 27th European conference on Object-Oriented Programming, ECOOP'13, pages 707--731, Berlin, Heidelberg, 2013. Springer-Verlag. Google Scholar
Digital Library
- S. Melnik, A. Adya, and P. A. Bernstein. Compiling mappings to bridge applications and databases. ACM Trans. Database Syst., 33(4):22:1--22:50, Dec. 2008. Google Scholar
Digital Library
- L. A. Meyerovich, A. Guha, J. Baskin, G. H. Cooper, M. Greenberg, A. Bromfield, and S. Krishnamurthi. Flapjax: A programming language for Ajax applications. SIGPLAN Not., 44(10):1--20, Oct. 2009. Google Scholar
Digital Library
- R. Mitschke, M. Eichberg, M. Mezini, A. Garcia, and I. Macia. Modular specification and checking of structural dependencies. In Proceedings of the 12th Annual International Conference on Aspect-oriented Software Development, AOSD '13, pages 85--96, New York, NY, USA, 2013. ACM. Google Scholar
Digital Library
- B. A. Myers, R. G. McDaniel, R. C. Miller, A. S. Ferrency, A. Faulring, B. D. Kyle, A. Mickish, A. Klimovitski, and P. Doane. The Amulet environment: New models for effective user interface software development. IEEE Trans. Softw. Eng., 23(6):347--365, 1997. Google Scholar
Digital Library
- V. Nerella, S. Surapaneni, S. Madria, and T. Weigert. Exploring optimization and caching for efficient collection operations. Automated Software Engineering, 21(1):3--40, 2014. Google Scholar
Digital Library
- R. Paige. Applications of finite differencing to database integrity control and query/transaction optimization. In Advances in Data Base Theory, pages 171--209, 1982.Google Scholar
- T. Palpanas, R. Sidle, R. Cochrane, and H. Pirahesh. Incremental maintenance for non-distributive aggregate functions. In Proceedings of the 28th international conference on Very Large Data Bases, VLDB '02, pages 802--813. VLDB Endowment, 2002. Google Scholar
Digital Library
- F. Pfenning and C. Elliot. Higher-order abstract syntax. SIGPLAN Not., 23(7):199--208, June 1988. Google Scholar
Digital Library
- X. Qian and G.Wiederhold. Incremental recomputation of active relational expressions. IEEE Transactions on Knowledge and Data Engineering, 3(3):337--341, 1991. Google Scholar
Digital Library
- G. Ramalingam and T. Reps. A categorized bibliography on incremental computation. In Proceedings of the 20th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, POPL '93, pages 502--510, New York, NY, USA, 1993. ACM. Google Scholar
Digital Library
- T. Rompf and M. Odersky. Lightweight modular staging: a pragmatic approach to runtime code generation and compiled dsls. SIGPLAN Not., 46(2):127--136, Oct. 2010. Google Scholar
Digital Library
- T. Rompf and M. Odersky. Lightweight modular staging: a pragmatic approach to runtime code generation and compiled dsls. Commun. ACM, 55(6):121--130, 2012. Google Scholar
Digital Library
- T. Rothamel and Y. A. Liu. Generating incremental implementations of object-set queries. In Proceedings of the 7th International Conference on Generative Programming and Component Engineering, GPCE '08, pages 55--66, New York, NY, USA, 2008. ACM. Google Scholar
Digital Library
- N. Roussopoulos. An incremental access method for view-cache: concept, algorithms, and cost analysis. ACM Trans. Database Syst., 16(3):535--563, Sept. 1991. Google Scholar
Digital Library
- K. Sagonas, T. Swift, and D. S. Warren. Xsb as an efficient deductive database engine. SIGMOD Rec., 23(2):442--453, May 1994. Google Scholar
Digital Library
- D. Saha and C. Ramakrishnan. Incremental evaluation of tabled logic programs. In C. Palamidessi, editor, Logic Programming, volume 2916 of Lecture Notes in Computer Science, pages 392--406. Springer Berlin Heidelberg, 2003.Google Scholar
Cross Ref
- D. Saha and C. Ramakrishnan. Symbolic support graph: A space efficient data structure for incremental tabled evaluation. In M. Gabbrielli and G. Gupta, editors, Logic Programming, volume 3668 of Lecture Notes in Computer Science, pages 235--249. Springer Berlin Heidelberg, 2005. Google Scholar
Digital Library
- G. Salvaneschi, G. Hintz, and M. Mezini. REScala: Bridging between object-oriented and functional style in reactive applications. In Proceedings of the 13th International Conference on Modularity, MODULARITY '14, pages 25--36, New York, NY, USA, 2014. ACM. Google Scholar
Digital Library
- A. Shankar and R. Bodík. Ditto: automatic incrementalization of data structure invariant checks (in Java). In Proceedings of the 2007 ACM SIGPLAN conference on Programming language design and implementation, PLDI '07, pages 310--319, New York, NY, USA, 2007. ACM. Google Scholar
Digital Library
- O. Shmueli and A. Itai. Maintenance of views. In Proceedings of the 1984 ACM SIGMOD international conference on Management of data, SIGMOD '84, pages 240--255, New York, NY, USA, 1984. ACM. Google Scholar
Digital Library
- O. Sumer, U. Acar, A. T. Ihler, and R. R. Mettu. Efficient bayesian inference for dynamically changing graphs. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 1441--1448. MIT Press, Cambridge, MA, 2008.Google Scholar
- J. D. Ullman, H. Garcia-Molina, and J. Widom. Database Systems: The Complete Book. Prentice Hall PTR, Upper Saddle River, NJ, USA, 1st edition, 2001. Google Scholar
Digital Library
- G.Wachsmuth, G. D. P. Konat, V. A. Vergu, D. M. Groenewegen, and E. Visser. A language independent task engine for incremental name and type analysis. In SLE, volume 8225 of Lecture Notes in Computer Science, pages 260--280. Springer, 2013.Google Scholar
Cross Ref
- T. A. Wagner and S. L. Graham. Efficient and flexible incremental parsing. ACM Trans. Program. Lang. Syst., 20(5):980--1013, Sept. 1998. Google Scholar
Digital Library
- D.Willis, D. J. Pearce, and J. Noble. Efficient object querying for Java. In Proceedings of the 20th European conference on Object-Oriented Programming, ECOOP'06, pages 28--49, Berlin, Heidelberg, 2006. Springer-Verlag. Google Scholar
Digital Library
- D. Willis, D. J. Pearce, and J. Noble. Caching and incrementalisation in the Java query language. SIGPLAN Not., 43(10):1--18, Oct. 2008. Google Scholar
Digital Library
- L. Wong. Kleisli, a functional query system. Journal of Functional Programming, 10:19--56, 0 2000. Google Scholar
Digital Library
Index Terms
i3QL: language-integrated live data views
Recommendations
i3QL: language-integrated live data views
OOPSLA '14: Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & ApplicationsAn incremental computation updates its result based on a change to its input, which is often an order of magnitude faster than a recomputation from scratch. In particular, incrementalization can make expensive computations feasible for settings that ...
Reactive Imperative Programming with Dataflow Constraints
Dataflow languages provide natural support for specifying constraints between objects in dynamic applications, where programs need to react efficiently to changes in their environment. In this article, we show that one-way dataflow constraints, largely ...
Efficiency by Incrementalization: An Introduction
Incremental computation takes advantage of repeated computations on inputs that differ slightly from one another, computing each output efficiently by exploiting the previous output. This paper gives an overview of a general and systematic approach to ...







Comments