skip to main content
research-article

Making an RDBMS data scientist friendly: advanced in-database interactive analytics with visualization support

Published:01 August 2019Publication History
Skip Abstract Section

Abstract

We are currently witnessing the rapid evolution and adoption of various data science frameworks that function external to the database. Any support from conventional RDBMS implementations for data science applications has been limited to procedural paradigms such as user-defined functions (UDFs) that lack exploratory programming support. Therefore, the current status quo is that during the exploratory phase, data scientists usually use the database system as the "data storage" layer of the data science framework, whereby the majority of computation and analysis is performed outside the database, e.g., at the client node. We demonstrate AIDA, an in-database framework for data scientists. AIDA allows users to write interactive Python code using a development environment such as a Jupyter notebook. The actual execution itself takes place inside the database (near-data), where a server component of AIDA, that resides inside the embedded Python interpreter of the RDBMS, manages the data sets and computations. The demonstration will also show the visualization capabilities of AIDA where the progress of computation can be observed through live updates. Our evaluations show that AIDA performs several times faster compared to contemporary external data science frameworks, but is much easier to use for exploratory development compared to database UDFs.

References

  1. J. V. D'Silva, F. De Moor, and B. Kemme. AIDA-Abstraction for Advanced In-Database Analytics. PVLDB, 11(11):1400--1413, 2018. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. J. Lajus and H. Mühleisen. Efficient Data Management and Statistics with Zero-Copy Integration. In SSDBM, pages 12:1--12:10. ACM, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. W. McKinney. pandas: a Foundational Python Library for Data Analysis and Statistics. Python for High Performance and Scientific Computing, pages 1--9, 2011.Google ScholarGoogle Scholar
  4. S. Melnik, A. Adya, and P. A. Bernstein. Compiling Mappings to Bridge Applications and Databases. Transactions on Database Systems, 33(4):22, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. H. Mühleisen and T. Lumley. Best of Both Worlds: Relational Databases and Statistics. In SSDBM, pages 32:1--32:4. ACM, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. M. Raasveldt and H. Mühleisen. Vectorized UDFs in Column-Stores. In SSDBM, pages 16:1--16:12. ACM, 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. T. Vincenty. Direct and Inverse Solutions of Geodesics on the Ellipsoid with Application of Nested Equations. Survey Review, 23(176):88--93, 1975.Google ScholarGoogle Scholar
  8. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: Cluster Computing With Working Sets. HotCloud, 10(10--10):95, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library

Recommendations

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Sign in

Full Access

  • Published in

    cover image Proceedings of the VLDB Endowment
    Proceedings of the VLDB Endowment  Volume 12, Issue 12
    August 2019
    547 pages

    Publisher

    VLDB Endowment

    Publication History

    • Published: 1 August 2019
    Published in pvldb Volume 12, Issue 12

    Qualifiers

    • research-article
  • Article Metrics

    • Downloads (Last 12 months)36
    • Downloads (Last 6 weeks)4

    Other Metrics

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader
About Cookies On This Site

We use cookies to ensure that we give you the best experience on our website.

Learn more

Got it!