skip to main content
research-article
Free Access

DataStorm: Coupled, Continuous Simulations for Complex Urban Environments

Published:21 July 2021Publication History
Skip Abstract Section

Abstract

Urban systems are characterized by complexity and dynamicity. Data-driven simulations represent a promising approach in understanding and predicting complex dynamic processes in the presence of shifting demands of urban systems. Yet, today’s silo-based, de-coupled simulation engines fail to provide an end-to-end view of the complex urban system, preventing informed decision-making. In this article, we present DataStorm to support integration of existing simulation, analysis and visualization components into integrated workflows. DataStorm provides a flow engine, DataStorm-FE, for coordinating data and decision flows among multiple actors (each representing a model, analytic operation, or a decision criterion) and enables ensemble planning and optimization across cloud resources. DataStorm provides native support for simulation ensemble creation through parameter space sampling to decide which simulations to run, as well as distributed instantiation and parallel execution of simulation instances on cluster resources. Recognizing that simulation ensembles are inherently sparse relative to the potential parameter space, we also present a density-boosting partition-stitch sampling scheme to increase the effective density of the simulation ensemble through a sub-space partitioning scheme, complemented with an efficient stitching mechanism that leverages partial and imperfect knowledge from partial dynamical systems to effectively obtain a global view of the complex urban process being simulated.

References

  1. Ilkay Altintas, Chad Berkley, Efrat Jaeger, Matthew Jones, Bertram Ludäscher, and Stephen Mock. 2004. Kepler: An extensible system for design and execution of scientific workflows. In Proceedings of the International Conference on Scientific and Statistical Database Management (SSDBM’04), Vol. 16. 423–424. DOI:https://doi.org/10.1109/SSDBM.2004.44 Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Ilkay Altintas, Jianwu Wang, Daniel Crawl, and Weizhong Li. 2012. Challenges and approaches for distributed workflow-driven analysis of large-scale biological data: Vision paper. In Proceedings of the Joint EDBT/ICDT Workshops (EDBT-ICDT’12). ACM, New York, NY, 73–78. DOI:https://doi.org/10.1145/2320765.2320791 Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Duygu Balcan, Bruno Gonçalves, Hao Hu, José J. Ramasco, Vittoria Colizza, and Alessandro Vespignani. 2010. Modeling the spatial spread of infectious diseases: The GLobal epidemic and mobility computational model. J. Comput. Sci. 1, 3 (Aug. 2010), 132–145. DOI:https://doi.org/10.1016/j.jocs.2010.07.002Google ScholarGoogle Scholar
  4. Michael Batty. 2013. Visually-driven urban simulation: Exploring fast and slow change in residential location. Environ. Plan. A: Econ. Space 45, 3 (2013), 532–552.Google ScholarGoogle Scholar
  5. Hans Walter Behrens, K. Selçuk Candan, Xilun Chen, Ashish Gadkari, Yash Garg, Mao-Lin Li, Xinsheng Li, Sicong Liu, Nicholas Martinez, Jiayong Mo, Elliot Nester, Silvestro Poccia, Manjusha Ravindranath, and Maria Luisa Sapino. 2018. Datastorm-FE: A data- and decision-flow and coordination engine for coupled simulation ensembles. Proc. VLDB Endow. 11, 12 (Aug. 2018), 1906–1909. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Hans Walter Behrens, Mao-Lin Li, Ashish Gadkari, Yash Garg, Xilun Chen, Sicong Liu, and K. Selçuk Candan. 2019. Load-adaptive continuous coupled-simulation ensembles with DataStorm and Chameleon. In Proceedings of the Chameleon User Meeting.Google ScholarGoogle Scholar
  7. Chun-Hung Chen and Loo Hay Lee. 2010. Stochastic Simulation Optimization - An Optimal Computing Budget Allocation. System Engineering and Operations Research, Vol. 1. WorldScientific. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Committee on Environment and Natural Resources. 2008. Grand challenges for disaster reduction. Executive Office of the President (2008). https://www.sdr.gov/docs/SDRGrandChallengesforDisasterReduction.pdf.Google ScholarGoogle Scholar
  9. Greg Cope. 2017. Programming the Lorenz Attractor. Retrieved from https://www.algosome.com/articles/lorenz-attractor-programming-code.htmlGoogle ScholarGoogle Scholar
  10. Laurent Guillaume Courty, Adrián Pedrozo-Acuña, and Paul David Bates. 2017. Itzï (Version 17.1): An open-source, distributed GIS model for dynamic flood simulation. Geosci. Model Dev. 10, 4 (May 2017), 1835–1847. DOI:https://doi.org/10.5194/gmd-10-1835-2017Google ScholarGoogle Scholar
  11. Christopher Davis, Wei Wang, Shuyi S. Chen, Yongsheng Chen, Kristen Corbosiero, Mark DeMaria, Jimy Dudhia, Greg Holland, Joe Klemp, John Michalakes, Heather Reeves, Richard Rotunno, Chris Snyder, and Qingnong Xiao. 2008. Prediction of landfalling hurricanes with the advanced hurricane WRF model. Month. Weath. Rev. 136, 6 (June 2008), 1990–2005. DOI:https://doi.org/10.1175/2007MWR2085.1Google ScholarGoogle Scholar
  12. Stefan B. Edlund, Matthew A. Davis, and James H. Kaufman. 2010. The spatiotemporal epidemiological modeler. In Proceedings of the 1st ACM International Health Informatics Symposium (IHI’10). ACM, New York, NY, 817–820. DOI:https://doi.org/10.1145/1882992.1883115 Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Christoph Federrath. 2017. Java Applets. Retrieved from http://www.ita.uni-heidelberg.de/˜chfeder/applets.shtml?lang=en.Google ScholarGoogle Scholar
  14. Ronald Aylmer Fisher. 1935. The Design of Eexperiments. Oliver and Boyd.Google ScholarGoogle Scholar
  15. Ignacio Garcia-Dorado, Daniel G. Aliaga, Saiprasanth Bhalachandran, Paul Schmid, and Dev Niyogi. 2017. Fast weather simulation for inverse procedural design of 3D urban models. ACM Trans. Graph. 36, 2 (2017), 21:1–21:19. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Michael Gollner, Arnaud Trouve, Ilkay Altintas, Jessica Block, Raymond de Callafon, Craig Clements, Anna Cortes, Evan Ellicott, Jean Baptiste Filippi, Mark Finney, Kayo Ide, Mary Ann Jenkins, Daniel Jimenez, Christopher Lautenberger, Jan Mandel, Melanie Rochoux, and Albert Simeoni. 2015. Towards data-driven operational wildfire spread modeling: A report of the NSF-funded WIFIRE workshop. (Jan. 2015). DOI:https://doi.org/10.13016/M2Z70RGoogle ScholarGoogle Scholar
  17. Richard A. Harshman. 1970. Foundations of The PARAFAC procedure: Models and conditions for an ‘explanatory” multimodal factor analysis. In UCLA Working Papers in Phonetics.Google ScholarGoogle Scholar
  18. Duncan Hull, Katy Wolstencroft, Robert Stevens, Carole Goble, Mathew R. Pocock, Peter Li, and Tom Oinn. 2006. Taverna: A tool for building and running workflows of services. Nucleic Acids Res. 34, Web Server issue (July 2006), W729–W732. DOI:https://doi.org/10.1093/nar/gkl320Google ScholarGoogle Scholar
  19. C. Jelesnianski, Jye Chen, W. Shaffer, and A. Gilad. 1984. SLOSH—A hurricane storm surge forecast model. In Proceedings of the OCEANS 1984 Conference. 314–317. DOI:https://doi.org/10.1109/OCEANS.1984.1152341Google ScholarGoogle Scholar
  20. I. Jeon, E. E. Papalexakis, U. Kang, and C. Faloutsos. 2015. HaTen2: Billion-scale tensor decompositions. In Proceedings of the IEEE 31st International Conference on Data Engineering. 1047–1058. DOI:https://doi.org/10.1109/ICDE.2015.7113355Google ScholarGoogle Scholar
  21. N. L. Johnson. 1961. Sequential analysis: A survey. J. Roy. Statist. Soc. Series A (General) 124, 3 (1961), 372–411. DOI:10.2307/2343243Google ScholarGoogle ScholarCross RefCross Ref
  22. Kate Keahey, Pierre Riteau, Dan Stanzione, Tim Cockerill, Joe Mambretti, Paul Rad, and Paul Ruth. 2018. Chameleon: A scalable production testbed for computer science research. In Contemporary High Performance Computing: From Petascale toward Exascale. CRC Press.Google ScholarGoogle Scholar
  23. Katarzyna Keahey, Mauricio Tsugawa, Andrea Matsunaga, and Jose Fortes. 2009. Sky computing. IEEE Internet Comput. 13, 5 (2009), 43–51. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Ari Keranen. 2008. Opportunistic network environment simulator. In Proceedings of the 2nd International Conference on Simulation Tools and Techniques (SIMUTools'09). https://www.netlab.tkk.fi/tutkimus/dtn/theone/pub/the_one.pdf.Google ScholarGoogle Scholar
  25. Mijung Kim and K. Selçuk Candan. 2016. Decomposition-by-normalization (DBN): Leveraging approximate functional dependencies for efficient CP and Tucker decompositions. Data Mining Knowl. Discov. 30, 1 (Jan. 2016), 1–46. DOI:https://doi.org/10.1007/s10618-015-0401-6. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Stefan Klus, Patrick Gelß, Sebastian Peitz, and Christof Schütte. 2018. Tensor-based dynamic mode decomposition. Nonlinearity 31, 7 (June 2018), 3359–3380. DOI:https://doi.org/10.1088/1361-6544/aabc8f.Google ScholarGoogle Scholar
  27. T. Kolda and B. Bader. 2009. Tensor decompositions and applications. SIAM Rev. 51, 3 (Aug. 2009), 455–500. DOI:https://doi.org/10.1137/07070111X. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. T. G. Kolda and J. Sun. 2008. Scalable tensor decompositions for multi-aspect data mining. In Proceedings of the 8th IEEE International Conference on Data Mining. 363–372. DOI:https://doi.org/10.1109/ICDM.2008.89. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. X. Li, K. S. Candan, and M. L. Sapino. 2018. M2TD: Multi-task tensor decomposition for sparse ensemble simulations. In Proceedings of the IEEE 34th International Conference on Data Engineering (ICDE’18). 1144–1155.Google ScholarGoogle Scholar
  30. X. Li, S. Huang, K. S. Candan, and M. L. Sapino. 2016. 2PCP: Two-phase CP decomposition for billion-scale dense tensors. In Proceedings of the IEEE 32nd International Conference on Data Engineering (ICDE’16). 835–846.Google ScholarGoogle Scholar
  31. Sicong Liu, Yash Garg, K. Selcuk Candan, Maria Luisa Sapino, and Gerardo Chowell-Puente. 2015. NOTES2: Networks-of-traces for epidemic spread simulations. In Proceedings of the Workshops at the 29th AAAI Conference on Artificial Intelligence.Google ScholarGoogle Scholar
  32. Sicong Liu, Silvestro Poccia, K. Selçuk Candan, Gerardo Chowell, and Maria Luisa Sapino. 2016. epiDMS: Data management and analytics for decision-making from epidemic spread simulation ensembles. J. Infect. Dis. 214, suppl_4 (2016), S427–S432.Google ScholarGoogle Scholar
  33. C. D. Tharindu Mathew, Paulo R. Knob, Soraia Raupp Musse, and Daniel G. Aliaga. 2019. Urban walkability design using virtual population simulation. Comput. Graph. Forum 38, 1 (2019), 455–469.Google ScholarGoogle Scholar
  34. Panayiotis Neophytou, Panos K. Chrysanthis, and Alexandros Labrinidis. 2011. CONFLuEnCE: CONtinuous workFLow ExeCution Engine. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD’11). ACM, New York, NY, 1311–1314. DOI:https://doi.org/10.1145/1989323.1989485 Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. K. B. Olsen, S. M. Day, L. A. Dalguer, J. Mayhew, Y. Cui, J. Zhu, V. M. Cruz-Atienza, D. Roten, P. Maechling, T. H. Jordan, D. Okaya, and A. Chourasia. 2009. ShakeOut-D: Ground motion estimates using an ensemble of large earthquakes on the Southern San Andreas fault with spontaneous rupture propagation. Geophys. Res. Lett. 36, 4 (2009). DOI:https://doi.org/10.1029/2008GL036832Google ScholarGoogle Scholar
  36. Evangelos E. Papalexakis, Christos Faloutsos, and Nicholas D. Sidiropoulos. 2012. ParCube: Sparse parallelizable tensor decompositions. In Machine Learning and Knowledge Discovery in Databases (Lecture Notes in Computer Science), Peter A. Flach, Tijl De Bie, and Nello Cristianini (Eds.). Springer Berlin, 521–536. Google ScholarGoogle Scholar
  37. Giulia Pedrielli, Yinchao Zhu, and Loo Hay Lee. 2015. Single-run simulation optimization through time dilation and optimal computing budget allocation. http://www.smmso.org/SMMSO2015/Downloads/Giulia%20Pedrielli.pdf.Google ScholarGoogle Scholar
  38. Giulia Pedrielli, Yinchao Zhu, Loo Hay Lee, and Haobin Li. 2016. Empirical analysis of the performance of variance estimators in sequential single-run ranking & selection: The case of time dilation algorithm. In Proceedings of the Winter Simulation Conference (WSC’16). IEEE Press, Piscataway, NJ, 738–748. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Silvestro Poccia, Maria Luisa Sapino, Sicong Liu, Xilun Chen, Yash Garg, Shengyu Huang, Jung Hyun Kim, Xinsheng Li, Parth Nagarkar, and Kasim Candan. 2017. SIMDMS: Data management and analysis to support decision making through large simulation ensembles. In Proceedings of the 20th International Conference on Extending Database Technology: Advances in Database Technology (EDBT’17). OpenProceedings.org, 582–585. DOI:https://doi.org/10.5441/002/edbt.2017.75.Google ScholarGoogle Scholar
  40. Mark Rogers, Lei Li, and Stuart J. Russell. 2013. Multilinear dynamical systems for tensor time series. In Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 2634–2642. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. L. W. Schruben. 1997. Simulation optimization using simultaneous replications and event time dilation. In Proceedings of the Winter Simulation Conference. 177–180. DOI:https://doi.org/10.1109/WSC.1997.640394. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. EmitLab DataStorm Team. 2018. DataStorm demo, v1. Retrieved from https://www.youtube.com/watch?v=L74jwa78Dg4.Google ScholarGoogle Scholar
  43. EmitLab DataStorm Team. 2018. DataStorm release, v1. Retrieved from https://github.com/EmitLab/DataStorm-Release.Google ScholarGoogle Scholar
  44. Ledyard R. Tucker. 1966. Some mathematical notes on three-mode factor analysis. Psychometrika 31, 3 (Sept. 1966), 279–311. DOI:https://doi.org/10.1007/BF02289464.Google ScholarGoogle ScholarCross RefCross Ref
  45. Paul Waddell. 2002. UrbanSim: Modeling urban development for land use, transportation, and environmental planning. J. Amer. Plan. Assoc. 68, 3 (2002), 297–314.Google ScholarGoogle ScholarCross RefCross Ref
  46. Denis Weerasiri, Moshe Chai Barukh, Boualem Benatallah, Quan Z. Sheng, and Rajiv Ranjan. 2017. A taxonomy and survey of cloud resource orchestration techniques. Comput. Surv. 50, 2 (May 2017), 1–41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Ming Zhao, K. Selçuk Candan, Huan Liu, Hasan Davulcu, and Fengbo Ren. 2016. NSF Award Search: Award#1629888 - II-NEW: GEARS - An Infrastructure for Energy-Efficient Big Data Research on Heterogeneous and Dynamic Data. Retrieved from https://nsf.gov/awardsearch/showAward?AWD_ID=1629888Google ScholarGoogle Scholar

Index Terms

  1. DataStorm: Coupled, Continuous Simulations for Complex Urban Environments

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM/IMS Transactions on Data Science
          ACM/IMS Transactions on Data Science  Volume 2, Issue 3
          August 2021
          302 pages
          ISSN:2691-1922
          DOI:10.1145/3465442
          Issue’s Table of Contents

          Copyright © 2021 Association for Computing Machinery.

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 21 July 2021
          • Accepted: 1 January 2021
          • Revised: 1 November 2020
          • Received: 1 June 2019
          Published in tds Volume 2, Issue 3

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed
        • Article Metrics

          • Downloads (Last 12 months)94
          • Downloads (Last 6 weeks)13

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!