skip to main content
research-article

A Dynamic Data-throttling Approach to Minimize Workflow Imbalance

Authors Info & Claims
Published:07 May 2019Publication History
Skip Abstract Section

Abstract

Scientific workflows enable scientists to undertake analysis on large datasets and perform complex scientific simulations. These workflows are often mapped onto distributed and parallel computational infrastructures to speed up their executions. Prior to its execution, a workflow structure may suffer transformations to accommodate the computing infrastructures, normally involving task clustering and partitioning. However, these transformations may cause workflow imbalance because of the difference between execution task times (runtime imbalance) or because of unconsidered data dependencies that lead to data locality issues (data imbalance). In this article, to mitigate these imbalances, we enhance the workflow lifecycle process in use by introducing a workflow imbalance phase that quantifies workflow imbalance after the transformations. Our technique is based on structural analysis of Petri nets, obtained by model transformation of a data-intensive workflow, and Linear Programming techniques. Our analysis can be used to assist workflow practitioners in finding more efficient ways of transforming and scheduling their workflows. Moreover, based on our analysis, we also propose a technique to mitigate workflow imbalance by data throttling. Our approach is based on autonomic computing principles that determine how data transmission must be throttled throughout workflow jobs. Our autonomic data-throttling approach mainly monitors the execution of the workflow and recompute data-throttling values when certain watchpoints are reached and time derivation is observed. We validate our approach by a formal proof and by simulations along with the Montage workflow. Our findings show that a dynamic data-throttling approach is feasible, does not introduce a significant overhead, and minimizes the usage of input buffers and network bandwidth.

References

  1. Wil M. P. van der Aalst, Alexander Hirnschall, and H. M. W. (Eric) Verbeek. 2002. An alternative way to analyze workflow graphs. In Proceedings of the 14th International Conference on Advanced Information Systems Engineering (CAiSE’02). Springer-Verlag, London, 535--552. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. M. Ajmone Marsan, G. Balbo, G. Conte, S. Donatelli, and G. Franceschinis. 1995a. Modelling with Generalized Stochastic Petri Nets. John Wiley 8 Sons. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. M. Ajmone Marsan, G. Balbo, G. Conte, S. Donatelli, and G. Franceschinis. 1995b. Modelling with Generalized Stochastic Petri Nets. John Wiley 8 Sons. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. L. Aversano, A. Cimitile, P. Gallucci, and M. L. Villani. 2002. FlowManager: A workflow management system based on Petri nets. In Proceedings of the 26th Annual International Computer Software and Applications Conference (COMPSAC’02). 1054--1059. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. G. Bruce Berriman, Ewa Deelman, John Good, Joseph C. Jacob, Daniel S. Katz, Anastasia C. Laity, Thomas A. Prince, Gurmeet Singh, and Mei-Hui Su. 2007. Generating complex astronomy workflows. In Workflows for e-Science, Ian J. Taylor, Ewa Deelman, Dennis B. Gannon, and Matthew Shields (Eds.). Springer, London, 19--38.Google ScholarGoogle Scholar
  6. J. Campos and M. Silva. 1992. Structural techniques and performance bounds of stochastic Petri net models. In Proceedings of the Advances in Petri Nets 1992, G. Rozenberg (Ed.). Lecture Notes in Computer Science, vol 609. Springer, Berlin, Heidelberg. 352--391. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. J. Carmona, J. Júlvez, J. Cortadella, and M. Kishinevsky. 2009. Scheduling synchronous elastic designs. In Proceedings of the 2009 Application of Concurrency to System Design Conference (ACSD’09). Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Henri Casanova, Arnaud Legrand, and Martin Quinson. 2008. SimGrid: A generic framework for large-scale distributed experiments. In Proceedings of the 10th IEEE International Conference on Computer Modeling and Simulation. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Weiwei Chen, Rafael Ferreira da Silva, Ewa Deelman, and Rizos Sakellariou. 2015. Using imbalance metrics to optimize task clustering in scientific workflow executions. Fut. Gener. Comput. Syst. 46, C (May 2015), 69--84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. G. Chiola, C. Anglano, J. Campos, J. M. Colom, and M. Silva. 1993. Operational analysis of timed Petri nets and application to the computation of performance bounds. In Proceedings of the 5th International Workshop on Petri Nets and Performance Models (PNPM’93). IEEE Computer Society Press, Los Alamitos, CA, 128--137.Google ScholarGoogle Scholar
  11. E. Deelman, G. Mehta, G. Singh, M. Su, and K. Vahi. 2007. Pegasus: Mapping large-scale workflows to distributed resources. In Workflows for eScience. Springer, 376--394.Google ScholarGoogle Scholar
  12. Rubing Duan, Thomas Fahringer, Radu Prodan, Jun Qin, Alex Villazón, and Marek Wieczorek. 2005. Real world workflow applications in the Askalon grid environment. In Proceedings of the Advances in Grid Computing European Grid Conference (EGC’05). Springer, Berlin, 454--463. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Rubing Duan, Farrukh Nadeem, Jie Wang, Yun Zhang, Radu Prodan, and Thomas Fahringer. 2009. A hybrid intelligent method for performance modeling and prediction of workflow activities in Grids. In Proceedings of the 9th IEEE International Symposium on Cluster Computing and the Grid (CCGrid’09). IEEE Computer Society, Los Alamitos, CA, 339--347. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Rubing Duan, Radu Prodan, and Thomas Fahringer. 2006. Run-time optimisation of grid workflow applications. In Proceedings of the 7th IEEE/ACM International Conference on Grid Computing (GRID’06). IEEE Computer Society, Los Alamitos, CA, 33--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. G. Florin and S. Natkin. 1985. Les réseaux de Petri stochastiques. Techn. Sci. Inf. 4 (1985), 143--160.Google ScholarGoogle Scholar
  16. Zhijie Guan, Francisco Hernandez, Purushotham Bangalore, Jeff Gray, Anthony Skjellum, Vijay Velusamy, and Yin Liu. 2006. Grid-flow: A Grid-enabled scientific workflow system with a Petri-net-based interface: Research articles. Concurr. Comput. Pract. Exper. 18 (Aug. 2006), 1115--1140. Issue 10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Andreas Hoheisel. 2006. User tools and languages for graph-based Grid workflows: Research articles. Concurr. Comput. Pract. Exper. 18, 10 (2006), 1101--1113. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Jeffrey O. Kephart and David M. Chess. 2003. The vision of autonomic computing. Computer 36, 1 (2003), 41--50. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. John D. C. Little. 1961. A proof for the queuing formula: L= λ W. Oper. Res. 9, 3 (1961), 383--387. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Tadao Murata. 1989. Petri nets: Properties, analysis and applications. In Proceedings of the IEEE, Vol. 77. 541--580.Google ScholarGoogle Scholar
  21. F. Nerieri, R. Prodan, T. Fahringer, and Hong-Linh Truong. 2006. Overhead analysis of Grid workflow applications. In Proceedings of the 7th IEEE/ACM International Conference on Grid Computing (GRID’06). IEEE Computer Society, Los Alamitos, CA, 17--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Tom Oinn, Mark Greenwood, Matthew Addis, M. Nedim Alpdemir, Justin Ferris, Kevin Glover, Carole Goble, Antoon Goderis, Duncan Hull, Darren Marvin, Peter Li, Phillip Lord, Matthew R. Pocock, Martin Senger, Robert Stevens, Anil Wipat, and Chris Wroe. 2006. Taverna: Lessons in creating a workflow environment for the life sciences: Research articles. Concurr. Comput. Pract. Exper. 18, 10 (2006), 1067--1100. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Sang-Min Park and M. Humphrey. 2008. Data throttling for data-intensive workflows. In Proceedings of the IEEE International Symposium on Parallel and Distributed Processing. 1--11.Google ScholarGoogle Scholar
  24. Simone Pellegrini, Andreas Hoheisel, Francesco Giacomini, and Antonia Ghiselli. 2008. Using GWorkflowDL for middleware-independent modeling and enactment of workflows. In Proceedings of the CoreGRID Integration Workshop 2008.Google ScholarGoogle Scholar
  25. C. Ramchandani. 1974. Analysis of Asynchronous Concurrent Systems by Petri Nets. Ph.D. Dissertation. Department of Electrical Engineering, Massachusetts Institute of Technology, Cambridge, MA.Google ScholarGoogle Scholar
  26. Ricardo J. Rodríguez and Jorge Júlvez. 2010. Accurate performance estimation for stochastic marked graphs by bottleneck regrowing. In Proceedings of the 7th European Performance Engineering Workshop (EPEW’10), Lecture Notes in Computer Science, Vol. 6342. Springer, 175--190. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Ricardo J. Rodríguez, Rafael Tolosana-Calasanz, and Omer F. Rana. 2012. Automating data-throttling analysis for data-intensive workflows. In Proceedings of the 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid’12). IEEE, 310--317. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Ricardo J. Rodríguez, Rafael Tolosana-Calasanz, and Omer F. Rana. 2012. Measuring the effectiveness of throttled data transfers on data-intensive workflows. In Proceedings of the 6th KES International Conference on Agent and Multi-Agent Systems, Technologies, and Applications (KES-AMSTA’12). 144--153. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Jacek Sroka and Jan Hidders. 2009. Towards a formal semantics for the process model of the Taverna workbench. Part ii. Fundam. Inf. 92, 4 (2009), 373--396. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. I. J. Taylor, E. Deelman, D. B. Gannon, and M. Shields (Eds.). 2007. Workflows for eScience: Scientific Workflows for Grids. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Rafael Tolosana-Calasanz, José A. Bañares, Pedro Álvarez, Joaquín Ezpeleta, and Omer F. Rana. 2010a. An uncoordinated asynchronous checkpointing model for hierarchical scientific workflows. J. Comput. Syst. Sci. 76, 6 (Sep. 2010), 403--415. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Rafael Tolosana-Calasanz, José A. Bañares, Omer F. Rana, Pedro Álvarez, Joaquín Ezpeleta, and Andreas Hoheisel. 2010b. Adaptive exception handling for scientific workflows. Concurr. Comput. Pract. Exper. 22, 5 (Apr. 2010), 617--642. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Rafael Tolosana-Calasanz, Omer Rana, and José Bañares. 2008. Automating performance analysis from Taverna workflows. In Component-Based Software Engineering, Michel Chaudron, Clemens Szyperski, and Ralf Reussner (Eds.). Lecture Notes in Computer Science, Vol. 5282. Springer, Berlin, 1--15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Wil van der Aalst and Kees van Hee. 2004. Workflow Management: Models, Methods, and Systems. The MIT Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Michal Vossberg, Andreas Hoheisel, Thomas Tolxdorff, and Dagmar Krefting. 2008. A reliable DICOM transfer grid service based on Petri net workflows. In Proceedings of the 2008 8th IEEE International Symposium on Cluster Computing and the Grid. IEEE Computer Society, Los Alamitos, CA, 441--448. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Jia Yu and Rajkumar Buyya. 2005. A taxonomy of workflow management systems for Grid computing. CoRR 34, 3 (Sep. 2005), 44--49.Google ScholarGoogle Scholar

Index Terms

  1. A Dynamic Data-throttling Approach to Minimize Workflow Imbalance

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Internet Technology
          ACM Transactions on Internet Technology  Volume 19, Issue 3
          Special Section on Advances in Internet-Based Collaborative Technologies
          August 2019
          289 pages
          ISSN:1533-5399
          EISSN:1557-6051
          DOI:10.1145/3329912
          • Editor:
          • Ling Liu
          Issue’s Table of Contents

          Copyright © 2019 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 7 May 2019
          • Accepted: 1 September 2018
          • Revised: 1 August 2018
          • Received: 1 January 2018
          Published in toit Volume 19, Issue 3

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed
        • Article Metrics

          • Downloads (Last 12 months)2
          • Downloads (Last 6 weeks)1

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!