skip to main content
research-article

Adaptive Speculative Processing of Out-of-Order Event Streams

Authors Info & Claims
Published:07 August 2014Publication History
Skip Abstract Section

Abstract

Distributed event-based systems are used to detect meaningful events with low latency in high data-rate event streams that occur in surveillance, sports, finances, etc. However, both known approaches to dealing with the predominant out-of-order event arrival at the distributed detectors have their shortcomings: buffering approaches introduce latencies for event ordering, and stream revision approaches may result in system overloads due to unbounded retraction cascades.

This article presents an adaptive speculative processing technique for out-of-order event streams that enhances typical buffering approaches. In contrast to other stream revision approaches developed so far, our novel technique encapsulates the event detector, uses the buffering technique to delay events but also speculatively processes a portion of it, and adapts the degree of speculation at runtime to fit the available system resources so that detection latency becomes minimal.

Our technique outperforms known approaches on both synthetical data and real sensor data from a realtime locating system (RTLS) with several thousands of out-of-order sensor events per second. Speculative buffering exploits system resources and reduces latency by 40% on average.

References

  1. Jagrati Agrawal, Yanlei Diao, Daniel Gyllstrom, and Neil Immerman. 2008. Efficient pattern matching over event streams. In Proceedings of the International Conference on Management of Data. 147--160. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Marcos Kawazoe Aguilera and Robert E. Strom. 2000. Efficient atomic broadcast using deterministic merge. In Proceedings of the 19th Symposium on Principles Distributed Computing. 209--218. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Mark Allman, Chris Hayes, and Shawn Ostermann. 1998. An evaluation of TCP with larger initial windows. ACM SIGCOMM Comput. Commun. Rev. 28, 3 (1998), 41--52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Rajagopal Ananthanarayanan, Venkatesh Basker, Sumit Das, Ashish Gupta, Haifeng Jiang, Tianhao Qiu, Alexey Reznichenko, Deomid Ryabkov, Manpreet Singh, and Shivakumar Venkataraman. 2013. Photon: Fault-tolerant and scalable joining of continuous data streams. In Proceedings of the International Conference on Management of Data. 577--588. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Darko Anicic, Sebastian Rudolph, Paul Fodor, and Nenad Stojanovic. 2011. Retractable complex event processing and stream reasoning. In Proceedings of the 5th International Conference on Rule-Based Reasoning, Programming, and Applications. 122--137. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Arvind Arasu, Mitch Cherniack, Eduardo Galvez, David Maier, Anurag S. Maskey, Esther Ryvkina, Michael Stonebraker, and Richard Tibbetts. 2004. Linear road: A stream data management benchmark. In Proceedings of the 13th International Conference on Very Large Data Bases. Toronto, Canada, 480--491. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Magdalena Balazinska, YongChul Kwon, Nathan Kuchta, and Dennis Lee. 2007. Moirae: History-enhanced monitoring. In Proceedings of the 3rd Biennial Conference on Innovative Data Systems Research. Pacific Grove, CA, 375--386.Google ScholarGoogle Scholar
  8. Roger S. Barga, Jonathan Goldstein, Mohamed Ali, and Mingsheng Hong. 2007. Consistent streaming through time: A vision for event stream processing. In Proceedings of the 3rd Biennial Conference on Innovative Data Systems Research. 363--374.Google ScholarGoogle Scholar
  9. Lars Brenna, Alan Demers, Johannes Gehrke, Mingsheng Hong, Joel Ossher, Biswanath Panda, Mirek Riedewald, Mohit Thatte, and Walker White. 2007. Cayuga: A high-performance event processing engine. In Proceedings of the International Conference on Management of Data. 1100--1102. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Lars Brenna, Johannes Gehrke, Mingsheng Hong, and Dag Johansen. 2009. Distributed event stream processing with non-deterministic finite automata. In Proceedings of the 3rd International Conference on Distributed Event-Based Systems. Article 3, 12 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Andrey Brito, Christof Fetzer, Heiko Sturzrehm, and Pascal Felber. 2008. Speculative out-of-order event processing with software transaction memory. In Proceedings of the 2nd International Conference on Distributed Event-Based Systems. 265--275. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Badrish Chandramouli, Jonathan Goldstein, and David Maier. 2010. High-performance dynamic pattern matching over disordered streams. In Proceedings of the International Conference on Very Large Databases. 220--231.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Alan Demers, Johannes Gehrke, Mingsheng Hong, Mirek Riedewald, and Walker White. 2006. Towards expressive publish/subscribe systems. In Proceedings of the 10th International Conference on Extending Database Technology. 627--644. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Alan Demers, Johannes Gehrke, Biswanath Panda, Mirek Riedewald, Varun Sharma, and Walker White. 2007. Cayuga: A general purpose event monitoring system. In Proceedings of the 3rd Biennial Conference on Innovative Data Systems Research. 412--422.Google ScholarGoogle Scholar
  15. Mootaz Elnozahy, Lorenzo Alvisi, Yi-Min Wang, and David B. Johnson. 2002. A survey of rollback-recovery protocols in message-passing systems. ACM Comput. Surv. 34, 3 (2002), 375--408. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Raul Castro Fernandez, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch. 2013. Integrating scale out and fault tolerance in stream processing using operator state management. In Proceedings of the International Conference on Management of Data. 725--736. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Jeong-Hyon Hwang, Ugur Çetintemel, and Stanley B. Zdonik. 2008. Fast and highly-available stream processing over wide area networks. In Proceedings of the 24th International Conference on Data Engineering. 804--813. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Zbigniew Jerzak and Christof Fetzer. 2008. BFSiena: A communication substrate for StreamMine. In Proceedings of the 2nd International Conference on Distributed Event-Based Systems. 321--324. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Zbigniew Jerzak and Holger Ziekow. 2014. The DEBS 2014 grand challenge. In Proceedings of the 8th International Conference on Distributed Event-Based Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Boris Koldehofe, Ruben Mayer, Umakishore Ramachandran, Kurt Rothermel, and Marco Völz. 2013. Rollback-recovery without checkpoints in distributed event processing systems. In Proceedings of the 7th International Conference on Distributed Event-Based Systems. 27--38. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. YongChul Kwon, Magdalena Balazinska, and Albert Greenberg. 2008. Fault-tolerant stream processing using a distributed, replicated file system. Proc. VLDB Endow. 1, 1 (2008), 574--585. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Chuan-Wen Li, Yu Gu, Ge Yu, and Bonghee Hong. 2011. Aggressive complex event processing with confidence over out-of-order streams. J. Comput. Sci. Technol. 26, 4 (2011), 685--696.Google ScholarGoogle ScholarCross RefCross Ref
  23. Jin Li, Kristin Tufte, Vladislav Shkapenyuk, Vassilis Papadimos, Theodore Johnson, and David Maier. 2008. Out-of-order processing: A new architecture for high-performance stream systems. Proce. VLDB Endow., 1 (2008), 274--288. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Ming Li, Mo Liu, Luping Ding, Elke Rundensteiner, and Murali Mani. 2007. Event stream processing with out-of-order data arrival. In Proceedings of the 27th International Conference on Distributed Computing Systems Workshops. 67--74. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Mo Liu, Ming Li, D. Golovnya, E. A. Rundensteiner, and K. Claypool. 2009. Sequence pattern query processing over out-of-order event streams. In Proceedings of the 25th International Conference on Data Engineering 784--795. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Anurag S. Maskey and Mitch Cherniack. 2008. Replay-based approaches to revision processing in stream query engines. In Proceedings of the 2nd International Workshop on Scalable Stream Processing Systems. 3--12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Gero Mühl, Ludger Fiege, and Peter Pietzuch. 2006. Distributed Event-Based Systems. Springer, Berlin.Google ScholarGoogle Scholar
  28. Christopher Mutschler. 2014. Latency Minimization of Order-Preserving Distributed Event-Based Systems. Ph.D. Dissertation. Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany.Google ScholarGoogle Scholar
  29. Christopher Mutschler, Zbigniew Jerzak, and Holger Ziekow. 2013. The DEBS 2013 grand challenge. In Proceedings of the 7th International Conference on Distributed Event-Based Systems. 289--294. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Christopher Mutschler and Michael Philippsen. 2013a. Distributed low-latency out-of-order event processing for high data rate sensor streams. In Proceedings of the 27th International Parallel and Distributed Processing Symposium. 1133--1144. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Christopher Mutschler and Michael Philippsen. 2013b. Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares. In Proceedings of the 7th International Conference on Distributed Event-Based Systems. 147--158. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Dan O'Keeffe and Jean Bacon. 2010. Reliable complex event detection for pervasive computing. In Proceedings of the 4th International Conference on Distributed Event-Based Systems. 73--84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Larry L. Peterson, Nick C. Buchholz, and Richard D. Schlichting. 1989. Preserving and using context information in interprocess communication. ACM Trans. Comput. Syst. 7, 3 (1989), 217--246. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Esther Ryvkina, Anurag Maskey, Mitch Cherniack, and Stan Zdonik. 2006. Revision processing in a stream processing engine: A high-level design. In Proceedings of the 22nd International Conference on Data Engineering. 141--143. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Zoe Sebepou and Kostas Magoutis. 2011. CEC: Continuous eventual checkpointing for data stream processing operators. In Proceedings of the 41st International Conference on Dependable Systems Networks. 145--156. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Utkarsh Srivastava and Jennifer Widom. 2004. Flexible time management in data stream systems. In Proceedings of the 23rd Symposium on Principles Database Systems. 263--274. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Michael Stonebraker, Uǧur Çetintemel, and Stan Zdonik. 2005. The 8 requirements of real-time stream processing. SIGMOD Record 34, 4 (2005), 42--47. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Peter A. Tucker, David Maier, Tim Sheard, and Leonidas Fegaras. 2003. Exploiting punctuation semantics in continuous data streams. Trans. Know. Data Eng. 15, 3 (2003), 555--568. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Thomas von der Grün, Norbert Franke, Daniel Wolf, Nicolas Witt, and Andreas Eidloth. 2011. A real-time tracking system for football match and training analysis. In Microelectronic Systems. Springer Berlin, 199--212.Google ScholarGoogle Scholar
  40. Benjamin Wester, James Cowling, Edmund B. Nightingale, Peter M. Chen, Jason Flinn, and Barbara Liskov. 2009. Tolerating latency in replicated state machines through client speculation. In Proceedings of the 6th Symposium on Networked Systems Design and Implementation. 245--260. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Eugene Wu, Yanlei Diao, and Shariq Rizvi. 2006. High-performance complex event processing over streams. In Proceedings of the International Conference on Management of Data. 407--418. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Kaiwen Zhang, Vinod Muthusamy, and Hans-Arno Jacobsen. 2012. Total order in content-based publish/subscribe systems. In Proceedings of the 32nd International Conference on Distributed Computing Systems. 335--344. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Adaptive Speculative Processing of Out-of-Order Event Streams

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!