Abstract
Distributed event-based systems are used to detect meaningful events with low latency in high data-rate event streams that occur in surveillance, sports, finances, etc. However, both known approaches to dealing with the predominant out-of-order event arrival at the distributed detectors have their shortcomings: buffering approaches introduce latencies for event ordering, and stream revision approaches may result in system overloads due to unbounded retraction cascades.
This article presents an adaptive speculative processing technique for out-of-order event streams that enhances typical buffering approaches. In contrast to other stream revision approaches developed so far, our novel technique encapsulates the event detector, uses the buffering technique to delay events but also speculatively processes a portion of it, and adapts the degree of speculation at runtime to fit the available system resources so that detection latency becomes minimal.
Our technique outperforms known approaches on both synthetical data and real sensor data from a realtime locating system (RTLS) with several thousands of out-of-order sensor events per second. Speculative buffering exploits system resources and reduces latency by 40% on average.
- Jagrati Agrawal, Yanlei Diao, Daniel Gyllstrom, and Neil Immerman. 2008. Efficient pattern matching over event streams. In Proceedings of the International Conference on Management of Data. 147--160. Google Scholar
Digital Library
- Marcos Kawazoe Aguilera and Robert E. Strom. 2000. Efficient atomic broadcast using deterministic merge. In Proceedings of the 19th Symposium on Principles Distributed Computing. 209--218. Google Scholar
Digital Library
- Mark Allman, Chris Hayes, and Shawn Ostermann. 1998. An evaluation of TCP with larger initial windows. ACM SIGCOMM Comput. Commun. Rev. 28, 3 (1998), 41--52. Google Scholar
Digital Library
- Rajagopal Ananthanarayanan, Venkatesh Basker, Sumit Das, Ashish Gupta, Haifeng Jiang, Tianhao Qiu, Alexey Reznichenko, Deomid Ryabkov, Manpreet Singh, and Shivakumar Venkataraman. 2013. Photon: Fault-tolerant and scalable joining of continuous data streams. In Proceedings of the International Conference on Management of Data. 577--588. Google Scholar
Digital Library
- Darko Anicic, Sebastian Rudolph, Paul Fodor, and Nenad Stojanovic. 2011. Retractable complex event processing and stream reasoning. In Proceedings of the 5th International Conference on Rule-Based Reasoning, Programming, and Applications. 122--137. Google Scholar
Digital Library
- Arvind Arasu, Mitch Cherniack, Eduardo Galvez, David Maier, Anurag S. Maskey, Esther Ryvkina, Michael Stonebraker, and Richard Tibbetts. 2004. Linear road: A stream data management benchmark. In Proceedings of the 13th International Conference on Very Large Data Bases. Toronto, Canada, 480--491. Google Scholar
Digital Library
- Magdalena Balazinska, YongChul Kwon, Nathan Kuchta, and Dennis Lee. 2007. Moirae: History-enhanced monitoring. In Proceedings of the 3rd Biennial Conference on Innovative Data Systems Research. Pacific Grove, CA, 375--386.Google Scholar
- Roger S. Barga, Jonathan Goldstein, Mohamed Ali, and Mingsheng Hong. 2007. Consistent streaming through time: A vision for event stream processing. In Proceedings of the 3rd Biennial Conference on Innovative Data Systems Research. 363--374.Google Scholar
- Lars Brenna, Alan Demers, Johannes Gehrke, Mingsheng Hong, Joel Ossher, Biswanath Panda, Mirek Riedewald, Mohit Thatte, and Walker White. 2007. Cayuga: A high-performance event processing engine. In Proceedings of the International Conference on Management of Data. 1100--1102. Google Scholar
Digital Library
- Lars Brenna, Johannes Gehrke, Mingsheng Hong, and Dag Johansen. 2009. Distributed event stream processing with non-deterministic finite automata. In Proceedings of the 3rd International Conference on Distributed Event-Based Systems. Article 3, 12 pages. Google Scholar
Digital Library
- Andrey Brito, Christof Fetzer, Heiko Sturzrehm, and Pascal Felber. 2008. Speculative out-of-order event processing with software transaction memory. In Proceedings of the 2nd International Conference on Distributed Event-Based Systems. 265--275. Google Scholar
Digital Library
- Badrish Chandramouli, Jonathan Goldstein, and David Maier. 2010. High-performance dynamic pattern matching over disordered streams. In Proceedings of the International Conference on Very Large Databases. 220--231.Google Scholar
Digital Library
- Alan Demers, Johannes Gehrke, Mingsheng Hong, Mirek Riedewald, and Walker White. 2006. Towards expressive publish/subscribe systems. In Proceedings of the 10th International Conference on Extending Database Technology. 627--644. Google Scholar
Digital Library
- Alan Demers, Johannes Gehrke, Biswanath Panda, Mirek Riedewald, Varun Sharma, and Walker White. 2007. Cayuga: A general purpose event monitoring system. In Proceedings of the 3rd Biennial Conference on Innovative Data Systems Research. 412--422.Google Scholar
- Mootaz Elnozahy, Lorenzo Alvisi, Yi-Min Wang, and David B. Johnson. 2002. A survey of rollback-recovery protocols in message-passing systems. ACM Comput. Surv. 34, 3 (2002), 375--408. Google Scholar
Digital Library
- Raul Castro Fernandez, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch. 2013. Integrating scale out and fault tolerance in stream processing using operator state management. In Proceedings of the International Conference on Management of Data. 725--736. Google Scholar
Digital Library
- Jeong-Hyon Hwang, Ugur Çetintemel, and Stanley B. Zdonik. 2008. Fast and highly-available stream processing over wide area networks. In Proceedings of the 24th International Conference on Data Engineering. 804--813. Google Scholar
Digital Library
- Zbigniew Jerzak and Christof Fetzer. 2008. BFSiena: A communication substrate for StreamMine. In Proceedings of the 2nd International Conference on Distributed Event-Based Systems. 321--324. Google Scholar
Digital Library
- Zbigniew Jerzak and Holger Ziekow. 2014. The DEBS 2014 grand challenge. In Proceedings of the 8th International Conference on Distributed Event-Based Systems. Google Scholar
Digital Library
- Boris Koldehofe, Ruben Mayer, Umakishore Ramachandran, Kurt Rothermel, and Marco Völz. 2013. Rollback-recovery without checkpoints in distributed event processing systems. In Proceedings of the 7th International Conference on Distributed Event-Based Systems. 27--38. Google Scholar
Digital Library
- YongChul Kwon, Magdalena Balazinska, and Albert Greenberg. 2008. Fault-tolerant stream processing using a distributed, replicated file system. Proc. VLDB Endow. 1, 1 (2008), 574--585. Google Scholar
Digital Library
- Chuan-Wen Li, Yu Gu, Ge Yu, and Bonghee Hong. 2011. Aggressive complex event processing with confidence over out-of-order streams. J. Comput. Sci. Technol. 26, 4 (2011), 685--696.Google Scholar
Cross Ref
- Jin Li, Kristin Tufte, Vladislav Shkapenyuk, Vassilis Papadimos, Theodore Johnson, and David Maier. 2008. Out-of-order processing: A new architecture for high-performance stream systems. Proce. VLDB Endow., 1 (2008), 274--288. Google Scholar
Digital Library
- Ming Li, Mo Liu, Luping Ding, Elke Rundensteiner, and Murali Mani. 2007. Event stream processing with out-of-order data arrival. In Proceedings of the 27th International Conference on Distributed Computing Systems Workshops. 67--74. Google Scholar
Digital Library
- Mo Liu, Ming Li, D. Golovnya, E. A. Rundensteiner, and K. Claypool. 2009. Sequence pattern query processing over out-of-order event streams. In Proceedings of the 25th International Conference on Data Engineering 784--795. Google Scholar
Digital Library
- Anurag S. Maskey and Mitch Cherniack. 2008. Replay-based approaches to revision processing in stream query engines. In Proceedings of the 2nd International Workshop on Scalable Stream Processing Systems. 3--12. Google Scholar
Digital Library
- Gero Mühl, Ludger Fiege, and Peter Pietzuch. 2006. Distributed Event-Based Systems. Springer, Berlin.Google Scholar
- Christopher Mutschler. 2014. Latency Minimization of Order-Preserving Distributed Event-Based Systems. Ph.D. Dissertation. Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany.Google Scholar
- Christopher Mutschler, Zbigniew Jerzak, and Holger Ziekow. 2013. The DEBS 2013 grand challenge. In Proceedings of the 7th International Conference on Distributed Event-Based Systems. 289--294. Google Scholar
Digital Library
- Christopher Mutschler and Michael Philippsen. 2013a. Distributed low-latency out-of-order event processing for high data rate sensor streams. In Proceedings of the 27th International Parallel and Distributed Processing Symposium. 1133--1144. Google Scholar
Digital Library
- Christopher Mutschler and Michael Philippsen. 2013b. Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares. In Proceedings of the 7th International Conference on Distributed Event-Based Systems. 147--158. Google Scholar
Digital Library
- Dan O'Keeffe and Jean Bacon. 2010. Reliable complex event detection for pervasive computing. In Proceedings of the 4th International Conference on Distributed Event-Based Systems. 73--84. Google Scholar
Digital Library
- Larry L. Peterson, Nick C. Buchholz, and Richard D. Schlichting. 1989. Preserving and using context information in interprocess communication. ACM Trans. Comput. Syst. 7, 3 (1989), 217--246. Google Scholar
Digital Library
- Esther Ryvkina, Anurag Maskey, Mitch Cherniack, and Stan Zdonik. 2006. Revision processing in a stream processing engine: A high-level design. In Proceedings of the 22nd International Conference on Data Engineering. 141--143. Google Scholar
Digital Library
- Zoe Sebepou and Kostas Magoutis. 2011. CEC: Continuous eventual checkpointing for data stream processing operators. In Proceedings of the 41st International Conference on Dependable Systems Networks. 145--156. Google Scholar
Digital Library
- Utkarsh Srivastava and Jennifer Widom. 2004. Flexible time management in data stream systems. In Proceedings of the 23rd Symposium on Principles Database Systems. 263--274. Google Scholar
Digital Library
- Michael Stonebraker, Uǧur Çetintemel, and Stan Zdonik. 2005. The 8 requirements of real-time stream processing. SIGMOD Record 34, 4 (2005), 42--47. Google Scholar
Digital Library
- Peter A. Tucker, David Maier, Tim Sheard, and Leonidas Fegaras. 2003. Exploiting punctuation semantics in continuous data streams. Trans. Know. Data Eng. 15, 3 (2003), 555--568. Google Scholar
Digital Library
- Thomas von der Grün, Norbert Franke, Daniel Wolf, Nicolas Witt, and Andreas Eidloth. 2011. A real-time tracking system for football match and training analysis. In Microelectronic Systems. Springer Berlin, 199--212.Google Scholar
- Benjamin Wester, James Cowling, Edmund B. Nightingale, Peter M. Chen, Jason Flinn, and Barbara Liskov. 2009. Tolerating latency in replicated state machines through client speculation. In Proceedings of the 6th Symposium on Networked Systems Design and Implementation. 245--260. Google Scholar
Digital Library
- Eugene Wu, Yanlei Diao, and Shariq Rizvi. 2006. High-performance complex event processing over streams. In Proceedings of the International Conference on Management of Data. 407--418. Google Scholar
Digital Library
- Kaiwen Zhang, Vinod Muthusamy, and Hans-Arno Jacobsen. 2012. Total order in content-based publish/subscribe systems. In Proceedings of the 32nd International Conference on Distributed Computing Systems. 335--344. Google Scholar
Digital Library
Index Terms
Adaptive Speculative Processing of Out-of-Order Event Streams
Recommendations
Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares
DEBS '13: Proceedings of the 7th ACM international conference on Distributed event-based systemsIn surveillance, sports, finances, etc., distributed event-based systems are used to detect meaningful events with low latency in high data rate event streams. Both known approaches to deal with the predominant out-of-order event arrival at the ...
Distributed Low-Latency Out-of-Order Event Processing for High Data Rate Sensor Streams
IPDPS '13: Proceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed ProcessingEvent-based Systems (EBS) are used to detect and analyze meaningful events in surveillance, sports, finances and many other areas. With rising data and event rates and with correlations among these events, sequential event processing becomes infeasible ...
Demo: do event-based systems have a passion for sports?
DEBS '13: Proceedings of the 7th ACM international conference on Distributed event-based systemsThe ubiquity of sensor data calls for automatic processing to extract valuable information. Realtime Locating Systems (RTLS) provide many parallel position data streams for interacting objects, and event-based systems are the method of choice to analyze ...






Comments