Abstract
We present SAPIENS, a software architecture designed to support engineering of interactive systems featuring peripheral interaction in the context of smart environments. SAPIENS introduces dedicated components for user and device tracking, attention detection, priority management for devices, tasks, and notifications, context-awareness inference, user interruptibility prediction, and device interchangeability that can be instantiated at will according to the needs of the application. To implement these components effectively, SAPIENS employs event-based processing by reusing the core engine of a recently introduced software architecture, Euphoria (Schipor et al., 2019), that was specifically designed for engineering interactions in smart environments with heterogeneous I/O devices, and relies entirely on web standards, protocols, and open data-interchange formats, such as JavaScript, WebSockets, HTTP, and JSON. This inheritance makes SAPIENS flexible and adaptable to support implementation of diverse application scenarios for peripheral interaction and for a wide variety of smart environments, devices, platforms, data formats, and contexts of use. We present our design criteria for SAPIENS regarding (1) event handling techniques, (2) quality, (3) contextual, and (4) attention-related properties, and describe its components and dataflows that make SAPIENS a specialized software architecture for peripheral interaction scenarios. We also demonstrate SAPIENS with a practical application, inspired and adapted from Bakker's (2013) classical example for peripheral interaction, for which we provide an online simulation tool that researchers and practitioners can readily use to consult actual JavaScript code implementing the inner logic of selected components of our architecture as well as to observe live JSON messages exchanged by the various components of SAPIENS.
- Gregory D. Abowd, Anind K. Dey, Peter J. Brown, Nigel Davies, Mark Smith, and Pete Steggles. 1999. Towards a better understanding of context and context-awareness. In Proceedings of the International Symposium on Handheld and Ubiquitous Computing. Springer, 304--307. Google Scholar
Digital Library
- Hamed S. Alavi and Pierre Dillenbourg. 2012. An Ambient Awareness Tool for Supporting Supervised Collaborative Problem Solving. IEEE Trans. Learn. Technol., Vol. 5, 3 (Jan. 2012), 264--274. Google Scholar
Digital Library
- Erik van Alphen and Saskia Bakker. 2016. Lernanto: Using an Ambient Display During Differentiated Instruction. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, New York, NY, USA, 2334--2340. Google Scholar
Digital Library
- Jorge Alves Lino, Benjamin Salem, and Matthias Rauterberg. 2010. Responsive environments: User experiences for ambient intelligence. Journal of Ambient Intelligence and Smart Environments, Vol. 2, 4 (2010), 347--367. Google Scholar
Digital Library
- Juan Carlos Augusto. 2007. Ambient Intelligence: The Confluence of Ubiquitous/Pervasive Computing and Artificial Intelligence .Springer, London.Google Scholar
- Juan Carlos Augusto, H. Nakashima, and H. Aghajan. 2010. Ambient Intelligence and Smart Environments: A State of the Art .Springer Science & Business Media.Google Scholar
- Saskia Bakker. 2013. Design for peripheral interaction. Eindhoven University of Technology (2013). http://www.saskiabakker.com/PhDthesis_SaskiaBakker.pdfGoogle Scholar
- Saskia Bakker, Doris Hausen, and Ted Selker. 2016. Introduction: Framing peripheral interaction. In Peripheral Interaction. Springer, 1--10.Google Scholar
Cross Ref
- Saskia Bakker, Elise Hoven, and Berry Eggen. 2015. Peripheral interaction: Characteristics and considerations. Personal and Ubiquitous Computing, Vol. 19, 1 (2015), 239--254. Google Scholar
Digital Library
- Saskia Bakker and Karin Niemantsverdriet. 2016. The interaction-attention continuum: considering various levels of human attention in interaction design. International Journal of Design, Vol. 10, 2 (2016), 1--14. http://ijdesign.org/index.php/IJDesign/article/view/2341/737Google Scholar
- Saskia Bakker, Elise van den Hoven, and Berry Eggen. 2013. FireFlies: Physical Peripheral Interaction Design for the Everyday Routine of Primary School Teachers. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction (TEI '13). ACM, New York, NY, USA, 57--64. Google Scholar
Digital Library
- Saskia Bakker, Elise van den Hoven, Berry Eggen, and Kees Overbeeke. 2012. Exploring Peripheral Interaction Design for Primary School Teachers. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI '12). ACM, New York, NY, USA, 245--252.Google Scholar
Digital Library
- Norbert Baumgartner, Wolfgang Gottesheim, Stefan Mitsch, Werner Retschitzegger, and Wieland Schwinger. 2010. Editorial: BeAware!-Situation Awareness, the Ontology-driven Way. Data Knowl. Eng., Vol. 69, 11 (Nov. 2010), 1181--1193.Google Scholar
Digital Library
- John N.A. Brown. 2014. Unifying interaction across distributed controls in a smart environment using anthropology-based computing to make human-computer interaction "Calm". (2014).Google Scholar
- Yaliang Chuang, Lin-Lin Chen, and Yoga Liu. 2018. Design Vocabulary for Human--IoT Systems Communication. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 274. Google Scholar
Digital Library
- Diane Cook and Sajal Das. 2004. Smart Environments: Technology, Protocols and Applications (Wiley Series on Parallel and Distributed Computing) .Wiley-Interscience, New York, NY, USA. Google Scholar
Digital Library
- Diane J Cook, Juan C Augusto, and Vikramaditya R Jakkula. 2009. Ambient intelligence: Technologies, applications, and opportunities. Pervasive and Mobile Computing, Vol. 5, 4 (2009), 277--298. Google Scholar
Digital Library
- Anind K. Dey. 2001. Understanding and Using Context. Personal Ubiquitous Comput., Vol. 5, 1 (Jan. 2001), 4--7. Google Scholar
Digital Library
- Krista M. Dombroviak and Rajiv Ramnath. 2007. A Taxonomy of Mobile and Pervasive Applications. In Proceedings of the 2007 ACM Symposium on Applied Computing (SAC '07). ACM, New York, NY, USA, 1609--1615. Google Scholar
Digital Library
- Antonio Fernández-Caballero, Arturo Mart'inez-Rodrigo, José Manuel Pastor, José Carlos Castillo, Elena Lozano-Monasor, Mar'ia T López, Roberto Zangróniz, José Miguel Latorre, and Alicia Fernández-Sotos. 2016. Smart environment architecture for emotion detection and regulation. Journal of Biomedical Informatics, Vol. 64 (2016), 55--73. Google Scholar
Digital Library
- George H. Forman and John Zahorjan. 1994. The Challenges of Mobile Computing. Computer, Vol. 27, 4 (April 1994), 38--47. Google Scholar
Digital Library
- Giancarlo Fortino, Daniele Parisi, Vincenzo Pirrone, and Giuseppe Di Fatta. 2014. BodyCloud: A SaaS Approach for Community Body Sensor Networks. Future Gener. Comput. Syst., Vol. 35 (June 2014), 62--79. Google Scholar
Digital Library
- Bogdan-Florin Gheran, Jean Vanderdonckt, and Radu-Daniel Vatavu. 2018. Gestures for Smart Rings: Empirical Results, Insights, and Design Implications. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS '18). ACM, New York, NY, USA, 623--635.Google Scholar
Digital Library
- Giuseppe Ghiani, Marco Manca, and Fabio Paternò. 2015. Authoring context-dependent cross-device user interfaces based on trigger/action rules. In Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia. ACM, 313--322.Google Scholar
Digital Library
- Milind S. Gide and Lina J. Karam. 2017. Computational Visual Attention Models. Foundations and Trends® in Signal Processing, Vol. 10, 4 (2017), 347--427.Google Scholar
- Sylvain Giroux, Tatjana Leblanc, Abdenour Bouzouane, Bruno Bouchard, Hélène Pigot, and Jérémy Bauchet. 2009. The Praxis of Cognitive Assistance in Smart Homes. In Ambient Intelligence and Smart Environments, volume 3: Behaviour Monitoring and Interpretation. 183--211.Google Scholar
- Christos Goumopoulos, Achilles Kameas, and Patras Hellas. 2009. Smart objects as components of ubicomp applications. International Journal of Multimedia and Ubiquitous Engineering, Vol. 4, 3 (2009). http://www.sersc.org/journals/IJMUE/vol4_no3_2009/1.pdfGoogle Scholar
- Saul Greenberg, Nicolai Marquardt, Till Ballendat, Rob Diaz-Marino, and Miaosen Wang. 2011. Proxemic Interactions: The New Ubicomp? Interactions, Vol. 18, 1 (Jan. 2011), 42--50. Google Scholar
Digital Library
- Robert P. Hawkins, Suzanne Pingree, Jacqueline Hitchon, Barry Radler, Bradley W. Gorham, Leeann Kahlor, Eilleen Gilligan, Ronald C. Serlin, Toni Schmidt, Prathana Kannaovakun, and Gudbjorg Hildur Kolbeins. 2006. What Produces Television Attention and Attention Style? Human Communication Research, Vol. 31, 1 (2006).Google Scholar
- Marigo Heijboer, Elise Hoven, Bert Bongers, and Saskia Bakker. 2016. Facilitating Peripheral Interaction: Design and Evaluation of Peripheral Interaction for a Gesture-based Lighting Control with Multimodal Feedback. Personal Ubiquitous Comput., Vol. 20, 1 (Feb. 2016), 1--22.Google Scholar
Digital Library
- Sumi Helal, William Mann, Hicham El-Zabadani, Jeffrey King, Youssef Kaddoura, and Erwin Jansen. 2005. The Gator Tech Smart House: A programmable pervasive space. Computer 3 (2005), 50--60.Google Scholar
Digital Library
- Laurent Itti. 2000. Models of Bottom-up and Top-down Visual Attention. Ph.D. Dissertation. Pasadena, CA, USA. Advisor(s) Koch, Christof. https://thesis.library.caltech.edu/4722/ AAI9972609.Google Scholar
Digital Library
- Jae Yeol Lee, Min Seok Kim, Dong Woo Seo, Sang Min Lee, and Jae Sung Kim. 2012. Smart and Space-aware Interactions Using Smartphones in a Shared Space. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services Companion (MobileHCI '12). ACM, New York, NY, USA, 53--58. Google Scholar
Digital Library
- Vicon Motion Systems Limited. 2019. Vicon Motion Capture. https://www.vicon.com/Google Scholar
- Jaime Lloret, Alejandro Canovas, Sandra Sendra, and Lorena Parra. 2015. A smart communication architecture for ambient assisted living. IEEE Communications Magazine, Vol. 53, 1 (2015), 26--33.Google Scholar
Digital Library
- Logbar. 2015. Ring ZERO | Shortcut anything. https://web.archive.org/web/20170511131824/http://ringzero.logbar.jp/ Retrieved April 18, 2019 fromGoogle Scholar
- J. Maisonnasse, N. Gourier, O. Brdiczka, P. Reignier, and J. L. Crowley. 2006. Detecting privacy in attention aware system. In Proceedings of the 2nd IET International Conference on Intelligent Environments (IE '06), Vol. 2. 231--239.Google Scholar
- Sathiya Kumaran Mani, Ramakrishnan Durairajan, Paul Barford, and Joel Sommers. 2018. An Architecture for IoT Clock Synchronization. In Proceedings of the 8th International Conference on the Internet of Things (IOT '18). ACM, New York, NY, USA, Article 17, bibinfonumpages8 pages. Google Scholar
Digital Library
- Steve Mann. 1997. Wearable Computing: A First Step Toward Personal Imaging. Computer, Vol. 30, 2 (Feb. 1997), 25--32. Google Scholar
Digital Library
- Alexander Mariakakis, Mayank Goel, Md Tanvir Islam Aumi, Shwetak N. Patel, and Jacob O. Wobbrock. 2015. SwitchBack: Using Focus and Saccade Tracking to Guide Users' Attention for Mobile Task Resumption. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2953--2962. Google Scholar
Digital Library
- Nicolai Marquardt, Robert Diaz-Marino, Sebastian Boring, and Saul Greenberg. 2011. The Proximity Toolkit: Prototyping Proxemic Interactions in Ubiquitous Computing Ecologies. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 315--326. Google Scholar
Digital Library
- Andrii Matviienko, Sebastian Horwege, Lennart Frick, Christoph Ressel, and Susanne Boll. 2016. CubeLendar: Design of a Tangible Interactive Event Awareness Cube. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, New York, NY, USA, 2601--2608. Google Scholar
Digital Library
- Microsoft. 2017. Kinect for Windows 1.5, 1.6, 1.7, 1.8. Skeleton Class. https://msdn.microsoft.com/en-us/library/microsoft.kinect.skeleton.aspxGoogle Scholar
- Irina Mocanu, Ovidiu-Andrei Schpor, Bogdan Cramariuc, and Lucia Rusu. 2017. [email protected]: A Smart Home Platform for Enhancing the Elderly Mobility. Advances in Electrical and Computer Engineering, Vol. 17, 4 (2017), 19--27.Google Scholar
Cross Ref
- Meredith Ringel Morris. 2012. Web on the Wall: Insights from a Multimodal Interaction Elicitation Study. In Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces (ITS '12). ACM, New York, NY, USA, 95--104. Google Scholar
Digital Library
- MyGaze. 2019. myGaze eye tracker. http://www.mygaze.com/products/mygaze-eye-tracker/Google Scholar
- Alfredo Navarra, Cristina M. Pinotti, Mario Francesco, and Sajal K. Das. 2015. Interference-free Scheduling with Minimum Latency in Cluster-based Wireless Sensor Networks. Wirel. Netw., Vol. 21, 7 (Oct. 2015), 2395--2411. Google Scholar
Digital Library
- Michael Nebeling, Elena Teunissen, Maria Husmann, and Moira C. Norrie. 2014. XDKinect: Development Framework for Cross-device Interaction Using Kinect. In Proceedings of the 2014 ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '14). ACM, New York, NY, USA, 65--74. Google Scholar
Digital Library
- North. 2019. Getting started with your Myo armband. https://support.getmyo.com/hc/en-us/articles/203398347-Getting-started-with-your-Myo-armband Retrieved April 18, 2019 fromGoogle Scholar
- Bogdan Pogorelc, Artur Lugmayr, Bjorn Stockleben, Radu-Daniel Vatavu, Nina Tahmasebi, Estefania Serral, Emilija Stojmenova, Bojan Imperl, Thomas Risse, Gideon Zenz, and Matjaz Gams. 2013. Ambient Bloom: New Business, Content, Design and Models to Increase the Semantic Ambient Media Experience. Multimedia Tools and Applications, Vol. 66, 1 (2013), 7--32. Google Scholar
Digital Library
- Bogdan Pogorelc, Radu-Daniel Vatavu, Artur Lugmayr, Björn Stockleben, Thomas Risse, Juha Kaario, Estefania Constanza Lomonaco, and Matjavz Gams. 2012. Semantic ambient media: From ambient advertising to ambient-assisted living. Multimedia Tools and Applications, Vol. 58, 2 (May 2012), 399--425. Google Scholar
Digital Library
- Irina Popovici, Ovidiu-Andrei Schipor, and Radu-Daniel Vatavu. 2019. Hover: Exploring cognitive maps and mid-air pointing for television control. International Journal of Human-Computer Studies, Vol. 129 (2019), 95--107.Google Scholar
Digital Library
- Robbie Schaefer and Wolfgang Mueller. 2003. Multimodal interactive user interfaces for mobile multi-device environments. In Workshop Multi-Device Interfaces for Ubiquitous Peripheral Interaction.Google Scholar
- Ovidiu-Andrei Schipor, Stefan-Gheorghe Pentiuc, and Maria-Doina Schipor. 2011. Towards a multimodal emotion recognition framework to be integrated in a Computer Based Speech Therapy System. In 2011 6th Conference on Speech Technology and Human-Computer Dialogue (SpeD). IEEE, 1--6.Google Scholar
Cross Ref
- Ovidiu-Andrei Schipor, Stefan-Gheorghe Pentiuc, and Maria-Doina Schipor. 2012. Toward Automatic Recognition of Children's Affective State Using Physiological Parameters and Fuzzy Model of Emotions. Advances in Electrical and Computer Engineering, Vol. 12, 2 (2012), 47--50.Google Scholar
Cross Ref
- Ovidiu-Andrei Schipor, Doina-Maria Schipor, Emilia Crismariu, and Stefan-Gheorghe Pentiuc. 2011. Finding key emotional states to be recognized in a computer based speech therapy system. Procedia-Social and Behavioral Sciences, Vol. 30 (2011), 1177--1182.Google Scholar
Cross Ref
- Ovidiu-Andrei Schipor and Radu-Daniel Vatavu. 2018. Invisible, Inaudible, and Impalpable: Users' Preferences and Memory Performance for Digital Content in Thin Air. IEEE Pervasive Computing, Vol. 17, 4 (2018), 76--85.Google Scholar
Cross Ref
- Ovidiu-Andrei Schipor, Radu-Daniel Vatavu, and Jean Vanderdonckt. 2019. Euphoria: A Scalable, event-driven architecture for designing interactions across heterogeneous devices in smart environments. Information and Software Technology, Vol. 109 (May 2019), 43--59.Google Scholar
Digital Library
- Teddy Seyed, Alaa Azazi, Edwin Chan, Yuxi Wang, and Frank Maurer. 2015. Sod-toolkit: A toolkit for interactively prototyping and developing multi-sensor, multi-device environments. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces. ACM, 171--180.Google Scholar
Digital Library
- StackOverflow. 2018. Stack Overflow Developer Survey 2018. https://insights.stackoverflow.com/survey/2018#technology-programming-languagesGoogle Scholar
- Cristian Andy Tua nase, Radu-Daniel Vatavu, Stefan-Gheorghe Pentiuc, and Adrian Graur. 2008. Detecting and tracking multiple users in the proximity of interactive tabletops. Advances in Electrical and Computer Engineering, Vol. 8, 15 (2008), 61--64.Google Scholar
Cross Ref
- Peter Tolmie, James Pycock, Tim Diggins, Allan MacLean, and Alain Karsenty. 2002. Unremarkable Computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, New York, NY, USA, 399--406. Google Scholar
Digital Library
- Sandra Trullemans, Lars Van Holsbeeke, and Beat Signer. 2017. The context modeling toolkit: A unified multi-layered context modeling approach. Proceedings of the ACM on Human-Computer Interaction, Vol. 1, EICS (2017), 8. Google Scholar
Digital Library
- Radu-Daniel Vatavu. 2012a. Presence Bubbles: Supporting and Enhancing Human-Human Interaction with Ambient Media. Multimedia Tools and Applications, Vol. 58, 2 (2012), 371--383. Google Scholar
Digital Library
- Radu-Daniel Vatavu. 2012b. User-defined Gestures for Free-hand TV Control. In Proceedings of the 10th European Conference on Interactive TV and Video (EuroITV '12). ACM, New York, NY, USA, 45--48.Google Scholar
Digital Library
- Radu-Daniel Vatavu. 2013. On Designing Interactivity Awareness for Ambient Displays. Multimedia Tools and Applications, Vol. 66, 1 (2013), 59--80. Google Scholar
Digital Library
- Radu-Daniel Vatavu. 2015. Audience Silhouettes: Peripheral Awareness of Synchronous Audience Kinesics for Social Television. In Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video (TVX '15). ACM, New York, NY, USA, 13--22.Google Scholar
Digital Library
- Radu-Daniel Vatavu. 2017. Smart-Pockets: Body-Deictic Gestures for Fast Access to Personal Data during Ambient Interactions. International Journal of Human-Computer Studies, Vol. 103, C (July 2017), 1--21.Google Scholar
Digital Library
- Radu-Daniel Vatavu and Ionut-Alexandru Zaiti. 2014. Leap Gestures for TV: Insights from an Elicitation Study. In Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video (TVX '14). ACM, New York, NY, USA, 131--138. Google Scholar
Digital Library
- Jo Vermeulen, Steven Houben, and Nicolai Marquardt. 2016. Fluent Transitions Between Focused and Peripheral Interaction in Proxemic Interactions. In Peripheral Interaction. Springer, 137--163.Google Scholar
- Daniel Vogel and Ravin Balakrishnan. 2004. Interactive Public Ambient Displays: Transitioning from Implicit to Explicit, Public to Personal, Interaction with Multiple Users. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, New York, NY, USA, 137--146. Google Scholar
Digital Library
- Emily Vraga, Leticia Bode, and Sonya Troller-Renfree. 2016. Beyond self-reports: Using eye tracking to measure topic and style differences in attention to social media content. Communication Methods and Measures, Vol. 10, 2--3 (2016), 149--164.Google Scholar
Cross Ref
- Florian Wahl, Martin Freund, and Oliver Amft. 2015. WISEglass: Smart Eyeglasses Recognising Context. In Proceedings of the 10th EAI International Conference on Body Area Networks (BodyNets '15). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, Belgium, 11--17. Google Scholar
Digital Library
- Miaosen Wang, Sebastian Boring, and Saul Greenberg. 2012. Proxemic Peddler: A Public Advertising Display That Captures and Preserves the Attention of a Passerby. In Proceedings of the 2012 International Symposium on Pervasive Displays (PerDis '12). ACM, New York, NY, USA, Article 3, bibinfonumpages6 pages.Google Scholar
Digital Library
- Mark Weiser. 1999. The Computer for the 21st Century. SIGMOBILE Mob. Comput. Commun. Rev., Vol. 3, 3 (July 1999), 3--11. Google Scholar
Digital Library
- Mark Weiser and John Seely Brown. 1997. The coming age of calm technology. In Beyond calculation. Springer, 75--85.Google Scholar
Digital Library
- Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-defined Gestures for Surface Computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 1083--1092. Google Scholar
Digital Library
Index Terms
SAPIENS: Towards Software Architecture to Support Peripheral Interaction in Smart Environments
Recommendations
Performance evaluation of ambient services by combining robotic frameworks and a smart environment platform
Mobile robots and smart environments are two areas of research that can easily profit from each other. Smart environments, which are spaces unobtrusively equipped with sensors and actuators, providing ambient services to the people living within. Mobile ...
uCanvas: A Web Framework for Spontaneous Smartphone Interaction with Ubiquitous Displays
Human-Computer Interaction – INTERACT 2015AbstractIn recent years the presence of displays has become ubiquitous. They range from small-sized screens, such as smartphones or tablets to large screens as they are found in projection screens or public displays. Each display requires a unique ...
Exploring peripheral interaction design for primary school teachers
TEI '12: Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied InteractionThis paper explores the concept of peripheral interactions; interactions with technology that take place in the background or periphery of the attention. We present two designs for a classroom setting. CawClock makes selected time frames audible in ...






Comments