Abstract

Building multi-touch multi-user applications for Shared Interactive Surfaces is a complex endeavour that requires fundamental knowledge in touch enabling hardware, gesture recognition, graphical representation of digital information and multi-user interaction. While several specialised toolkits help developers in this effort, we identified a variety of challenges with these toolkits, as for example the lack of cross-platform support, the limited number of touch-enabled multi-user widgets, missing documentation, and lacking community support -- all raising the barriers to entry. In this paper, we present TUIOFX, a toolkit for developing multi-touch, multi-user applications for Shared Interactive Surfaces in Java, which tackles all of the identified problems. The sophisticated implementation of TUIOFX adds support for TUIO-enabled hardware and multi-user interaction under the hood of JavaFX, and leaves the well-learned JavaFX API for the developers fully intact -- thus allowing particularly novices a very quick start. In this paper we provide the technical insights, in the concepts and their elegant implementation.
- Hrvoje Benko, Shahram Izadi, Andrew D Wilson, Xiang Cao, Dan Rosenfeld and Ken Hinckley. Design and Evaluation of Interaction Models for Multi-touch Mice. In Proceedings of Graphics Interface 2010 - GI 2010 (May 31 - Jun 2, Ottawa, Canada). Canadian Information Processing Society, Toronto, Canada, 2010. pp. 253--260. Google Scholar
Digital Library
- Christophe Bortolaso, T. Nicholas C. Graham, Stacey D. Scott, Doug Brown and Liam Porter. Design of a Multi-Touch Tabletop for Simulation-Based Training. In Proceedings of International Command and Control Research and Technology Symposium - ICCRTS 2014 (Jun 16--19, Alexandria, VA, USA). International Command and Control Research Institute, Washington, DC, USA, 2014.Google Scholar
- Tom Brinck and Louis M. Gomez. A Collaborative Medium for the Support of Conversational Props. In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work - CSCW 1992 (Oct. 31 - Nov. 4, Toronto, Ontario, Canada). ACM Press, New York, NY, USA 1992. pp. 171--178. Google Scholar
Digital Library
- Andrew Clayphan, Anthony Collins, Christopher Ackad, Bob Kummerfeld and Judy Kay. Firestorm: A Brainstorming Application for Collaborative Group Work at Tabletops. In Proceedings of the 2011 International Conference on Interactive Tabletops & Surfaces - ITS 2011 (Oct. 30 - Nov. 2, Kobe, Japan). ACM Press, New York, NY, USA, 2011. pp. 162--171. Google Scholar
Digital Library
- Paul Dietz and Darren Leigh. Diamondtouch: A Multi-User Touch Technology. In Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology - UIST 2001 (Nov. 11--14, Orlando, FL, USA). ACM Press, New York, NY, USA, 2001. pp. 219--226. Google Scholar
Digital Library
- Florian Echtler and Andreas Butz. GISpL: Gestures Made Easy. In Proceedings of the Sixth International Conference on Tangible Embedded Interaction - TEI 2012 (Feb.19--22, Kingston, Ontario, Canada). ACM Press, New York, NY, USA, 2012. pp. 233--240. Google Scholar
Digital Library
- Florian Echtler and Gudrun Klinker. A Multitouch Software Architecture. In Proceedings of the 5th Nordic Conference on Human-Computer Interaction - NordiCHI 2008 (Lund, Sweden). ACM Press, New York, NY, USA, 2008. pp. 463--466. Google Scholar
Digital Library
- Mirko Fetter and David Bimamisa. TUIOFX-Toolkit Support for the Development of JavaFX Applications for Interactive Tabletops. In Proceedings of the 15th IFIP TC.13 International Conference on Human-Computer Interaction - INTERACT 2015 (Sept. 14--18, Bamberg, Germany). Springer, Heidelberg, Germany, 2015. pp. 476--479.Google Scholar
- Mirko Fetter, David Bimamisa and Tom Gross. Task-Based Focus and AdHoc-Focus-Territory-Novel Concepts for Shared Interactive Surfaces. In Extended Abstracts of the Conference on Human Factors in Computing Systems - CHI 2016 (May 7--12, San Jose, CA, USA). ACM Press, New York, NY, USA, 2016. pp. 1193--1200. Google Scholar
Digital Library
- Mirko Fetter, Tom Gross and Maxi Hucke. Supporting Social Protocols in Tabletop Interaction through Visual Cues. In Proceedings of the Thirteenth IFIP TC.13 International Conference on Human-Computer Interaction - INTERACT 2011 (Sept. 5--9, Lisbon, Portugal). Springer, Heidelberg, 2011. pp. 435--442. Google Scholar
Digital Library
- Mirko Fetter, Sascha Leicht, David Bimamisa and Tom Gross. Structuring Interaction in Group Decision Making on Tabletops. In Mensch & Computer - 13. Fachuebergreifende Konferenz fuer interaktive und kooperative Medien - M&C 2013 (Sept. 8--11, Bremen, Germany). Oldenbourg, Munich, Germany, 2013. pp. 277--280.Google Scholar
- Thomas E. Hansen, Juan Pablo Hourcade, Mathieu Virbel, Sharath Patali and Tiago Serra. PyMT: A post-WIMP Multi-touch User Interface Toolkit. In Proceedings of the International Conference on Interactive Tabletops and Surfaces - ITS 2009 (Nov. 23--25, Banff, Alberta, Canada). ACM Press, New York, NY, USA, 2009. pp. 17--24. Google Scholar
Digital Library
- Björn Hartmann, Meredith Ringel Morris, Hrvoje Benko and Andrew D. Wilson. Augmenting Interactive Tables with Mice & Keyboards. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology - UIST 2009 (Oct. 4--7, Victoria, BC, Canada). ACM Press, New York, NY, USA, 2009. pp. 149--152. Google Scholar
Digital Library
- Uta Hinrichs, Mark Hancock, Sheelagh Carpendale and Christopher Collins. Examination of Text-Entry Methods for Tabletop Displays. In Proceedings of the Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems - TABLETOP 2007 (Oct. 10--12, Newport, RI, USA). IEEE Computer Society, Los Alamitos, CA, USA, 2007. pp. 105--112.Google Scholar
- IntuiLab SA. IntuiFace. https://http://www.intuilab.com/, 2017. (Last accessed: 29/03//2017).Google Scholar
- Martin Kaltenbrunner, T Bovermann, Ross Bencina and E Costanza. TUIO - A Protocol for Table-Top Tangible User Interfaces. In Proceedings of the 6th International Workshop on Gesture in Human-Computer Interaction and Simulation - GW 2005 (May 18--20, Ile de Berder, France). 2005.Google Scholar
- Kenrick Kin, Björn Hartmann, Tony DeRose and Maneesh Agrawala. Proton++: A Customizable Declarative Multitouch Framework. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology - UIST 2012 (Oct. 7--10, Cambridge, MA, USA). ACM Press, New York, NY, USA, 2012. pp. 477--486. Google Scholar
Digital Library
- Ulrike Kister, Patrick Reipschlaeger, Fabrice Matulic and Raimund Dachselt. BodyLenses: Embodied Magic Lenses and Personal Territories for Wall Displays. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces - ITS 2015 (Nov. 15--18, Madeira, Portugal). ACM Press, New York, NY, USA, 2015. pp. 117--126. Google Scholar
Digital Library
- Kivy. Kivy: Cross-Platform Python Framework for NUI Development. http://kivy.org/, 2017. (Last accessed: 29/03//2017).Google Scholar
- Sungahn Ko, KyungTae Kim, Tejas Kulkarni and Niklas Elmqvist. Applying Mobile Device Soft Keyboards to Collaborative Multitouch Tabletop Displays. In Proceedings of the 2011 International Conference on Interactive Tabletops & Surfaces - ITS 2011 (Oct. 30 - Nov. 2, Kobe, Japan). ACM Press, New York, NY, USA, 2011. pp. 130--139. Google Scholar
Digital Library
- Werner A Koenig, Roman Raedle and Harald Reiterer. Squidy: A Zoomable Design Environment for Natural User Interfaces. In Extended Abstracts of the Conference on Human Factors in Computing Systems - CHI 2009 (Apr. 4--9, Boston, MA, USA). ACM Press, New York, NY, USA, 2009. pp. 4561--4566. Google Scholar
Digital Library
- Russell Kruger, Sheelagh Carpendale, Stacey D. Scott and Saul Greenberg. Roles of Orientation in Tabletop Collaboration: Comprehension, Coordination and Communication. Computer Supported Cooperative Work (CSCW) 13, 5--6 (2004). pp. 501--537. Google Scholar
Digital Library
- Russell Kruger, Sheelagh Carpendale, Stacey D. Scott and Anthony Tang. Fluid Integration of Rotation and Translation. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 2005 (Apr. 2--7, Portland, USA). ACM Press, New York, NY, USA, 2005. pp. 601--610. Google Scholar
Digital Library
- Uwe Laufs, Christopher Ruff and Jan Zibuschka. MT4j -- A Cross-platform Multi-touch Development Framework. CoRR - http://arxiv.org/abs/1012.0467, 2010. (Last accessed: 12/01/2017).Google Scholar
- I. Scott MacKenzie, Shawn X. Zhang and R. William Soukoreff. Text Entry Using Soft Keyboards. Behaviour & Information Technology 18, 4 (1999). pp. 235--244.Google Scholar
- Nicolai Marquardt, Johannes Kiemer, David Ledo, Sebastian Boring and Saul Greenberg. Designing User-, Hand-, and Handpart-aware Tabletop Interactions with the TouchID Toolkit. In Proceedings of the 2011 International Conference on Interactive Tabletops & Surfaces - ITS 2011 (Oct. 30 - Nov. 2, Kobe, Japan). ACM Press, New York, NY, USA, 2011. pp. 21--30. Google Scholar
Digital Library
- Microsoft Corporation. The Microsoft Surface 2.0 SDK. https://msdn.microsoft.com/en-us/library/ee692162(v=surface.10).aspx, 2014. (Last accessed: 29/03//2017).Google Scholar
- Meredith Ringel Morris, Anthony Cassanego, Andreas Paepcke, Terry Winograd, Anne Marie Piper and Anqi Huang. Mediating Group Dynamics through Tabletop Interface Design. IEEE Computer Graphics and Applications 26, 5 (2006). pp. 65--73. Google Scholar
Digital Library
- Meredith Ringel Morris, Jarrod Lombardo and Daniel Wigdor. WeSearch: Supporting Collaborative Search and Sensemaking on a Tabletop Display. In Proceedings of the ACM 2010 Conference on Computer-Supported Cooperative Work - CSCW 2010 (Feb. 6--10, Savannah, GA, USA). ACM Press, New York, NY, USA, 2010. pp. 401--410. Google Scholar
Digital Library
- Meredith Ringel Morris, Kathy Ryall, Chia Shen, Clifton Forlines and Frederic Vernier. Beyond "Social Protocols": Multi-User Coordination Policies for Co-located Groupware. In Proceedings of the 2004 ACM Conference on Computer-Supported Cooperative Work - CSCW 2004 (Nov. 6--10, Chicago, IL, USA). ACM Press, New York, NY, USA, 2004. pp. 262--265. Google Scholar
Digital Library
- Meredith Ringel Morris, Kathy Ryall, Chia Shen, Clifton Forlines and Frederic Vernier. Release, Relocate, Reorient, Resize. In Extended Abstracts of the Conference on Human Factors in Computing Systems - CHI 2004 (Apr. 24--29, Vienna, Austria). ACM Press, New York, NY, USA, 2004. pp. 1441--1444. Google Scholar
Digital Library
- MultiTouch Ltd. Cornerstone SDK. https://cornerstone.multitouch.fi/developer_guide, 2015. (Last accessed: 29/03//2017).Google Scholar
- Miguel A. Nacenta, David Pinelle, Dane Stuckel and Carl Gutwin. The Effects of Interaction Technique on Coordination in Tabletop Groupware. In Proceedings of Graphics Interface 2007 - GI 2007 (May 28 - 30, Montréal, Canada). Canadian Information Processing Society, Mississauga, Ontario, Canada, 2007. pp. 191--198. Google Scholar
Digital Library
- Michael Nebeling and Moira Norrie. jQMultiTouch: Lightweight Toolkit and Development Framework for Multi-touch/Multi-device Web Interfaces. In Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS 2012 (Jun. 25 - 28, Copenhagen, Denmark). ACM Press, New York, NY, USA, 2012. pp. 61--70. Google Scholar
Digital Library
- David Pinelle, Miguel Nacenta, Carl Gutwin and Tadeusz Stach. The Effects of Co-Present Embodiments on Awareness and Collaboration in Tabletop Groupware. In Proceedings of Graphics Interface 2008 - GI 2008 (May 28--30, Windsor, Ontario, Canada). Canadian Information Processing Society, Mississauga, Ontario, Canada, 2008. pp. 1--8. Google Scholar
Digital Library
- Mark Roseman and Saul Greenberg. Building Real Time Groupware with GroupKit, A Groupware Toolkit. ACM Transactions on Computer-Human Interaction (TOCHI) 3, 1 (March 1996). pp. 66--106. Google Scholar
Digital Library
- Volker Roth, Philipp Schmidt and Benjamin Gueldenring. The IR Ring: Authenticating Users' Touches on a Multi-touch Display. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology - UIST 2010 (Oct. 3--6, New York, NY, USA). ACM Press, New York, NY, USA, 2010. pp. 259--262. Google Scholar
Digital Library
- Kathy Ryall, A. Esenther, C. Forlines, C. Shen, S. Shipman, Meredith Ringel Morris, K. Everitt and F. D. Vernier. Identity-Differentiating Widgets for Multiuser Interactive Surfaces. IEEE Computer Graphics and Applications 26, 5 (2006). pp. 56--64. Google Scholar
Digital Library
- Dominik Schmidt, Ming Ki Chong and Hans W Gellersen. IdLenses: Dynamic Personal Areas on Shared Surfaces. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces - ITS 2010 (Nov. 7--10, Saarbrücken, Germany). ACM Press, New York, NY, USA, 2010. pp. 131--134. Google Scholar
Digital Library
- Stacey D. Scott, M. Sheelagh T Carpendale and Kori M Inkpen. Territoriality in Collaborative Tabletop Workspaces. In Proceedings of the 2004 ACM Conference on Computer-Supported Cooperative Work - CSCW 2004 (Nov. 6--10, Chicago, IL, USA). ACM Press, New York, NY, USA, 2004. pp. 294--303. Google Scholar
Digital Library
- Stacey D. Scott, Karen D. Grant and Regan L. Mandryk. System Guidelines for Co-located, Collaborative Work on a Tabletop Display. In Proceedings of the 8th European Conference on Computer-Supported Cooperative Work - ECSCW 2003 (Sept. 14--18, Helsinki, Finland). Kluwer Academic Publishers, Netherlands, 2003. pp. 159--178. Google Scholar
Digital Library
- Chia Shen, Frédéric D. Vernier, Clifton Forlines and Meredith Ringel. DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 2004 (Apr. 24--29, Vienna, Austria). ACM Press, New York, NY, USA, 2004. pp. 167--174. Google Scholar
Digital Library
- Jason Stewart, Benjamin B. Bederson and Allison Druin. Single Display Groupware: A Model for Co-present Collaboration. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 1999 (May 15--20, Pittsburgh, PA, USA). ACM Press, New York, NY, USA, 1999. pp. 286--293. Google Scholar
Digital Library
- John C. Tang. Findings from Observational Studies of Collaborative Work. International Journal of Man-Machine Studies 34, 2 (1991). pp. 143--160. Google Scholar
Digital Library
- Edward Tse, Jonathan Histon, Stacey D. Scott and Saul Greenberg. Avoiding Interference: How People Use Spatial Separation and Partitioning in SDG Workspace. In Proceedings of the ACM 2004 Conference on Computer-Supported Cooperative Work - CSCW 2004 (Nov. 6--10, Chicago, IL, USA). ACM, New York, NY, USA, 2004. pp. 252--261. Google Scholar
Digital Library
- Malte Weiss, Julie Wagner, Yvonne Jansen, Roger Jennings, Ramsin Khoshabeh, James D. Hollan and Jan Borchers. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 2006 (Apr. 22--27, Montreal, Canada). ACM Press, New York, NY, USA, 2006. pp. 481--490. Google Scholar
Digital Library
- Daniel Wigdor and Ravin Balakrishnan. Empirical Investigation into the Effect of Orientation on Text Readability in Tabletop Displays. In Proceedings of the Ninth Conference on European Conference on Computer-Supported Cooperative Work - ECSCW 2005 (Paris, France). Springer-Verlag, Berlin/Heidelberg, Germany, 2005. pp. 205--224. Google Scholar
Digital Library
Index Terms
TUIOFX: A JavaFX Toolkit for Shared Interactive Surfaces
Recommendations
TUIOFX—Toolkit Support for the Development of JavaFX Applications for Interactive Tabletops
Human-Computer Interaction – INTERACT 2015AbstractTUIOFX is a novel toolkit for developing multi-touch, multi-user applications for interactive tabletops and surfaces. By seamlessly integrating with JavaFX, TUIOFX provides a low entry barrier for developing state-of-the-art applications with ...
Multi-user chorded toolkit for multi-touch screens
EICS '11: Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systemsIn this work, we present the design and implementation of a chorded menu for multiple users on a large multi-touch vertical display. Instead of selecting an item in a fixed menu by reaching for it, users make a selection by touching multiple fingers ...
TouchAll: A Multi-Touch, Gestures, and Fiducials API for Flash/Action Script 3.0
MUE '11: Proceedings of the 2011 Fifth FTRA International Conference on Multimedia and Ubiquitous EngineeringIn this paper, we present Touch All, an high-level API (Application Programming Interface) for Flash Action Script 3.0, that supports all-in-one multi-touch, gestures, and fiducials, either through UDP, TCP, or Local Connection. Touch All API was built ...






Comments