skip to main content
research-article

TUIOFX: A JavaFX Toolkit for Shared Interactive Surfaces

Published:30 June 2017Publication History
Skip Abstract Section

Abstract

Building multi-touch multi-user applications for Shared Interactive Surfaces is a complex endeavour that requires fundamental knowledge in touch enabling hardware, gesture recognition, graphical representation of digital information and multi-user interaction. While several specialised toolkits help developers in this effort, we identified a variety of challenges with these toolkits, as for example the lack of cross-platform support, the limited number of touch-enabled multi-user widgets, missing documentation, and lacking community support -- all raising the barriers to entry. In this paper, we present TUIOFX, a toolkit for developing multi-touch, multi-user applications for Shared Interactive Surfaces in Java, which tackles all of the identified problems. The sophisticated implementation of TUIOFX adds support for TUIO-enabled hardware and multi-user interaction under the hood of JavaFX, and leaves the well-learned JavaFX API for the developers fully intact -- thus allowing particularly novices a very quick start. In this paper we provide the technical insights, in the concepts and their elegant implementation.

References

  1. Hrvoje Benko, Shahram Izadi, Andrew D Wilson, Xiang Cao, Dan Rosenfeld and Ken Hinckley. Design and Evaluation of Interaction Models for Multi-touch Mice. In Proceedings of Graphics Interface 2010 - GI 2010 (May 31 - Jun 2, Ottawa, Canada). Canadian Information Processing Society, Toronto, Canada, 2010. pp. 253--260. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Christophe Bortolaso, T. Nicholas C. Graham, Stacey D. Scott, Doug Brown and Liam Porter. Design of a Multi-Touch Tabletop for Simulation-Based Training. In Proceedings of International Command and Control Research and Technology Symposium - ICCRTS 2014 (Jun 16--19, Alexandria, VA, USA). International Command and Control Research Institute, Washington, DC, USA, 2014.Google ScholarGoogle Scholar
  3. Tom Brinck and Louis M. Gomez. A Collaborative Medium for the Support of Conversational Props. In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work - CSCW 1992 (Oct. 31 - Nov. 4, Toronto, Ontario, Canada). ACM Press, New York, NY, USA 1992. pp. 171--178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Andrew Clayphan, Anthony Collins, Christopher Ackad, Bob Kummerfeld and Judy Kay. Firestorm: A Brainstorming Application for Collaborative Group Work at Tabletops. In Proceedings of the 2011 International Conference on Interactive Tabletops & Surfaces - ITS 2011 (Oct. 30 - Nov. 2, Kobe, Japan). ACM Press, New York, NY, USA, 2011. pp. 162--171. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Paul Dietz and Darren Leigh. Diamondtouch: A Multi-User Touch Technology. In Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology - UIST 2001 (Nov. 11--14, Orlando, FL, USA). ACM Press, New York, NY, USA, 2001. pp. 219--226. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Florian Echtler and Andreas Butz. GISpL: Gestures Made Easy. In Proceedings of the Sixth International Conference on Tangible Embedded Interaction - TEI 2012 (Feb.19--22, Kingston, Ontario, Canada). ACM Press, New York, NY, USA, 2012. pp. 233--240. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Florian Echtler and Gudrun Klinker. A Multitouch Software Architecture. In Proceedings of the 5th Nordic Conference on Human-Computer Interaction - NordiCHI 2008 (Lund, Sweden). ACM Press, New York, NY, USA, 2008. pp. 463--466. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Mirko Fetter and David Bimamisa. TUIOFX-Toolkit Support for the Development of JavaFX Applications for Interactive Tabletops. In Proceedings of the 15th IFIP TC.13 International Conference on Human-Computer Interaction - INTERACT 2015 (Sept. 14--18, Bamberg, Germany). Springer, Heidelberg, Germany, 2015. pp. 476--479.Google ScholarGoogle Scholar
  9. Mirko Fetter, David Bimamisa and Tom Gross. Task-Based Focus and AdHoc-Focus-Territory-Novel Concepts for Shared Interactive Surfaces. In Extended Abstracts of the Conference on Human Factors in Computing Systems - CHI 2016 (May 7--12, San Jose, CA, USA). ACM Press, New York, NY, USA, 2016. pp. 1193--1200. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Mirko Fetter, Tom Gross and Maxi Hucke. Supporting Social Protocols in Tabletop Interaction through Visual Cues. In Proceedings of the Thirteenth IFIP TC.13 International Conference on Human-Computer Interaction - INTERACT 2011 (Sept. 5--9, Lisbon, Portugal). Springer, Heidelberg, 2011. pp. 435--442. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Mirko Fetter, Sascha Leicht, David Bimamisa and Tom Gross. Structuring Interaction in Group Decision Making on Tabletops. In Mensch & Computer - 13. Fachuebergreifende Konferenz fuer interaktive und kooperative Medien - M&C 2013 (Sept. 8--11, Bremen, Germany). Oldenbourg, Munich, Germany, 2013. pp. 277--280.Google ScholarGoogle Scholar
  12. Thomas E. Hansen, Juan Pablo Hourcade, Mathieu Virbel, Sharath Patali and Tiago Serra. PyMT: A post-WIMP Multi-touch User Interface Toolkit. In Proceedings of the International Conference on Interactive Tabletops and Surfaces - ITS 2009 (Nov. 23--25, Banff, Alberta, Canada). ACM Press, New York, NY, USA, 2009. pp. 17--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Björn Hartmann, Meredith Ringel Morris, Hrvoje Benko and Andrew D. Wilson. Augmenting Interactive Tables with Mice & Keyboards. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology - UIST 2009 (Oct. 4--7, Victoria, BC, Canada). ACM Press, New York, NY, USA, 2009. pp. 149--152. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Uta Hinrichs, Mark Hancock, Sheelagh Carpendale and Christopher Collins. Examination of Text-Entry Methods for Tabletop Displays. In Proceedings of the Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems - TABLETOP 2007 (Oct. 10--12, Newport, RI, USA). IEEE Computer Society, Los Alamitos, CA, USA, 2007. pp. 105--112.Google ScholarGoogle Scholar
  15. IntuiLab SA. IntuiFace. https://http://www.intuilab.com/, 2017. (Last accessed: 29/03//2017).Google ScholarGoogle Scholar
  16. Martin Kaltenbrunner, T Bovermann, Ross Bencina and E Costanza. TUIO - A Protocol for Table-Top Tangible User Interfaces. In Proceedings of the 6th International Workshop on Gesture in Human-Computer Interaction and Simulation - GW 2005 (May 18--20, Ile de Berder, France). 2005.Google ScholarGoogle Scholar
  17. Kenrick Kin, Björn Hartmann, Tony DeRose and Maneesh Agrawala. Proton++: A Customizable Declarative Multitouch Framework. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology - UIST 2012 (Oct. 7--10, Cambridge, MA, USA). ACM Press, New York, NY, USA, 2012. pp. 477--486. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Ulrike Kister, Patrick Reipschlaeger, Fabrice Matulic and Raimund Dachselt. BodyLenses: Embodied Magic Lenses and Personal Territories for Wall Displays. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces - ITS 2015 (Nov. 15--18, Madeira, Portugal). ACM Press, New York, NY, USA, 2015. pp. 117--126. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Kivy. Kivy: Cross-Platform Python Framework for NUI Development. http://kivy.org/, 2017. (Last accessed: 29/03//2017).Google ScholarGoogle Scholar
  20. Sungahn Ko, KyungTae Kim, Tejas Kulkarni and Niklas Elmqvist. Applying Mobile Device Soft Keyboards to Collaborative Multitouch Tabletop Displays. In Proceedings of the 2011 International Conference on Interactive Tabletops & Surfaces - ITS 2011 (Oct. 30 - Nov. 2, Kobe, Japan). ACM Press, New York, NY, USA, 2011. pp. 130--139. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Werner A Koenig, Roman Raedle and Harald Reiterer. Squidy: A Zoomable Design Environment for Natural User Interfaces. In Extended Abstracts of the Conference on Human Factors in Computing Systems - CHI 2009 (Apr. 4--9, Boston, MA, USA). ACM Press, New York, NY, USA, 2009. pp. 4561--4566. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Russell Kruger, Sheelagh Carpendale, Stacey D. Scott and Saul Greenberg. Roles of Orientation in Tabletop Collaboration: Comprehension, Coordination and Communication. Computer Supported Cooperative Work (CSCW) 13, 5--6 (2004). pp. 501--537. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Russell Kruger, Sheelagh Carpendale, Stacey D. Scott and Anthony Tang. Fluid Integration of Rotation and Translation. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 2005 (Apr. 2--7, Portland, USA). ACM Press, New York, NY, USA, 2005. pp. 601--610. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Uwe Laufs, Christopher Ruff and Jan Zibuschka. MT4j -- A Cross-platform Multi-touch Development Framework. CoRR - http://arxiv.org/abs/1012.0467, 2010. (Last accessed: 12/01/2017).Google ScholarGoogle Scholar
  25. I. Scott MacKenzie, Shawn X. Zhang and R. William Soukoreff. Text Entry Using Soft Keyboards. Behaviour & Information Technology 18, 4 (1999). pp. 235--244.Google ScholarGoogle Scholar
  26. Nicolai Marquardt, Johannes Kiemer, David Ledo, Sebastian Boring and Saul Greenberg. Designing User-, Hand-, and Handpart-aware Tabletop Interactions with the TouchID Toolkit. In Proceedings of the 2011 International Conference on Interactive Tabletops & Surfaces - ITS 2011 (Oct. 30 - Nov. 2, Kobe, Japan). ACM Press, New York, NY, USA, 2011. pp. 21--30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Microsoft Corporation. The Microsoft Surface 2.0 SDK. https://msdn.microsoft.com/en-us/library/ee692162(v=surface.10).aspx, 2014. (Last accessed: 29/03//2017).Google ScholarGoogle Scholar
  28. Meredith Ringel Morris, Anthony Cassanego, Andreas Paepcke, Terry Winograd, Anne Marie Piper and Anqi Huang. Mediating Group Dynamics through Tabletop Interface Design. IEEE Computer Graphics and Applications 26, 5 (2006). pp. 65--73. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Meredith Ringel Morris, Jarrod Lombardo and Daniel Wigdor. WeSearch: Supporting Collaborative Search and Sensemaking on a Tabletop Display. In Proceedings of the ACM 2010 Conference on Computer-Supported Cooperative Work - CSCW 2010 (Feb. 6--10, Savannah, GA, USA). ACM Press, New York, NY, USA, 2010. pp. 401--410. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Meredith Ringel Morris, Kathy Ryall, Chia Shen, Clifton Forlines and Frederic Vernier. Beyond "Social Protocols": Multi-User Coordination Policies for Co-located Groupware. In Proceedings of the 2004 ACM Conference on Computer-Supported Cooperative Work - CSCW 2004 (Nov. 6--10, Chicago, IL, USA). ACM Press, New York, NY, USA, 2004. pp. 262--265. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Meredith Ringel Morris, Kathy Ryall, Chia Shen, Clifton Forlines and Frederic Vernier. Release, Relocate, Reorient, Resize. In Extended Abstracts of the Conference on Human Factors in Computing Systems - CHI 2004 (Apr. 24--29, Vienna, Austria). ACM Press, New York, NY, USA, 2004. pp. 1441--1444. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. MultiTouch Ltd. Cornerstone SDK. https://cornerstone.multitouch.fi/developer_guide, 2015. (Last accessed: 29/03//2017).Google ScholarGoogle Scholar
  33. Miguel A. Nacenta, David Pinelle, Dane Stuckel and Carl Gutwin. The Effects of Interaction Technique on Coordination in Tabletop Groupware. In Proceedings of Graphics Interface 2007 - GI 2007 (May 28 - 30, Montréal, Canada). Canadian Information Processing Society, Mississauga, Ontario, Canada, 2007. pp. 191--198. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Michael Nebeling and Moira Norrie. jQMultiTouch: Lightweight Toolkit and Development Framework for Multi-touch/Multi-device Web Interfaces. In Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS 2012 (Jun. 25 - 28, Copenhagen, Denmark). ACM Press, New York, NY, USA, 2012. pp. 61--70. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. David Pinelle, Miguel Nacenta, Carl Gutwin and Tadeusz Stach. The Effects of Co-Present Embodiments on Awareness and Collaboration in Tabletop Groupware. In Proceedings of Graphics Interface 2008 - GI 2008 (May 28--30, Windsor, Ontario, Canada). Canadian Information Processing Society, Mississauga, Ontario, Canada, 2008. pp. 1--8. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Mark Roseman and Saul Greenberg. Building Real Time Groupware with GroupKit, A Groupware Toolkit. ACM Transactions on Computer-Human Interaction (TOCHI) 3, 1 (March 1996). pp. 66--106. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Volker Roth, Philipp Schmidt and Benjamin Gueldenring. The IR Ring: Authenticating Users' Touches on a Multi-touch Display. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology - UIST 2010 (Oct. 3--6, New York, NY, USA). ACM Press, New York, NY, USA, 2010. pp. 259--262. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Kathy Ryall, A. Esenther, C. Forlines, C. Shen, S. Shipman, Meredith Ringel Morris, K. Everitt and F. D. Vernier. Identity-Differentiating Widgets for Multiuser Interactive Surfaces. IEEE Computer Graphics and Applications 26, 5 (2006). pp. 56--64. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Dominik Schmidt, Ming Ki Chong and Hans W Gellersen. IdLenses: Dynamic Personal Areas on Shared Surfaces. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces - ITS 2010 (Nov. 7--10, Saarbrücken, Germany). ACM Press, New York, NY, USA, 2010. pp. 131--134. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Stacey D. Scott, M. Sheelagh T Carpendale and Kori M Inkpen. Territoriality in Collaborative Tabletop Workspaces. In Proceedings of the 2004 ACM Conference on Computer-Supported Cooperative Work - CSCW 2004 (Nov. 6--10, Chicago, IL, USA). ACM Press, New York, NY, USA, 2004. pp. 294--303. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Stacey D. Scott, Karen D. Grant and Regan L. Mandryk. System Guidelines for Co-located, Collaborative Work on a Tabletop Display. In Proceedings of the 8th European Conference on Computer-Supported Cooperative Work - ECSCW 2003 (Sept. 14--18, Helsinki, Finland). Kluwer Academic Publishers, Netherlands, 2003. pp. 159--178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Chia Shen, Frédéric D. Vernier, Clifton Forlines and Meredith Ringel. DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 2004 (Apr. 24--29, Vienna, Austria). ACM Press, New York, NY, USA, 2004. pp. 167--174. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Jason Stewart, Benjamin B. Bederson and Allison Druin. Single Display Groupware: A Model for Co-present Collaboration. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 1999 (May 15--20, Pittsburgh, PA, USA). ACM Press, New York, NY, USA, 1999. pp. 286--293. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. John C. Tang. Findings from Observational Studies of Collaborative Work. International Journal of Man-Machine Studies 34, 2 (1991). pp. 143--160. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Edward Tse, Jonathan Histon, Stacey D. Scott and Saul Greenberg. Avoiding Interference: How People Use Spatial Separation and Partitioning in SDG Workspace. In Proceedings of the ACM 2004 Conference on Computer-Supported Cooperative Work - CSCW 2004 (Nov. 6--10, Chicago, IL, USA). ACM, New York, NY, USA, 2004. pp. 252--261. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Malte Weiss, Julie Wagner, Yvonne Jansen, Roger Jennings, Ramsin Khoshabeh, James D. Hollan and Jan Borchers. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 2006 (Apr. 22--27, Montreal, Canada). ACM Press, New York, NY, USA, 2006. pp. 481--490. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Daniel Wigdor and Ravin Balakrishnan. Empirical Investigation into the Effect of Orientation on Text Readability in Tabletop Displays. In Proceedings of the Ninth Conference on European Conference on Computer-Supported Cooperative Work - ECSCW 2005 (Paris, France). Springer-Verlag, Berlin/Heidelberg, Germany, 2005. pp. 205--224. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. TUIOFX: A JavaFX Toolkit for Shared Interactive Surfaces

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Article Metrics

        • Downloads (Last 12 months)14
        • Downloads (Last 6 weeks)0

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!