skip to main content
research-article

Towards Rapid Prototyping of Foldable Graphical User Interfaces with Flecto

Authors Info & Claims
Published:04 November 2020Publication History
Skip Abstract Section

Abstract

Whilst new patents and announcements advertise the technical availability of foldable displays, which are capable to be folded to some extent, there is still a lack of fundamental and applied understanding of how to model, to design, and to prototype graphical user interfaces for these devices before actually implementing them. Without waiting for their off-the-shelf availability and without being tied to any physical foldable mechanism, Flecto defines a model, an associated notation, and a supporting software for prototyping graphical user interfaces running on foldable displays, such as foldable smartphone or assemblies of foldable surfaces. For this purpose, we use an extended notation of the Yoshizawa-Randlett diagramming system, used to describe the folds of origami models, to characterize a foldable display and define possible interactive actions based on its folding operations. A guiding method for rapidly prototyping foldable user interfaces is devised and supported by Flecto, a design environment where foldable user interfaces are simulated in 3D environment instead of in physical reality. We report on a case study to demonstrate Flecto in action and we gather the feedback from users on Flecto, using Microsoft Product Reaction Cards.

Skip Supplemental Material Section

Supplemental Material

V4issA194.ISS2020.VIDEO.mp4

Supplemental video

References

  1. Joey Benedek and Trish Miner. 2002. Measuring Desirability: New methods for evaluating desirability in a usability lab setting. In Proceedings of UPA 2002 Conference (Orlando, FL, USA).Google ScholarGoogle Scholar
  2. Jesse Burstyn, Amartya Banerjee, and Roel Vertegaal. 2013. FlexView: An Evaluation of Depth Navigation on Deformable Mobile Devices. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction (Barcelona, Spain) (TEI '13). ACM, New York, NY, USA, 193--200. https://doi.org/10.1145/2460625.2460655Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Gaëlle Calvary, Joëlle Coutaz, David Thevenin, Quentin Limbourg, Laurent Bouillon, and Jean Vanderdonckt. 2003. A Unifying Reference Framework for multi-target user interfaces. Interacting with Computers 15, 3 (06 2003), 289--308. https://doi.org/10.1016/S0953--5438(03)00010--9 arXiv:http://oup.prod.sis.lan/iwc/article-pdf/15/3/289/7800242/iwc15- 0289.pdfGoogle ScholarGoogle Scholar
  4. M.Sheelagh T. Carpendale, David J. Cowperthwaite, F.David Fracchia, and Thomas Shermer. 1995. Graph Folding: Extending Detail and Context Viewing into a Tool for Subgraph Comparisons. In Proceedings of the Symposium on Graph Drawing. 127--139.Google ScholarGoogle Scholar
  5. Joëlle Coutaz, Christophe Lachenal, and Sophie Dupuy-Chessa. 2003. Ontology for Multi-surface Interaction. In Proceedings of 7th IFIP TC13 International Conference on Human-Computer Interaction (Zurich, Switzerland, September 1--5, 2003) (INTERACT'03), Matthias Rauterberg, Marino Menozzi, and Janet Wesson (Eds.). IOS Press, Amsterdam, 447--454.Google ScholarGoogle Scholar
  6. Pierre Dragicevic. 2004. Combining Crossing-based and Paper-based Interaction Paradigms for Dragging and Dropping Between Overlapping Windows. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (Santa Fe, NM, USA) (UIST '04). ACM, NewYork, NY, USA, 193--196. https://doi.org/10.1145/1029632.1029667 See video at https://www.youtube.com/watch'v=OvC2hKp6G8s.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Alexandra Fuchs, Miriam Sturdee, and Johannes Schöning. 2018. Foldwatch: Using Origami-Inspired Paper Prototypes to Explore the Extension of Output Space in Smartwatches. In Proceedings of the 10th Nordic Conference on Human- Computer Interaction (Oslo, Norway) (NordiCHI '18). Association for Computing Machinery, New York, NY, USA, 47--59. https://doi.org/10.1145/3240167.3240173Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. David T. Gallant, Andrew G. Seniuk, and Roel Vertegaal. 2008. Towards More Paper-like Input: Flexible Input Devices for Foldable Interaction Styles. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST '08). ACM, New York, NY, USA, 283--286. https://doi.org/10.1145/1449715.1449762 See video at https://www.youtube.com/watch?v=e6jJp-AWFFY&feature=emb_logo.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Antonio Gomes and Roel Vertegaal. 2014. PaperFold: A Shape Changing Mobile Device with Multiple Reconfigurable Electrophoretic Magnetic Display Tiles. In Proceedings of ACM International Conference on Human Factors in Computing Systems, Extended Abstracts (Toronto, Ontario, Canada) (CHI EA '14). ACM, New York, NY, USA, 535--538. https: //doi.org/10.1145/2559206.2574770 See video at https://www.youtube.com/watch?v=IBdFQ6ICD-4.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Antonio Gomes and Roel Vertegaal. 2015. PaperFold: Evaluating Shape Changes for Viewport Transformations in Foldable Thin-Film Display Devices. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (Stanford, California, USA) (TEI '15). ACM, New York, NY, USA, 153--160. https://doi.org/10. 1145/2677199.2680572 See video at https://www.youtube.com/watch?v=IBdFQ6ICD-4.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Yukinori Hidaka, Ken Ootsuka, and Naoto Kinoshita. 2018. DISPLAY DEVICE AND ELECTRONIC APPARATUS. http://www.freepatentsonline.com/y2018/0336851.html Proc. ACM Hum.-Comput. Interact., Vol. 4, No. ISS, Article 194. Publication date: November 2020. Towards Rapid Prototyping of Foldable Graphical User Interfaces with Flecto 194:21Google ScholarGoogle Scholar
  12. David Holman, Jesse Burstyn, Ryan Brotman, Audrey Younkin, and Roel Vertegaal. 2013. Flexkit: A Rapid Prototyping Platform for Flexible Displays. In Proceedings of the Adjunct Publication of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST '13 Adjunct). ACM, New York, NY, USA, 17--18. https://doi.org/10.1145/2508468.2514934Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. David Holman and Roel Vertegaal. 2008. Organic User Interfaces: Designing Computers in Any Way, Shape, or Form. Commun. ACM 51, 6 (June 2008), 48--55. https://doi.org/10.1145/1349026.1349037Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. David Holman, Roel Vertegaal, Mark Altosaar, Nikolaus Troje, and Derek Johns. 2005. Paper Windows: Interaction Techniques for Digital Paper. In Proceedings of the ACM International Conference on Human Factors in Computing Systems (Portland, Oregon, USA) (CHI '05). ACM, New York, NY, USA, 591--599. https://doi.org/10.1145/1054972.1055054Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Iyad Khaddam, Hanaa Barakat, and Jean Vanderdonckt. 2016. Enactment of User Interface Development Methods in Software Life Cycles. In 13th International Conference on Human Computer Interaction, RoCHI 2016, Iasi, Romania, September 8--9, 2016, Adrian Iftene and Jean Vanderdonckt (Eds.). Matrix Rom, 26--35.Google ScholarGoogle Scholar
  16. I. Khaddam, N. Mezhoudi, and J. Vanderdonckt. 2015. Adapt-first: A MDE transformation approach for supporting user interface adaptation. In 2015 2nd World Symposium on Web Applications and Networking (WSWAN). 1--9. https: //doi.org/10.1109/WSWAN.2015.7209080Google ScholarGoogle ScholarCross RefCross Ref
  17. Iyad Khaddam, Nesrine Mezhoudi, and Jean Vanderdonckt. 2015. Towards Task-Based Linguistic Modeling for Designing GUIs. In Proceedings of the 27th Conference on l?Interaction Homme-Machine (Toulouse, France) (IHM '15). Association for Computing Machinery, New York, NY, USA, Article 17, 10 pages. https://doi.org/10.1145/2820619.2820636Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Mohammadreza Khalilbeigi, Roman Lissermann, Wolfgang Kleine, and Jürgen Steimle. 2012. FoldMe: Interacting with Double-sided Foldable Displays. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (Kingston, Ontario, Canada) (TEI '12). ACM, New York, NY, USA, 33--40. https://doi.org/10.1145/ 2148131.2148142 See video at https://www.youtube.com/watch?v=LZyClk2rslI.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Hoonsik Kim, Young Do Kim, and Suk Choi. 2018. Foldable display. http://www.freepatentsonline.com/9891670.htmlGoogle ScholarGoogle Scholar
  20. Yuichiro Kinoshita, Kentaro Go, Reiji Kozono, and Kohei Kaneko. 2014. Origami Tessellation Display: Interaction Techniques Using Origami-Based Deformable Surfaces. In CHI '14 Extended Abstracts on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI EA '14). Association for Computing Machinery, New York, NY, USA, 1837--1842. https://doi.org/10.1145/2559206.2581172Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Hyuk-Jun Kwon, HongShik Shim, Sunkook Kim, Woong Choi, Youngtea Chun, InSeo Kee, and SangYoon Lee. 2011. Mechanically and optically reliable folding structure with a hyperelastic material for seamless foldable displays. Applied Physics Letters 98, 15 (2011), 151904. https://doi.org/10.1063/1.3576906 arXiv:https://doi.org/10.1063/1.3576906Google ScholarGoogle ScholarCross RefCross Ref
  22. Byron Lahey, Audrey Girouard, Winslow Burleson, and Roel Vertegaal. 2011. PaperPhone: Understanding the Use of Bend Gestures in Mobile Devices with Flexible Electronic Paper Displays. In Proceedings of the ACM International Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI '11). ACM, New York, NY, USA, 1303--1312. https://doi.org/10.1145/1978942.1979136 See video at https://www.youtube.com/watch?v=_69pIorQ59U.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Robert J. Lang. 2011. Origami Diagramming Conventions - Diagramming, Part 1. Web site. Retrieved December 12, 2019 from http://langorigami.com/article/origami-diagramming-conventions/.Google ScholarGoogle Scholar
  24. Haklim Lee, Hanseok Chae, and Wonseok Joo. 2018. MOBILE TERMINAL. http://www.freepatentsonline.com/y2018/ 0183911.htmlGoogle ScholarGoogle Scholar
  25. Johnny C. Lee, Scott E. Hudson, and Edward Tse. 2008. Foldable Interactive Displays. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST '08). ACM, New York, NY, USA, 287--290. https://doi.org/10.1145/1449715.1449763Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Jessica Lo and Audrey Girouard. 2017. Bendy: Exploring Mobile Gaming with Flexible Devices. In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction (Yokohama, Japan) (TEI '17). ACM, New York, NY, USA, 163--172. https://doi.org/10.1145/3024969.3024970 See video at https://www.youtube.com/watch? v=mgG7E7p8SoY.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Nicolai Marquardt, Frederik Brudy, Can Liu, Ben Bengler, and Christian Holz. 2018. SurfaceConstellations: A Modular Hardware Platform for Ad-Hoc Reconfigurable Cross-Device Workspaces. In Proceedings of the ACM International Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). ACM, New York, NY, USA, Article 354, 14 pages. https://doi.org/10.1145/3173574.3173928 See video at https://www.youtube.com/watch?v=--Yxc7lD2gg.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. José Pascual Molina Massó, Jean Vanderdonckt, and Pascual González López. 2006. Direct Manipulation of User Interfaces for Migration. In Proceedings of the 11th International Conference on Intelligent User Interfaces (Sydney, Australia) (IUI '06). ACM, New York, NY, USA, 140--147. https://doi.org/10.1145/1111449.1111483 See video at https://www.youtube.com/watch?v=w-lG9Ho7z54.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Simon Olberding, Sergio Soto Ortega, Klaus Hildebrandt, and Jürgen Steimle. 2015. Foldio: Digital Fabrication of Interactive and Shape-Changing Objects With Foldable Printed Electronics. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST '15). ACM, New York, NY, USA, 223--232. https://doi.org/10.1145/2807442.2807494 See video at https://www.youtube.com/watch?v=gA17s9MyfiM. Proc. ACM Hum.-Comput. Interact., Vol. 4, No. ISS, Article 194. Publication date: November 2020. 194:22 Iyad Khaddam et al.Google ScholarGoogle Scholar
  30. Michael Ortega and Alix Goguey. 2019. BEXHI: A Mechanical Structure for Prototyping Bendable and EXpandable Handheld Interfaces. In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces (Daejeon, Republic of Korea) (ISS '19). Association for Computing Machinery, New York, NY, USA, 269--273. https: //doi.org/10.1145/3343055.3359703Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Michaël Ortega, Jérôme Maisonnasse, and Laurence Nigay. 2017. EXHI-Bit: A Mechanical Structure for Prototyping EXpandable Handheld Interfaces. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Vienna, Austria) (MobileHCI '17). Association for Computing Machinery, New York, NY, USA, Article 4, 11 pages. https://doi.org/10.1145/3098279.3098533Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Julian Petford, Miguel A. Nacenta, Carl Gutwin, Joseph Eremondi, and Cody Ede. 2016. The ASPECTA Toolkit: Affordable Full Coverage Displays. In Proceedings of the 5th ACM International Symposium on Pervasive Displays (Oulu, Finland) (PerDis '16). Association for Computing Machinery, New York, NY, USA, 87--105. https://doi.org/10.1145/ 2914920.2915006Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Jürgen Steimle, Andreas Jordt, and Pattie Maes. 2013. Flexpad: Highly Flexible Bending Interactions for Projected Handheld Displays. In Proceedings of the ACM International Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). ACM, New York, NY, USA, 237--246. https://doi.org/10.1145/2470654.2470688 See video at Video:https://www.youtube.com/watch?v=AZwUW7HqcQg.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Miriam Sturdee and Jason Alexander. 2018. Analysis and Classification of Shape-Changing Interfaces for Design and Application-Based Research. ACM Comput. Surv. 51, 1, Article 2 (Jan. 2018), 32 pages. https://doi.org/10.1145/3143559Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Dominique Tan, Maciej Kumorek, Andres A. Garcia, Adam Mooney, and Derek Bekoe. 2015. Projectagami: A Foldable Mobile Device with Shape Interactive Applications. In Proceedings of the 33rd ACM International Conference on Human Factors in Computing Systems, Extended Abstracts (Seoul, Republic of Korea) (CHI EA '15). ACM, New York, NY, USA, 1555--1560. https://doi.org/10.1145/2702613.2732801 See video at https://www.youtube.com/watch?v=AhTHDxe_lqI.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Aneesh P. Tarun, Byron Lahey, Audrey Girouard, Winslow Burleson, and Roel Vertegaal. 2011. Snaplet: Using Body Shape to Inform Function in Mobile Flexible Display Devices. In Proceedings of ACM International Conference on Human Factors in Computing Systems, Extended Abstracts (Vancouver, BC, Canada) (CHI EA '11). ACM, New York, NY, USA, 329--334. https://doi.org/10.1145/1979742.1979701 See video at https://www.youtube.com/watch?v=ol_uu5pMmq8.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Jean Vanderdonckt, Iyad Khaddam, and Radu-Daniel Vatavu. 2020. The Foldinterface Editor: A Visual Tool for Designing User Interfaces for Foldable Displays. In Companion Proceedings of the 12th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Sophia Antipolis, France) (EICS '20 Companion). Association for Computing Machinery, New York, NY, USA, Article 1, 6 pages. https://doi.org/10.1145/3393672.3398490Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Jun-ichiro Watanabe, Arito Mochizuki, and Youichi Horry. 2008. Bookisheet: Bendable Device for Browsing Content Using the Metaphor of Leafing Through the Pages. In Proceedings of the 10th International Conference on Ubiquitous Computing (Seoul, Korea) (UbiComp '08). ACM, New York, NY, USA, 360--369. https://doi.org/10.1145/1409635.1409684 See video at http://mobblog.cs.ucl.ac.uk/category/ubicomp2008/.Google ScholarGoogle Scholar
  39. MichaelWessely, Nadiya Morenko, Jürgen Steimle, and Michael Schmitz. 2018. Interactive Tangrami: Rapid Prototyping with Modular Paper-folded Electronics. In Proceedings of 31st Annual ACM Symposium on User Interface Software and Technology, Adjunct Proceedings (Berlin, Germany) (UIST '18). ACM, New York, NY, USA, 143--145. https: //doi.org/10.1145/3266037.3271630 See video at https://www.youtube.com/watch?v=A8VCApp_R6M.Google ScholarGoogle Scholar
  40. Zi Ye and Hammad Khalid. 2010. Cobra: Flexible Displays for Mobilegaming Scenarios. In Proceedings of the ACM International Conference on Human Factors in Computing Systems, Extended Abstracts (Atlanta, Georgia, USA) (CHI EA '10). ACM, New York, NY, USA, 4363--4368. https://doi.org/10.1145/1753846.1754154Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Zhen Zhang, Victor H. Yin, Cyrus Y. Liu, Paul S. Drzaic, Sungwon Bae, Chun-hao Tung, Kiarash Vakhshouri, Sunggu Kang, and John Z. Zhong. 2017. Electronic Devices With Flexible Displays. http://www.freepatentsonline.com/y2017/ 0336831.htmlGoogle ScholarGoogle Scholar
  42. Kening Zhu, Morten Fjeld, and Ayça Ünlüer. 2018. WristOrigami: Exploring Foldable Design for Multi-Display Smartwatch. In Proceedings of the ACM International Conference on Designing Interactive (Hong Kong, China) (DIS '18). ACM, New York, NY, USA, 1207--1218. https://doi.org/10.1145/3196709.3196713 See video at https://www.youtube. com/watch?v=1_2D79zntIk.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Towards Rapid Prototyping of Foldable Graphical User Interfaces with Flecto

                Recommendations

                Comments

                Login options

                Check if you have access through your login credentials or your institution to get full access on this article.

                Sign in

                Full Access

                PDF Format

                View or Download as a PDF file.

                PDF

                eReader

                View online with eReader.

                eReader
                About Cookies On This Site

                We use cookies to ensure that we give you the best experience on our website.

                Learn more

                Got it!