Abstract

Whilst new patents and announcements advertise the technical availability of foldable displays, which are capable to be folded to some extent, there is still a lack of fundamental and applied understanding of how to model, to design, and to prototype graphical user interfaces for these devices before actually implementing them. Without waiting for their off-the-shelf availability and without being tied to any physical foldable mechanism, Flecto defines a model, an associated notation, and a supporting software for prototyping graphical user interfaces running on foldable displays, such as foldable smartphone or assemblies of foldable surfaces. For this purpose, we use an extended notation of the Yoshizawa-Randlett diagramming system, used to describe the folds of origami models, to characterize a foldable display and define possible interactive actions based on its folding operations. A guiding method for rapidly prototyping foldable user interfaces is devised and supported by Flecto, a design environment where foldable user interfaces are simulated in 3D environment instead of in physical reality. We report on a case study to demonstrate Flecto in action and we gather the feedback from users on Flecto, using Microsoft Product Reaction Cards.
Supplemental Material
- Joey Benedek and Trish Miner. 2002. Measuring Desirability: New methods for evaluating desirability in a usability lab setting. In Proceedings of UPA 2002 Conference (Orlando, FL, USA).Google Scholar
- Jesse Burstyn, Amartya Banerjee, and Roel Vertegaal. 2013. FlexView: An Evaluation of Depth Navigation on Deformable Mobile Devices. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction (Barcelona, Spain) (TEI '13). ACM, New York, NY, USA, 193--200. https://doi.org/10.1145/2460625.2460655Google Scholar
Digital Library
- Gaëlle Calvary, Joëlle Coutaz, David Thevenin, Quentin Limbourg, Laurent Bouillon, and Jean Vanderdonckt. 2003. A Unifying Reference Framework for multi-target user interfaces. Interacting with Computers 15, 3 (06 2003), 289--308. https://doi.org/10.1016/S0953--5438(03)00010--9 arXiv:http://oup.prod.sis.lan/iwc/article-pdf/15/3/289/7800242/iwc15- 0289.pdfGoogle Scholar
- M.Sheelagh T. Carpendale, David J. Cowperthwaite, F.David Fracchia, and Thomas Shermer. 1995. Graph Folding: Extending Detail and Context Viewing into a Tool for Subgraph Comparisons. In Proceedings of the Symposium on Graph Drawing. 127--139.Google Scholar
- Joëlle Coutaz, Christophe Lachenal, and Sophie Dupuy-Chessa. 2003. Ontology for Multi-surface Interaction. In Proceedings of 7th IFIP TC13 International Conference on Human-Computer Interaction (Zurich, Switzerland, September 1--5, 2003) (INTERACT'03), Matthias Rauterberg, Marino Menozzi, and Janet Wesson (Eds.). IOS Press, Amsterdam, 447--454.Google Scholar
- Pierre Dragicevic. 2004. Combining Crossing-based and Paper-based Interaction Paradigms for Dragging and Dropping Between Overlapping Windows. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (Santa Fe, NM, USA) (UIST '04). ACM, NewYork, NY, USA, 193--196. https://doi.org/10.1145/1029632.1029667 See video at https://www.youtube.com/watch'v=OvC2hKp6G8s.Google Scholar
Digital Library
- Alexandra Fuchs, Miriam Sturdee, and Johannes Schöning. 2018. Foldwatch: Using Origami-Inspired Paper Prototypes to Explore the Extension of Output Space in Smartwatches. In Proceedings of the 10th Nordic Conference on Human- Computer Interaction (Oslo, Norway) (NordiCHI '18). Association for Computing Machinery, New York, NY, USA, 47--59. https://doi.org/10.1145/3240167.3240173Google Scholar
Digital Library
- David T. Gallant, Andrew G. Seniuk, and Roel Vertegaal. 2008. Towards More Paper-like Input: Flexible Input Devices for Foldable Interaction Styles. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST '08). ACM, New York, NY, USA, 283--286. https://doi.org/10.1145/1449715.1449762 See video at https://www.youtube.com/watch?v=e6jJp-AWFFY&feature=emb_logo.Google Scholar
Digital Library
- Antonio Gomes and Roel Vertegaal. 2014. PaperFold: A Shape Changing Mobile Device with Multiple Reconfigurable Electrophoretic Magnetic Display Tiles. In Proceedings of ACM International Conference on Human Factors in Computing Systems, Extended Abstracts (Toronto, Ontario, Canada) (CHI EA '14). ACM, New York, NY, USA, 535--538. https: //doi.org/10.1145/2559206.2574770 See video at https://www.youtube.com/watch?v=IBdFQ6ICD-4.Google Scholar
Digital Library
- Antonio Gomes and Roel Vertegaal. 2015. PaperFold: Evaluating Shape Changes for Viewport Transformations in Foldable Thin-Film Display Devices. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (Stanford, California, USA) (TEI '15). ACM, New York, NY, USA, 153--160. https://doi.org/10. 1145/2677199.2680572 See video at https://www.youtube.com/watch?v=IBdFQ6ICD-4.Google Scholar
Digital Library
- Yukinori Hidaka, Ken Ootsuka, and Naoto Kinoshita. 2018. DISPLAY DEVICE AND ELECTRONIC APPARATUS. http://www.freepatentsonline.com/y2018/0336851.html Proc. ACM Hum.-Comput. Interact., Vol. 4, No. ISS, Article 194. Publication date: November 2020. Towards Rapid Prototyping of Foldable Graphical User Interfaces with Flecto 194:21Google Scholar
- David Holman, Jesse Burstyn, Ryan Brotman, Audrey Younkin, and Roel Vertegaal. 2013. Flexkit: A Rapid Prototyping Platform for Flexible Displays. In Proceedings of the Adjunct Publication of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST '13 Adjunct). ACM, New York, NY, USA, 17--18. https://doi.org/10.1145/2508468.2514934Google Scholar
Digital Library
- David Holman and Roel Vertegaal. 2008. Organic User Interfaces: Designing Computers in Any Way, Shape, or Form. Commun. ACM 51, 6 (June 2008), 48--55. https://doi.org/10.1145/1349026.1349037Google Scholar
Digital Library
- David Holman, Roel Vertegaal, Mark Altosaar, Nikolaus Troje, and Derek Johns. 2005. Paper Windows: Interaction Techniques for Digital Paper. In Proceedings of the ACM International Conference on Human Factors in Computing Systems (Portland, Oregon, USA) (CHI '05). ACM, New York, NY, USA, 591--599. https://doi.org/10.1145/1054972.1055054Google Scholar
Digital Library
- Iyad Khaddam, Hanaa Barakat, and Jean Vanderdonckt. 2016. Enactment of User Interface Development Methods in Software Life Cycles. In 13th International Conference on Human Computer Interaction, RoCHI 2016, Iasi, Romania, September 8--9, 2016, Adrian Iftene and Jean Vanderdonckt (Eds.). Matrix Rom, 26--35.Google Scholar
- I. Khaddam, N. Mezhoudi, and J. Vanderdonckt. 2015. Adapt-first: A MDE transformation approach for supporting user interface adaptation. In 2015 2nd World Symposium on Web Applications and Networking (WSWAN). 1--9. https: //doi.org/10.1109/WSWAN.2015.7209080Google Scholar
Cross Ref
- Iyad Khaddam, Nesrine Mezhoudi, and Jean Vanderdonckt. 2015. Towards Task-Based Linguistic Modeling for Designing GUIs. In Proceedings of the 27th Conference on l?Interaction Homme-Machine (Toulouse, France) (IHM '15). Association for Computing Machinery, New York, NY, USA, Article 17, 10 pages. https://doi.org/10.1145/2820619.2820636Google Scholar
Digital Library
- Mohammadreza Khalilbeigi, Roman Lissermann, Wolfgang Kleine, and Jürgen Steimle. 2012. FoldMe: Interacting with Double-sided Foldable Displays. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (Kingston, Ontario, Canada) (TEI '12). ACM, New York, NY, USA, 33--40. https://doi.org/10.1145/ 2148131.2148142 See video at https://www.youtube.com/watch?v=LZyClk2rslI.Google Scholar
Digital Library
- Hoonsik Kim, Young Do Kim, and Suk Choi. 2018. Foldable display. http://www.freepatentsonline.com/9891670.htmlGoogle Scholar
- Yuichiro Kinoshita, Kentaro Go, Reiji Kozono, and Kohei Kaneko. 2014. Origami Tessellation Display: Interaction Techniques Using Origami-Based Deformable Surfaces. In CHI '14 Extended Abstracts on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI EA '14). Association for Computing Machinery, New York, NY, USA, 1837--1842. https://doi.org/10.1145/2559206.2581172Google Scholar
Digital Library
- Hyuk-Jun Kwon, HongShik Shim, Sunkook Kim, Woong Choi, Youngtea Chun, InSeo Kee, and SangYoon Lee. 2011. Mechanically and optically reliable folding structure with a hyperelastic material for seamless foldable displays. Applied Physics Letters 98, 15 (2011), 151904. https://doi.org/10.1063/1.3576906 arXiv:https://doi.org/10.1063/1.3576906Google Scholar
Cross Ref
- Byron Lahey, Audrey Girouard, Winslow Burleson, and Roel Vertegaal. 2011. PaperPhone: Understanding the Use of Bend Gestures in Mobile Devices with Flexible Electronic Paper Displays. In Proceedings of the ACM International Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI '11). ACM, New York, NY, USA, 1303--1312. https://doi.org/10.1145/1978942.1979136 See video at https://www.youtube.com/watch?v=_69pIorQ59U.Google Scholar
Digital Library
- Robert J. Lang. 2011. Origami Diagramming Conventions - Diagramming, Part 1. Web site. Retrieved December 12, 2019 from http://langorigami.com/article/origami-diagramming-conventions/.Google Scholar
- Haklim Lee, Hanseok Chae, and Wonseok Joo. 2018. MOBILE TERMINAL. http://www.freepatentsonline.com/y2018/ 0183911.htmlGoogle Scholar
- Johnny C. Lee, Scott E. Hudson, and Edward Tse. 2008. Foldable Interactive Displays. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST '08). ACM, New York, NY, USA, 287--290. https://doi.org/10.1145/1449715.1449763Google Scholar
Digital Library
- Jessica Lo and Audrey Girouard. 2017. Bendy: Exploring Mobile Gaming with Flexible Devices. In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction (Yokohama, Japan) (TEI '17). ACM, New York, NY, USA, 163--172. https://doi.org/10.1145/3024969.3024970 See video at https://www.youtube.com/watch? v=mgG7E7p8SoY.Google Scholar
Digital Library
- Nicolai Marquardt, Frederik Brudy, Can Liu, Ben Bengler, and Christian Holz. 2018. SurfaceConstellations: A Modular Hardware Platform for Ad-Hoc Reconfigurable Cross-Device Workspaces. In Proceedings of the ACM International Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). ACM, New York, NY, USA, Article 354, 14 pages. https://doi.org/10.1145/3173574.3173928 See video at https://www.youtube.com/watch?v=--Yxc7lD2gg.Google Scholar
Digital Library
- José Pascual Molina Massó, Jean Vanderdonckt, and Pascual González López. 2006. Direct Manipulation of User Interfaces for Migration. In Proceedings of the 11th International Conference on Intelligent User Interfaces (Sydney, Australia) (IUI '06). ACM, New York, NY, USA, 140--147. https://doi.org/10.1145/1111449.1111483 See video at https://www.youtube.com/watch?v=w-lG9Ho7z54.Google Scholar
Digital Library
- Simon Olberding, Sergio Soto Ortega, Klaus Hildebrandt, and Jürgen Steimle. 2015. Foldio: Digital Fabrication of Interactive and Shape-Changing Objects With Foldable Printed Electronics. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST '15). ACM, New York, NY, USA, 223--232. https://doi.org/10.1145/2807442.2807494 See video at https://www.youtube.com/watch?v=gA17s9MyfiM. Proc. ACM Hum.-Comput. Interact., Vol. 4, No. ISS, Article 194. Publication date: November 2020. 194:22 Iyad Khaddam et al.Google Scholar
- Michael Ortega and Alix Goguey. 2019. BEXHI: A Mechanical Structure for Prototyping Bendable and EXpandable Handheld Interfaces. In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces (Daejeon, Republic of Korea) (ISS '19). Association for Computing Machinery, New York, NY, USA, 269--273. https: //doi.org/10.1145/3343055.3359703Google Scholar
Digital Library
- Michaël Ortega, Jérôme Maisonnasse, and Laurence Nigay. 2017. EXHI-Bit: A Mechanical Structure for Prototyping EXpandable Handheld Interfaces. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Vienna, Austria) (MobileHCI '17). Association for Computing Machinery, New York, NY, USA, Article 4, 11 pages. https://doi.org/10.1145/3098279.3098533Google Scholar
Digital Library
- Julian Petford, Miguel A. Nacenta, Carl Gutwin, Joseph Eremondi, and Cody Ede. 2016. The ASPECTA Toolkit: Affordable Full Coverage Displays. In Proceedings of the 5th ACM International Symposium on Pervasive Displays (Oulu, Finland) (PerDis '16). Association for Computing Machinery, New York, NY, USA, 87--105. https://doi.org/10.1145/ 2914920.2915006Google Scholar
Digital Library
- Jürgen Steimle, Andreas Jordt, and Pattie Maes. 2013. Flexpad: Highly Flexible Bending Interactions for Projected Handheld Displays. In Proceedings of the ACM International Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). ACM, New York, NY, USA, 237--246. https://doi.org/10.1145/2470654.2470688 See video at Video:https://www.youtube.com/watch?v=AZwUW7HqcQg.Google Scholar
Digital Library
- Miriam Sturdee and Jason Alexander. 2018. Analysis and Classification of Shape-Changing Interfaces for Design and Application-Based Research. ACM Comput. Surv. 51, 1, Article 2 (Jan. 2018), 32 pages. https://doi.org/10.1145/3143559Google Scholar
Digital Library
- Dominique Tan, Maciej Kumorek, Andres A. Garcia, Adam Mooney, and Derek Bekoe. 2015. Projectagami: A Foldable Mobile Device with Shape Interactive Applications. In Proceedings of the 33rd ACM International Conference on Human Factors in Computing Systems, Extended Abstracts (Seoul, Republic of Korea) (CHI EA '15). ACM, New York, NY, USA, 1555--1560. https://doi.org/10.1145/2702613.2732801 See video at https://www.youtube.com/watch?v=AhTHDxe_lqI.Google Scholar
Digital Library
- Aneesh P. Tarun, Byron Lahey, Audrey Girouard, Winslow Burleson, and Roel Vertegaal. 2011. Snaplet: Using Body Shape to Inform Function in Mobile Flexible Display Devices. In Proceedings of ACM International Conference on Human Factors in Computing Systems, Extended Abstracts (Vancouver, BC, Canada) (CHI EA '11). ACM, New York, NY, USA, 329--334. https://doi.org/10.1145/1979742.1979701 See video at https://www.youtube.com/watch?v=ol_uu5pMmq8.Google Scholar
Digital Library
- Jean Vanderdonckt, Iyad Khaddam, and Radu-Daniel Vatavu. 2020. The Foldinterface Editor: A Visual Tool for Designing User Interfaces for Foldable Displays. In Companion Proceedings of the 12th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Sophia Antipolis, France) (EICS '20 Companion). Association for Computing Machinery, New York, NY, USA, Article 1, 6 pages. https://doi.org/10.1145/3393672.3398490Google Scholar
Digital Library
- Jun-ichiro Watanabe, Arito Mochizuki, and Youichi Horry. 2008. Bookisheet: Bendable Device for Browsing Content Using the Metaphor of Leafing Through the Pages. In Proceedings of the 10th International Conference on Ubiquitous Computing (Seoul, Korea) (UbiComp '08). ACM, New York, NY, USA, 360--369. https://doi.org/10.1145/1409635.1409684 See video at http://mobblog.cs.ucl.ac.uk/category/ubicomp2008/.Google Scholar
- MichaelWessely, Nadiya Morenko, Jürgen Steimle, and Michael Schmitz. 2018. Interactive Tangrami: Rapid Prototyping with Modular Paper-folded Electronics. In Proceedings of 31st Annual ACM Symposium on User Interface Software and Technology, Adjunct Proceedings (Berlin, Germany) (UIST '18). ACM, New York, NY, USA, 143--145. https: //doi.org/10.1145/3266037.3271630 See video at https://www.youtube.com/watch?v=A8VCApp_R6M.Google Scholar
- Zi Ye and Hammad Khalid. 2010. Cobra: Flexible Displays for Mobilegaming Scenarios. In Proceedings of the ACM International Conference on Human Factors in Computing Systems, Extended Abstracts (Atlanta, Georgia, USA) (CHI EA '10). ACM, New York, NY, USA, 4363--4368. https://doi.org/10.1145/1753846.1754154Google Scholar
Digital Library
- Zhen Zhang, Victor H. Yin, Cyrus Y. Liu, Paul S. Drzaic, Sungwon Bae, Chun-hao Tung, Kiarash Vakhshouri, Sunggu Kang, and John Z. Zhong. 2017. Electronic Devices With Flexible Displays. http://www.freepatentsonline.com/y2017/ 0336831.htmlGoogle Scholar
- Kening Zhu, Morten Fjeld, and Ayça Ünlüer. 2018. WristOrigami: Exploring Foldable Design for Multi-Display Smartwatch. In Proceedings of the ACM International Conference on Designing Interactive (Hong Kong, China) (DIS '18). ACM, New York, NY, USA, 1207--1218. https://doi.org/10.1145/3196709.3196713 See video at https://www.youtube. com/watch?v=1_2D79zntIk.Google Scholar
Digital Library
Index Terms
Towards Rapid Prototyping of Foldable Graphical User Interfaces with Flecto
Recommendations
Towards virtualization of user interfaces based on UsiXML
Web3D '05: Proceedings of the tenth international conference on 3D Web technologyA model-based approach is presented for structuring a development process of virtual user interfaces based on UsiXML, a XML-compliant User Interface Description Language. UsiXML provides a Concrete User Interface description that remains independent ...
Lessons learned from SUIT, the simple user interface toolkit
Special issue on user interface software and technologyIn recent years, the computer science community has realized the advantages of GUIs (Graphical User Interfaces). Because high-quality GUIs are difficult to build, support tools such as UIMSs, UI Toolkits, and Interface Builders have been developed. ...
Rapidly Prototyping Marker Based Tangible User Interfaces
VMR '09: Proceedings of the 3rd International Conference on Virtual and Mixed Reality: Held as Part of HCI International 2009Tangible user interfaces (TUIs) can create engaging and useful interactive systems. However, along with the power of these interfaces comes challenges; they are often so specialized and novel that building a TUI system involves working at a low level ...






Comments