Abstract
Copy-and-paste operations are the most popular features on computing devices such as desktop computers, smartphones and tablets. However, the copy-and-paste operations are not sufficiently addressed on the Augmented Reality (AR) smartglasses designated for real-time interaction with texts in physical environments. This paper proposes two system solutions, namely Granularity Scrolling (GS) and Two Ends (TE), for the copy-and-paste operations on AR smartglasses. By leveraging a thumb-size button on a touch-sensitive and pressure-sensitive surface, both the multi-step solutions can capture the target texts through indirect manipulation and subsequently enables the copy-and-paste operations. Based on the system solutions, we implemented an experimental prototype named Press-n-Paste (PnP). After the eight-session evaluation capturing 1,296 copy-and-paste operations, 18 participants with GS and TE achieve the peak performance of 17,574 ms and 13,951 ms per copy-and-paste operation, with 93.21% and 98.15% accuracy rates respectively, which are as good as the commercial solutions using direct manipulation on touchscreen devices. The user footprints also show that PnP has a distinctive feature of miniaturized interaction area within 12.65 mm * 14.48 mm. PnP not only proves the feasibility of copy-and-paste operations with the flexibility of various granularities on AR smartglasses, but also gives significant implications to the design space of pressure widgets as well as the input design on smart wearables.
- Toshiyuki Ando, Toshiya Isomoto, Buntarou Shizuki, and Shin Takahashi. 2018. Press Tilt: One-Handed Text Selection and Command Execution on Smartphone. In Proceedings of the 30th Australian Conference on Computer-Human Interaction (OzCHI '18). Association for Computing Machinery, New York, NY, USA, 401--405. https://doi.org/10.1145/3292147.3292178 Google Scholar
Digital Library
- Axel Antoine, Sylvain Malacria, and Géry Casiez. 2017. ForceEdge: Controlling Autoscroll on Both Desktop and Mobile Computers Using the Force. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). Association for Computing Machinery, New York, NY, USA, 3281--3292. https://doi.org/10.1145/3025453.3025605 Google Scholar
Digital Library
- Shiri Azenkot and Nicole B. Lee. 2013. Exploring the Use of Speech Input by Blind People on Mobile Devices. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '13). Association for Computing Machinery, New York, NY, USA, Article Article 11, 8 pages. https://doi.org/10.1145/2513383.2513440 Google Scholar
Digital Library
- Daniel Buschek, Alexander De Luca, and Florian Alt. 2015. Improving Accuracy, Applicability and Usability of Keystroke Biometrics on Mobile Touchscreen Devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). Association for Computing Machinery, New York, NY, USA, 1393--140 https://doi.org/10.1145/2702123.2702252 Google Scholar
Digital Library
- Olivier Chapuis and Nicolas Roussel. 2007. Copy-and-Paste between Overlapping Windows. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). Association for Computing Machinery, New York, NY, USA, 201--210. https://doi.org/10.1145/1240624.1240657 Google Scholar
Digital Library
- Chen Chen, Simon T. Perrault, Shengdong Zhao, and Wei Tsang Ooi. 2014. BezelCopy: An Efficient Cross-Application Copy-Paste Technique for Touchscreen Smartphones. In Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces (AVI '14). Association for Computing Machinery, New York, NY, USA, 185--192. https://doi.org/10.1145/2598153.2598162 Google Scholar
Digital Library
- Christian Corsten, Marcel Lahaye, Jan Borchers, and Simon Voelker. 2019. ForceRay: Extending Thumb Reach via Force Input Stabilizes Device Grip for Mobile Touch Input. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Article Paper 212, 12 pages. https://doi.org/10.1145/3290605.3300442 Google Scholar
Digital Library
- Christian Corsten, Simon Voelker, and Jan Borchers. 2017. Release, Don't Wait! Reliable Force Input Confirmation with Quick Release. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). Association for Computing Machinery, New York, NY, USA, 246--251. https://doi.org/10.1145/3132272.3134116 Google Scholar
Digital Library
- Vittorio Fuccella, Poika Isokoski, and Benoit Martin. 2013. Gestures and Widgets: Performance in Text Editing on Multi-Touch Capable Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). Association for Computing Machinery, New York, NY, USA, 2785--2794. https://doi.org/10.1145/2470654.2481385 Google Scholar
Digital Library
- Alix Goguey, Sylvain Malacria, and Carl Gutwin. 2018. Improving Discoverability and Expert Performance in Force-Sensitive Text Selection for Touch Devices with Mode Gauges. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). Association for Computing Machinery, New York, NY, USA, Article Paper 477, 12 pages. https://doi.org/10.1145/3173574.3174051 Google Scholar
Digital Library
- Yizheng Gu, Chun Yu, Zhipeng Li, Weiqi Li, Shuchang Xu, Xiaoying Wei, and Yuanchun Shi. 20Accurate and Low-Latency Sensing of Touch Contact on Any Surface with Finger-Worn IMU Sensor. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST '19). Association for Computing Machinery, New York, NY, USA, 1059--1070. https://doi.org/10.1145/3332165.3347947 Google Scholar
Digital Library
- Seongkook Heo and Geehyuk Lee. 2011. Force Gestures: Augmenting Touch Screen Gestures with Normal and Tangential Forces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). Association for Computing Machinery, New York, NY, USA, 621--626. https://doi.org/10.1145/2047196.2047278 Google Scholar
Digital Library
- Jonggi Hong and Leah Findlater. 2018. Identifying Speech Input Errors Through Audio-Only Interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). Association for Computing Machinery, New York, NY, USA, Article Paper 567, 12 pages. https://doi.org/10.1145/3173574.3174141 Google Scholar
Digital Library
- Eva Hornecker. 2012. Beyond Affordance: Tangibles' Hybrid Nature. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI '12). Association for Computing Machinery, New York, NY, USA, 175--182. https://doi.org/10.1145/2148131.2148168 Google Scholar
Digital Library
- Weiyang Huan, Huawei Tu, and Zhuying Li. 2019. Enabling Finger Pointing Based Text Selection on Touchscreen Mobile Devices. In Proceedings of the Seventh International Symposium of Chinese CHI (Chinese CHI '19). Association for Computing Machinery, New York, NY, USA, 93--96. https://doi.org/10.1145/3332169.3332172 Google Scholar
Digital Library
- NASA AMES Research Center Human Performance Research Group. 1999. NASA Task Load Index (TLX). https://humansystems.arc.nasa.gov/groups/TLX/downloads/TLX.pdfGoogle Scholar
- Kaori Ikematsu and Itiro Siio. 2013. Memory Stones: An Intuitive Copy-and-Paste Method between Multi-Touch Computers. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). Association for Computing Machinery, New York, NY, USA, 1287--1292. https://doi.org/10.1145/2468356.2468586 Google Scholar
Digital Library
- Howell Istance, Richard Bates, Aulikki Hyrskykari, and Stephen Vickers. 2008. Snap Clutch, a Moded Approach to Solving the Midas Touch Problem. Eye Tracking Research and Applications Symposium (ETRA), 221--228. https://doi.org/10.1145/1344471.1344523 Google Scholar
Digital Library
- Ricardo Jota, Albert Ng, Paul Dietz, and Daniel Wigdor. 2013. How Fast is Fast Enough? A Study of the Effects of Latency in Direct-Touch Pointing Tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). Association for Computing Machinery, New York, NY, USA, 2291--2300. https://doi.org/10.1145/2470654.2481317 Google Scholar
Digital Library
- R. Kishi and T. Hayashi. 2015. Effective gazewriting with support of text copy and paste. In 2015 IEEE/ACIS 14th International Conference on Computer and Information Science (ICIS). 125--130. https://doi.org/10.1109/ICIS.2015.7166581Google Scholar
Cross Ref
- Abinaya Kumar, Aishwarya Radjesh, Sven Mayer, and Huy Viet Le. 2019. Improving the Input Accuracy of Touchscreens Using Deep Learning. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). Association for Computing Machinery, New York, NY, USA, Article Paper LBW1514, 6 pages. https://doi.org/10.1145/3290607.3312928 Google Scholar
Digital Library
- Jianwei Lai, Navid Rajabi, and Elahe Javadi. 2019. A Shortcut for Caret Positioning on Touch-Screen Phones. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '19). Association for Computing Machinery, New York, NY, USA, Article Article 35, 6 pages. https://doi.org/10.1145/3338286.3340146 Google Scholar
Digital Library
- K. Y. Lam, L. H. Lee, T. Braud, and P. Hui. 2019. M2A: A Framework for Visualizing Information from Mobile Web to Mobile Augmented Reality. In 2019 IEEE International Conference on Pervasive Computing and Communications (PerCom. 1--10. https://doi.org/10.1109/ PERCOM.2019.8767388Google Scholar
Cross Ref
- Felix Lauber, Anna Follmann, and Andreas Butz. 2014. What You See is What You Touch: Visualizing Touch Screen Interaction in the Head-up Display. In Proceedings of the 2014 Conference on Designing Interactive Systems (DIS '14). Association for Computing Machinery, New York, NY, USA, 171--180. https://doi.org/10.1145/2598510.2598521 Google Scholar
Digital Library
- L. Lee and P. Hui. 2018. Interaction Methods for Smart Glasses: A Survey. IEEE Access 6 (2018), 28712--28732. https://doi.org/10.1109/ ACCESS.2018.2831081Google Scholar
Cross Ref
- Lik Hang Lee, Tristan Braud, Farshid Hassani Bijarbooneh, and Pan Hui. 2019. TiPoint: Detecting Fingertip for Mid-Air Interaction on Computational Resource Constrained Smartglasses. In Proceedings of the 23rd International Symposium on Wearable Computers (ISWC '19). Association for Computing Machinery, New York, NY, USA, 118--12 https://doi.org/10.1145/3341163.3347723 Google Scholar
Digital Library
- Lik Hang Lee, Kit Yung Lam, Tong Li, Tristan Braud, Xiang Su, and Pan Hui. 2019. Quadmetric Optimized Thumb-to-Finger Interaction for Force Assisted One-Handed Text Entry on Mobile Headsets. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 3, Article Article 94 (Sept. 2019), 27 pages. https://doi.org/10.1145/3351252 Google Scholar
Digital Library
- L. H. Lee, K. Yung Lam, Y. P. Yau, T. Braud, and P. Hui. 2019. HIBEY: Hide the Keyboard in Augmented Reality. In 2019 IEEE International Conference on Pervasive Computing and Communications (PerCom. 1--10. https://doi.org/10.1109/PERCOM.2019.8767420Google Scholar
- Hannah Limerick, James W. Moore, and David Coyle. 2015. Empirical Evidence for a Diminished Sense of Agency in Speech Interfaces. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). Association for Computing Machinery, New York, NY, USA, 3967--3970. https://doi.org/10.1145/2702123.2702379 Google Scholar
Digital Library
- Sylvain Malacria, Jonathan Aceituno, Philip Quinn, Géry Casiez, Andy Cockburn, and Nicolas Roussel. 2015. Push-Edge and Slide-Edge: Scrolling by Pushing Against the Viewport Edge. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). Association for Computing Machinery, New York, NY, USA, 2773--277 https://doi.org/10.1145/2702123.2702132 Google Scholar
Digital Library
- Yuki Matsuura, Tsutomu Terada, Tomohiro Aoki, Susumu Sonoda, Naoya Isoyama, and Masahiko Tsukamoto. 2019. Readability and Legibility of Fonts Considering Shakiness of Head Mounted Displays. In Proceedings of the 23rd International Symposium on Wearable Computers (ISWC '19). Association for Computing Machinery, New York, NY, USA, 150--159. https://doi.org/10.1145/3341163.3347748 Google Scholar
Digital Library
- Motoki Miura and Kenji Saisho. 2014. A Text Selection Technique Using Word Snapping. Procedia Computer Science 35 (2014), 1644 -- 1651. https://doi.org/10.1016/j.procs.2014.08.257 Knowledge-Based and Intelligent Information Engineering Systems 18th Annual Conference, KES-2014 Gdynia, Poland, September 2014 Proceedings.Google Scholar
Cross Ref
- Sachi Mizobuchi, Shinya Terasaki, Turo Keski-Jaskari, Jari Nousiainen, Matti Ryynanen, and Miika Silfverberg. 2005. Making an Impression: Force-Controlled Pen Input for Handheld Devices. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA '05). Association for Computing Machinery, New York, NY, USA, 1661--1664. https://doi.org/10.1145/1056808.1056991 Google Scholar
Digital Library
- P. Mohan, W. B. Goh, C. Fu, and S. Yeung. 2018. DualGaze: Addressing the Midas Touch Problem in Gaze Mediated VR Interaction. In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 79--84.Google Scholar
- Alex Olwal, Steven Feiner, and Susanna Heyman. 2008. Rubbing and Tapping for Precise and Rapid Selection on Touch-Screen Displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). Association for Computing Machinery, New York, NY, USA, 295--304. https://doi.org/10.1145/1357054.1357105 Google Scholar
Digital Library
- Henri Palleis, Julie Wagner, and Heinrich Hussmann. 2016. Novel Indirect Touch Input Techniques Applied to Finger-Forming 3D Models. In Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '16). Association for Computing Machinery, New York, NY, USA, 228--235. https://doi.org/10.1145/2909132.2909257 Google Scholar
Digital Library
- X. Qin, Z. Zhang, C. Huang, C. Gao, M. Dehghan, and M. Jagersand. 2019. BASNet: Boundary-Aware Salient Object Detection. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7471--7481.Google Scholar
- Gonzalo Ramos and Ravin Balakrishnan. 2005. Zliding: Fluid Zooming and Sliding for High Precision Parameter Manipulation. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST '05). Association for Computing Machinery, New York, NY, USA, 143--152. https://doi.org/10.1145/1095034.1095059 Google Scholar
Digital Library
- Gonzalo Ramos, Matthew Boulos, and Ravin Balakrishnan. 2004. Pressure Widgets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). Association for Computing Machinery, New York, NY, USA, 487--494. https://doi.org/10.1145/ 985692.985754 Google Scholar
Digital Library
- Gonzalo Ramos, Matthew Boulos, and Ravin Balakrishnan. 2004. Pressure Widgets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). Association for Computing Machinery, New York, NY, USA, 487--494. https://doi.org/10.1145/ 985692.985754 Google Scholar
Digital Library
- Umar Rashid, Miguel A. Nacenta, and Aaron Quigley. 2012. Factors Influencing Visual Attention Switch in Multi-Display User Interfaces: A Survey. In Proceedings of the 2012 International Symposium on Pervasive Displays (PerDis '12). Association for Computing Machinery, New York, NY, USA, Article Article 1, 6 pages. https://doi.org/10.1145/2307798.2307799 Google Scholar
Digital Library
- Radiah Rivu, Yasmeen Abdrabou, Ken Pfeuffer, Mariam Hassib, and Florian Alt. 2020. Gaze'N'Touch: Enhancing Text Selection on Mobile Devices Using Gaze. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1--8. https://doi.org/10.1145/3334480.3382802 Google Scholar
Digital Library
- Sherry Ruan, Jacob O. Wobbrock, Kenny Liou, Andrew Ng, and James A. Landay. 2018. Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4, Article Article 159 (Jan. 2018), 23 pages. https://doi.org/10.1145/3161187 Google Scholar
Digital Library
- Dominik Schmidt, Corina Sas, and Hans Gellersen. 2013. Personal Clipboards for Individual Copy-and-Paste on Shared Multi-User Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). Association for Computing Machinery, New York, NY, USA, 3335--3344. https://doi.org/10.1145/2470654.2466457 Google Scholar
Digital Library
- Roy Shilkrot, Jochen Huber, Jürgen Steimle, Suranga Nanayakkara, and Pattie Maes. 2015. Digital Digits: A Comprehensive Survey of Finger Augmentation Devices. ACM Comput. Surv. 48, 2, Article Article 30 (Nov. 2015), 29 pages. https://doi.org/10.1145/2828993 Google Scholar
Digital Library
- Jeffrey Stylos, Brad A. Myers, and Andrew Faulring. 2004. Citrine: Providing Intelligent Copy-and-Paste. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). Association for Computing Machinery, New York, NY, USA, 185--188. https://doi.org/10.1145/1029632.1029665 Google Scholar
Digital Library
- Kenji Suzuki, Kazumasa Okabe, Ryuuki Sakamoto, and Daisuke Sakamoto. 2016. Fix and Slide: Caret Navigation with Movable Background. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '16). Association for Computing Machinery, New York, NY, USA, 478--482. https://doi.org/10.1145/2935334.2935357 Google Scholar
Digital Library
- Kenji Suzuki, Ryuuki Sakamoto, Daisuke Sakamoto, and Tetsuo Ono. 2018. Pressure-Sensitive Zooming-out Interfaces for One-Handed Mobile Interaction. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '18). Association for Computing Machinery, New York, NY, USA, Article Article 30, 8 pages. https://doi.org/10.1145/3229434.3229446 Google Scholar
Digital Library
- Eric Whitmire, Mohit Jain, Divye Jain, Greg Nelson, Ravi Karkar, Shwetak Patel, and Mayank Goel. 2017. DigiTouch: Reconfigurable Thumb-to-Finger Input and Text Entry on Head-Mounted Displays. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article Article 113 (Sept. 2017), 21 pages. https://doi.org/10.1145/3130978 Google Scholar
Digital Library
- Graham Wilson, Craig Stewart, and Stephen A. Brewster. 2010. Pressure-Based Menu Selection for Mobile Devices. In Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '10). Association for Computing Machinery, New York, NY, USA, 181--190. https://doi.org/10.1145/1851600.1851631 Google Scholar
Digital Library
- R. Xiao, J. Schwarz, N. Throm, A. D. Wilson, and H. Benko. 2018. MRTouch: Adding Touch Input to Head-Mounted Mixed Reality. IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1653--1660. Google Scholar
Digital Library
- Zhican Yang, Chun Yu, Xin Yi, and Yuanchun Shi. 2019. Investigating Gesture Typing for Indirect Touch. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 3, Article Article 117 (Sept. 2019), 22 pages. https://doi.org/10.1145/3351275 Google Scholar
Digital Library
- Yui-Pan Yau, Lik Hang Lee, Zheng Li, Tristan Braud, Yi-Hsuan Ho, and Pan Hui. 2020. How Subtle Can It Get? A Trimodal Study of Ring-Sized Interfaces for One-Handed Drone Control. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 2, Article 63 (June 2020), 29 pages. https://doi.org/10.1145/3397319 Google Scholar
Digital Library
- Shengdong Zhao, Fanny Chevalier, Wei Tsang Ooi, Chee Yuan Lee, and Arpit Agarwal. 2012. AutoComPaste: Auto-Completing Text as an Alternative to Copy-Paste. In Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '12). Association for Computing Machinery, New York, NY, USA, 365--372. https://doi.org/10.1145/2254556.2254626 Google Scholar
Digital Library
- Mingyuan Zhong, Chun Yu, Qian Wang, Xuhai Xu, and Yuanchun Shi. 2018. ForceBoard: Subtle Text Entry Leveraging Pressure. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). Association for Computing Machinery, New York, NY, USA, Article Paper 528, 10 pages. https://doi.org/10.1145/3173574.3174102 Google Scholar
Digital Library
- Fengyuan Zhu and Tovi Grossman. 2020. BISHARE: Exploring Bidirectional Interactions Between Smartphones and Head-Mounted Augmented Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3313831.3376233 Google Scholar
Digital Library
Index Terms
Press-n-Paste: Copy-and-Paste Operations with Pressure-sensitive Caret Navigation for Miniaturized Surface in Mobile Augmented Reality
Recommendations
Using an Augmented Reality Cube-Like Interface and 3D Gesture-Based Interaction to Navigate and Manipulate Data
VINCI '18: Proceedings of the 11th International Symposium on Visual Information Communication and InteractionIn this paper we describe our work-in-progress to create an interface that enables users to browse and select data within an Augmented Reality environment, using a virtual cube object that can be interacted with through 3D gestural input. We present the ...
Experiencing 3D interactions in virtual reality and augmented reality
EUSAI '04: Proceedings of the 2nd European Union symposium on Ambient intelligenceWe demonstrate basic 2D and 3D interactions in both a Virtual Reality (VR) system, called the Personal Space Station, and an Augmented Reality (AR) system, called the Visual Interaction Platform. Since both platforms use identical (optical) tracking ...
Interaction methods for mobile augmented reality
CHINZ '12: Proceedings of the 13th International Conference of the NZ Chapter of the ACM's Special Interest Group on Human-Computer InteractionIn recent years mobile phone based Augmented Reality (AR) has become very popular. The current generation of smart phones has cameras, GPS and compass sensors, and significant CPU and GPU power, making them idea for mobile AR applications. Interaction ...






Comments