Touch & activate: adding interactivity to existing objects using active acoustic sensing

ABSTRACT

In this paper, we present a novel acoustic touch sensing technique called Touch & Activate. It recognizes a rich context of touches including grasp on existing objects by attaching only a vibration speaker and a piezo-electric microphone paired as a sensor. It provides easy hardware configuration for prototyping interactive objects that have touch input capability. We conducted a controlled experiment to measure the accuracy and trade-off between the accuracy and number of training rounds for our technique. From its results, per-user recognition accuracies with five touch gestures for a plastic toy as a simple example and six hand postures for the posture recognition as a complex example were 99.6% and 86.3%, respectively. Walk up user recognition accuracies for the two applications were 97.8% and 71.2%, respectively. Since the results of our experiment showed a promising accuracy for the recognition of touch gestures and hand postures, Touch & Activate should be feasible for prototype interactive objects that have touch input capability.

References

1. Arduino - HomePage. http://www.arduino.cc/, Accessed: June 2013.
2. mbed - Rapid Prototyping for Microcontrollers. http://mbed.org/, Accessed: June 2013.
3. Amento, B., Hill, W., and Terveen, L. The sound of one hand: a wrist-mounted bio-acoustic fingertip gesture interface. In CHI EA '02, 724--725.
4. Azenkot, S., and Zhai, S. Touch behavior with different postures on soft smartphone keyboards. In MobileHCI '12, 251--260.
5. Brenner, M.C., and Fitzgibbon, J.J. Surface acoustic wave touch panel system, Feb.17 1987. US Patent 4,644,100.
6. Chang, C.-C., Lin, C.-J. LIBSVM: a library for support vector machines., http://www.csie.ntu.edu.tw/cjlin/livsvm
7. Cheng, L.-P., Hsiao, F.-I., Liu, Y.-T., and Chen, M.Y. iRotate grasp: automatic screen rotation based on grasp of mobile devices. In UIST '12 Adjunct, 15--16.
8. Christopher, D. Frequency Range of Human Hearing. In The Physics Factbook. 2003.
9. Collins, T. Active acoustic touch interface. Electronics Letters 45, 20 (24 2009), 1055 --1056.
10. Cralg, L. Right--rapid--rough. Ask Magazine, 13 (2003), 12--15.
11. Ewins, D.J. Modal testing: theory, practice and application, vol.2. Research studies press Baldock, 2000.
12. Fourney, A., and Terry, M. PICL: portable in-circuit learner. In UIST '12, 569--578.
13. Goel, M., Wobbrock, J., and Patel, S. GripSense: using built-in sensors to detect hand posture and pressure on commodity mobile phones. In UIST '12, 545--554.
14. Greenberg, S., and Fitchett, C. Phidgets: easy development of physical interfaces through physical widgets. In UIST '01 , 209--218.
15. Gupta, A., Fox, D., Curless, B., and Cohen, M. DuploTrack: a real-time system for authoring and guiding duplo block assembly. In UIST '12, 389--402.
16. Gupta, S., Morris, D., Patel, S., and Tan, D. SoundWave: using the Doppler effect to sense gestures. In CHI '12, 1911--1914.
17. Han, J.Y. Low-cost multi-touch sensing through frustrated total internal reflection. In UIST '05, 115--118.
18. Harrison, C., Benko, H., and Wilson, A.D. OmniTouch: wearable multitouch interaction everywhere. In UIST '11, 441--450.
19. Harrison, C., and Hudson, S.E. Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces. In UIST '08, 205--208.
20. Harrison, C., Sato, M., and Poupyrev, I. Capacitive fingerprinting: exploring user differentiation by sensing electrical properties of the human body. In UIST '12, 537--544.
21. Harrison, C., Schwarz, J., and Hudson, S.E. TapSense: enhancing finger interaction on touch surfaces. In UIST '11, 627--636.
22. Harrison, C., Tan, D., and Morris, D. Skinput: appropriating the body as an input surface. In CHI '10, 453--462.
23. Harrison, C., Xiao, R., and Hudson, S. Acoustic barcodes: passive, durable and inexpensive notched identification tags. In UIST '12, 563--568.
24. Hartmann, B., Abdulla, L., Mittal, M., and Klemmer, S.R. Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition. In CHI '07, 145--154.
25. Hartmann, B., Klemmer, S.R., Bernstein, M., Abdulla, L., Burr, B., Robinson-Mosher, A., and Gee, J. Reflective physical prototyping through integrated design, test, and analysis. In UIST '06, 299--308.
26. Hoggan, E., Stewart, C., Haverinen, L., Jacucci, G., and Lantz, V. Pressages: augmenting phone calls with non-verbal messages. In UIST '12, 555--562.
27. Holman, D., and Vertegaal, R. TactileTape: low-cost touch sensing on curved surfaces. In UIST '11 Adjunct, 17--18.
28. Hudson, S.E., and Mankoff, J. Rapid construction of functioning physical interfaces from cardboard, thumbtacks, tin foil and masking tape. In UIST '06, 289--298.
29. Hwang, S., Bianchi, A., and Wohn, K. MicPen: pressure-sensitive pen interaction using microphone with standard touchscreen. In CHI EA '12, 1847--1852.
30. Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., and Fitzgibbon, A. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In UIST '11, 559--568.
31. Karlson, A.K., and Bederson, B.B. Understanding single-handed mobile device interaction. Tech. rep., Dep. of Computer Science, Univ. of Maryland, 2006.
32. Kim, K.-E., Chang, W., Cho, S.-J., Shim, J., Lee, H., Park, J., Lee, Y., and Kim, S. Hand grip pattern recognition for mobile user interfaces. In IAAI '06, 1789--1794.
33. Liu, S., and Guimbretiére, F. FlexAura: a flexible near-surface range sensor. In UIST '12, 327--330.
34. Lopes, P., Jota, R., and Jorge, J.A. Augmenting touch interaction through acoustic sensing. In ITS '11, 53--56.
35. Miyaki, T., and Rekimoto, J. Graspzoom: zooming and scrolling control model for single-handed mobile interaction. In MobileHCI '09, 11.
36. Murray-Smith, R., Williamson, J., Hughes, S., and Quaade, T. Stane: synthesized surfaces for tactile input. In CHI '08, 1299--1302.
37. Paradiso, J.A., Leo, C.K., Checka, N., and Hsiao, K. Passive acoustic knock tracking for interactive windows. In CHI EA'02, 732--733.
38. Parhi, P., Karlson, A.K., and Bederson, B.B. Target size study for one-handed thumb use on small touchscreen devices. In MobileHCI '06, 203--210.
39. Pham, D., Ji, Z., Yang, M., Wang, Z., and Al-Kutubi, M. A novel human-computer interface based on passive acoustic localisation. In HCII '07, 901--909.
40. Rekimoto, J., and Sciammarella, E. ToolStone: effective use of the physical manipulation vocabularies of input devices. In UIST '00, 109--117.
41. Sato, M., Poupyrev, I., and Harrison, C. Touché: enhancing touch interaction on humans, screens, liquids, and everyday objects. In CHI '12, 483--492.
42. Savage, V., Zhang, X., and Hartmann, B. Midas: fabricating custom capacitive touch sensors to prototype interactive objects. In UIST '12, 579--588.
43. Schwarz, B.J., and Richardson, M.H. Experimental modal analysis. CSI Reliability week (1999).
44. Song, H., Benko, H., Guimbretiere, F., Izadi, S., Cao, X., and Hinckley, K. Grips and gestures on a multi-touch pen. In CHI'11, 1323--1332.
45. Stewart, J., and Westerfield, E. A theory of active sonar detection. Proceedings of the IRE 47, 5 (1959), 872--881.
46. Takemura, K., Ito, A., Takamatsu, J., and Ogasawara, T. Active bone-conducted sound sensing for wearable interfaces. In UIST'11 Adjunct, 53--54.
47. Taylor, B.T., and Bove, Jr., V.M. Graspables: grasp-recognition as a user interface. In CHI '09, 917--926.
48. Villar, N., Scott, J., Hodges, S., Hammil, K., and Miller, C. . NET Gadgeteer: a platform for custom devices. Pervasive Computing (2012), 216--233.
49. Vogel, D., and Casiez, G. Conté: multimodal input inspired by an artist's crayon. In UIST '11, 357--366.
50. Wilson, G., Stewart, C., and Brewster, S.A. Pressure-based menu selection for mobile devices. In MobileHCI '10, 181--190.
51. Wimmer, R. FlyEye: grasp-sensitive surfaces using optical fiber. In TEI '10, 245--248.
52. Wimmer, R., and Baudisch, P. Modular and deformable touch-sensitive surfaces based on time domain reflectometry. In UIST '11, 517--526.
53. Wimmer, R., and Boring, S. HandSense: discriminating different ways of grasping and holding a tangible user interface. In TEI '09, 359--362.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Touch screens

    • Interaction paradigms

      • Graphical user interfaces