skip to main content
research-article

Spatial-geometric approach to physical mobile interaction based on accelerometer and IR sensory data fusion

Published:26 November 2010Publication History
Skip Abstract Section

Abstract

Interaction with the physical environment using mobile phones has become increasingly desirable and feasible. Nowadays mobile phones are being used to control different devices and access information/services related to those devices. To facilitate such interaction, devices are usually marked with RFID tags or visual markers, which are read by a mobile phone equipped with an integrated RFID reader or camera to fetch related information about those objects and initiate further actions. This article contributes in this domain of mobile physical interaction; however, using a spatial-geometric approach for interacting with indoor physical objects and artifacts instead of RFID based solutions. Using this approach, a mobile phone can point from a distance to an annotated object or a spatial subregion of that object for the purpose of interaction. The pointing direction and location is determined based on the fusion of IR camera and accelerometer data, where the IR cameras are used to calculate the 3D position of the mobile phone users and the accelerometer in the phone provides its tilting and orientation information. The annotation of objects and their subregions with which the mobile phone interacts is performed by specifying their geometric coordinates and associating related information or services with them. We perform experiment in a technology-augmented smart space and show the applicability and potential of the proposed approach.

References

  1. Ailisto, H., Pohjanheimo, L., Valkkynen, P., Strmmer, E., Tuomisto, T., and Korhonen, I. 2006. Bridging the physical and virtual worlds by local connectivity-based physical selection. Perso. Ubiq. Comput. 10, 6, 333--344. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Ballagas, R., Borchers, J., Rohs, M., and Sheridan, J. 2006. The smart phone: a ubiquitous input device. IEEE Pervasive Comput. 5, 1, 70--77. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Ballagas, R., Rohs, M., and Sheridan, J. G. 2005. Sweep and point and shoot: phonecam-based interactions for large public displays. In CHI'05 Extended Abstracts on Human Factors in Computing Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Broll, G., Rukzio, E., Paolucci, M., Wagner, M., Schmidt, A., and HuBmann, H. 2009. Perci: Pervasive service interaction with the internet of things. IEEE Internet Comput. 13, 6, 74--81. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Broll, G., Siorpaes, S., Rukzio, E., Paolucci, M., Hamard, J., Wagner, M., and Schmidt, A. 2007. Supporting mobile service usage through physical mobile interaction. In Proceedings of the 5th Annual IEEE International Conference on Pervasive Computing and Communications. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Egenhofer, M. 1999. Spatial information appliances: A next generation of geographic information systems. In Proceedings of the 1st Brazilian Workshop on GeoInformatics.Google ScholarGoogle Scholar
  7. Fahiem, M., Haq, S., and Saleemi, F. 2007. A review of 3d reconstruction techniques from 2d orthographic line drawings. Proceedings of the Conference on Geometric Modeling and Imaging. (GMAI'07), 60--66. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Faugeras, O. 1993. Three-Dimensional Computer Vision. MIT Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Froelelich, P., Simon, R., Baillie, L., Roberts, J., and Murray-Smith, R. 2007. Mobile spatial interaction. In CHI'07 Extended Abstracts on Human Factors in Computing Systems. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Froehlich, P., Baillie, L., and Simon, R. 2008. Realizing the vision of mobile spatial interaction. Interactions 15, 1. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Gellersen, H., Fischer, C., Guinard, D., Gostner, R., Kortuem, G., Kray, C., Rukzio, E., and Streng, S. 2008. Supporting device discovery and spontaneous interaction with spatial references. Perso. Ubiq. Comput. 13, 4. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Gujar, U. G. and Nagendra, I. V. 1989. Construction of 3d solid objects from orthographic views. Comput. Graph. 13, 4, 505--521.Google ScholarGoogle ScholarCross RefCross Ref
  13. Hinske, S. 2007. Human-computer interaction. In interaction platforms and techniques. Lecture Notes in Computer Science. Springer, Berlin 306--310. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Holleis, P., Rukzio, E., Otto, F., and Schmidt, A. 2007. Privacy and curiosity in mobile interactions with public displays. In Proceedings of the Workshop on Mobile Spatial Interaction (CHI'07).Google ScholarGoogle Scholar
  15. Hosoya, E., Sato, H., Harada, M. K. I., Nojima, H., and Onozawa, A. 2004. Computer vision in human-computer interaction. Lecture Notes in Computer Science. Springer, Berlin, 72--82.Google ScholarGoogle Scholar
  16. Likert, R. 1932. A technique for the measurement of attitudes. Arch. Psych. 140, 1--55.Google ScholarGoogle Scholar
  17. Madhavapeddy, A., Scott, D., Sharp, R., and Upton, E. 2004. Using camera phones to enhance human-computer interaction. In Proceedings of the 6th International Conference on Ubiquitous Computing (UbiComp'04).Google ScholarGoogle Scholar
  18. Mantyjarvi, J., Paterno, F., Salvador, Z., and Santoro, C. 2006. Scan and tilt: towards natural interaction for mobile museum guides. In Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 191--194. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Marr, D. and Poggio, T. 1979. A Computational Theory of Human Stereo Vision. Vol. 204. The Royal Society, 301--328.Google ScholarGoogle Scholar
  20. Nakazato, Y., Kanbara, M., and Yokoya, N. 2008. Localization system for large indoor environments using invisible markers. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Pering, T., Ballagas, R., and Want, R. 2005. Spontaneous marriages of mobile devices and interactive spaces. Comm. ACM 48, 9, 53--59. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Rahman, A. S. M. M., Hossain, M. A., Parra, J., and El Saddik, A. 2009. Motion-path based gesture interaction with smart home services. In Proceedings of the 17th ACM International Conference on Multimedia. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Riekki, J., Salminen, T., and Alakarppa, I. 2006. Requesting pervasive services by touching rfid tags. Pervasive Comput. IEEE 5, 2, 40--46. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Rohs, M. 2004. Real-world interaction with camera-phones. In Proceedings of the 2nd International Symposium on Ubiquitous Computing Systems. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Rohs, M. 2005. Visual code widgets for marker-based interaction. In Proceedings of the 25th IEEE International Conference on Distributed Computing Systems Workshops. 506--513. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Rukzio, E., Leichtenstern, K., Callaghan, V., Holleis, P., Schmidt, A., and Chin, J. 2006. An Experimental Comparison of Physical Mobile Interaction Techniques: Touching, Pointing and Scanning. Springer Berlin, 4206/2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Ruzkio, E., Broll, G., Leichtenstern, K., and Schmidt, A. 2007. Mobile interaction with the real world: An evaluation and comparison of physical mobile interaction techniques. In Proceedings of the European Conference on Ambient Intelligence. Springer, Berlin. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Sanchez, I., Riekki, J., and Pyykknen, M. 2008. Touch and control: Interacting with services by touching rfid tags. In Proceedings of the International Workshop on RFID Tedondosy.Google ScholarGoogle Scholar
  29. Schoo, P. and Paolucci, M. 2009. Do you talk to each poster? security and privacy for interactions with web service by means of contact free tag readings. In 1st International Workshop on Near Field Communication. IEEE Computer Society, 81--86. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Siltanen, S., Woodward, C., Valli, S., Honkamaa, P., and Rauber, A. 2008. User Interaction for Mobile Devices. Springer, 33.Google ScholarGoogle Scholar
  31. Simon, R., Frohlich, P., and Grechenig, T. 2008. Geopointing: evaluating the performance of orientation-aware location-based interaction under real-world conditions. J. Locat. Based Serv. 2, 1, 24--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Strachan, S. and Murray-Smith, R. 2009. Bearing-based selection in mobile spatial interaction. Perso. Ubiq. Comput. 13, 4, 265--280. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Tomitsch, M., Schl, R., Grechenig, T., Wimmer, C., and Thomas, K. 2008. Accessible real-world tagging through audio-tactile location markers. In Proceedings of the 5th Nordic conference on Human-Computer Interaction. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Valkkynen, P. and Tuomisto, T. 2005. Physical browsing research. In Pervasive Mobile Interaction Devices. Springer.Google ScholarGoogle Scholar
  35. Want, R., Fishkin, K. P., Gujar, A., and Harrison, B. L. 1999. Bridging physical and virtual worlds with electronic tags. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 370--377. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Spatial-geometric approach to physical mobile interaction based on accelerometer and IR sensory data fusion

        Recommendations

        Reviews

        Gerald Friedland

        Wouldn't it be cool if your mobile phone could interact with the real world as a kind of "universal joystick"__?__ This is the basic premise of this paper. The paper presents a system for the fusion of data from accelerometers and two different infrared sensors that allows us to determine, with high accuracy, the 3D position of a user's mobile device in an indoor environment. The goal is to enable smartphones to be universal pointers, which would allow users to control electronic devices by pointing their smartphones at them. The paper presents a long range of related work, and discusses the technical approach, including pseudocode. The authors evaluate the discussed interaction paradigms with a usability study, measuring and extensively analyzing the accuracy of the localization approach. In summary, this is a well-written, deep technical paper that presents two novelties at once: first, the fusion algorithm is novel, and combines two media that are rarely integrated; second, the interaction paradigm has not been previously explored, and seems, at the same time, very natural. On the negative side, the authors use some terms that they do not explain, and that are not commonly familiar to a computer scientist. Nevertheless, this paper is definitely recommended reading for any multimedia researcher. Online Computing Reviews Service

        Access critical reviews of Computing literature here

        Become a reviewer for Computing Reviews.

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!