skip to main content
research-article

Touchable 3D video system

Published:06 November 2009Publication History
Skip Abstract Section

Abstract

Multimedia technologies are reaching the limits of providing audio-visual media that viewers consume passively. An important factor, which will ultimately enhance the user's experience in terms of impressiveness and immersion, is interaction. Among daily life interactions, haptic interaction plays a prominent role in enhancing the quality of experience of users, and in promoting physical and emotional development. Therefore, a critical step in multimedia research is expected to bring the sense of touch, or haptics, into multimedia systems and applications. This article proposes a touchable 3D video system where viewers can actively touch a video scene through a force-feedback device, and presents the underlying technologies in three functional components: (1) contents generation, (2) contents transmission, and (3) viewing and interaction. First of all, we introduce a depth image-based haptic representation (DIBHR) method that adds haptic and heightmap images, in addition to the traditional depth image-based representation (DIBR), to encode the haptic surface properties of the video media. In this representation, the haptic image contains the stiffness, static friction, and dynamic friction, whereas the heightmap image contains roughness of the video contents. Based on this representation method, we discuss how to generate synthetic and natural (real) video media through a 3D modeling tool and a depth camera, respectively. Next, we introduce a transmission mechanism based on the MPEG-4 framework where new MPEG-4 BIFS nodes are designed to describe the haptic scene. Finally, a haptic rendering algorithm to compute the interaction force between the scene and the viewer is described. As a result, the performance of the haptic rendering algorithm is evaluated in terms of computational time and smooth contact force. It operates marginally within a 1 kHz update rate that is required to provide stable interaction force and provide smoother contact force with the depth image that has high frequency geometrical noise using a median filter.

References

  1. Alatan, A. A., Yemez, Y., Güdükbay, U., Zabulis, X., Müller, K., Erdem, Ç. E., Weigel, C., and Smolic, A. 2007. Scene representation technologies for 3DTV—A survey. IEEE Trans. Circuits Syst. Video Tech. 17, 11, 1587--1605. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Avila, R. S. and Sobierajski, L. M. 1996. A haptic interaction method for volume visualization. In Proceedings of the IEEE Visualization Conference. 197--204. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Bukowska, M. 2001. Winky Dink half a century later. Interaction with broadcast content: Concept development based on an interactive storytelling application for children. Tech. rep., Media Interaction Group, Philips Research Laboratories, Eindhoven, The Netherland.Google ScholarGoogle Scholar
  4. Cha, J., Eid, M., and El Saddik, A. 2008. DIBHR: Depth image-based haptic rendering. In Proceedings of the EuroHaptics Conference. 640--650. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Cha, J., Kim, S., Ho, Y., and Ryu, J. 2006. 3D video player system with haptic interaction based on depth image-based representation. IEEE Trans. Consumer Elect. 52, 2, 477--484. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Cha, J., Ryu, J., Kim, S., Eom, S., and Ahn, B. 2004. Haptic interaction in realistic multimedia broadcasting. In Proceedings of the 5th Pacific Rim Conference on Multimedia, Advances in Multimedia Information Processing. 482--490. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Cha, J., Seo, Y., Kim, Y., and Ryu, J. 2007. An authoring/editing framework for haptic broadcasting: Passive haptic interactions using MPEG-4 BIFS. In Proceedings of the Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 274--279. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Chorianopoulos, K. and Lekakos, G. 2007. Learn and play with interactive TV. ACM Comput. Entertain. 5, 2. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Conti, F., Barbagli, F., Morris, D., and Sewell, C. 2005. CHAI: An open-source library for the rapid development of haptic scenes. In Proceedings of the Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.Google ScholarGoogle Scholar
  10. Eid, M., Andrews, S., Alamri, A., and El Saddik, A. 2008. HAMLAT: A HAML-based authoring tool for haptic application development. In Proceedings of the EuroHaptics Conference. Vol. 5024. 857--866. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Eid, M., Orozco, M., and El Saddik, A. 2007. A guided tour in haptic audio visual environments and applications. Inter. J. Advanced Media. Comm. 1, 3, 265--297. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. El Saddik, A. 2007. The potential of haptics technologies. IEEE Instrum. Measure. Mag. 10, 1, 10--17.Google ScholarGoogle ScholarCross RefCross Ref
  13. Fehn, C., Schüür, K., Kauff, P., and Smolic, A. 2003. Meta-data requirements for EE4 in MPEG 3DAV. Pattaya, Thailand. ISO/IEC JTC1/SC29/WG11, document M9559.Google ScholarGoogle Scholar
  14. Gao, Z. and Gibson, I. 2005. Haptic B-spline surface sculpting with a shaped tool of implicit surface. Comput. Aid. Des. Appl. 2, 1-4, 263--272.Google ScholarGoogle Scholar
  15. Gaw, D., Morris, D., and Salisbury, K. 2006. Haptically annotated movies: Reaching out and touching the silver screen. In Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 287--288. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Gibson, J. J. 1962. Observations on active touch. Psych. Rev. 69, 477--491.Google ScholarGoogle ScholarCross RefCross Ref
  17. Hale, K. S. and Stanney, K. M. 2004. Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations. IEEE Comput. Graph. Appl. 24, 2, 33--39. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Ho, C.-H., Basdogan, C., and Srinivasan, M. A. 1999. Efficient point-based rendering techniques for haptic display of virtual objects. Presence: Teleop. Virtual Environ. 8, 5, 477--491. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Ignatenko, A. and Konushin, A. 2003. A framework for depth image-based modeling and rendering. In Proceedings of Graphicon'03. 169--172.Google ScholarGoogle Scholar
  20. Ikits, M., Brederson, J. D., Hansen, C. D., and Johnson, C. R. 2003. A constraint-based technique for haptic volume exploration. In Proceedings of the IEEE Visualization Conference. 263--269. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Kauff, P., Fehn, C., Cooke, E., and Schreer, O. 2001. Advanced incomplete 3D representation of video objects using trilinear warping for novel view synthesis. In Proceedings of the Picture Coding Symposium. 429--432.Google ScholarGoogle Scholar
  22. Kim, L., Sukhatme, G. S., and Desbrun, M. 2004. A haptic-rendering technique based on hybrid surface representation. IEEE Trans. Comput. Graph. Appl. 24, 2, 66--75. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Kim, S.-M., Cha, J., Ryu, J., and Lee, K. H. 2006. Depth video enhancement for haptic interaction using a smooth surface reconstruction. IEICE Trans. Inform. Syst. E89-D, 1, 37--44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Lawrence, D. A., Lee, C. D., Pao, L. Y., and Novoselov, R. Y. 2000. Shock and vortex visualization using a combined visual/haptic interface. In Proceedings of the IEEE Visualization Conference. 131--137. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Lee, M. H. and Nicholls, H. R. 1999. Tactile sensing for mechatronics--A state of the art survey. Mechatronics 9, 1, 1--31.Google ScholarGoogle ScholarCross RefCross Ref
  26. Levkovich-Maslyuk, L., Ignatenko, A., Zhirkov, A., Konushin, A., Park, I. K., Han, M., and Bayakovski, Y. 2004. Depth image-based representation and compression for static and animated 3-D objects. IEEE Trans. Circ. Syst. Video Tech. 14, 7, 1032--1045. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Lucchese, L. and Mitra, S. K. 2001. Color image segmentation: A state-of-the-art survey. In Proceedings of the Indian National Science Academy (INSA-A). Vol. 67, 207--221.Google ScholarGoogle Scholar
  28. Magnenat-Thalmann, N. and Bonanni, U. 2006. Haptics in virtual reality and multimedia. IEEE Multimedia 13, 3, 6--11. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Matusik, W. and Pfister, H. 2004. 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graph. 23, 3 (Aug.), 814--824. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. McNeely, W. A., Puterbaugh, K. D., and Troy, J. J. 1999. Six degree-of-freedom haptic rendering using voxel sampling. In Proceedings of ACM SIGGRAPH. 401--408. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Melder, N. and Harwin, W. S. 2004. Extending the friction cone algorithm for arbitrary polygon based haptic objects. In Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 234--241. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Michelitsch, G., Ruf, A., van Veen, H., and van Erp, J. 2002. Multi-finger haptic interaction within the MIAMM project. In Proceedings of the EuroHaptics Conference.Google ScholarGoogle Scholar
  33. O'Modhrain, S. and Oakley, I. 2003. Touch TV: Adding feeling to broadcast media. In Proceedings of the European Conference on Interactive Television: From Viewers to Actors. 41--47.Google ScholarGoogle Scholar
  34. Reachin AB. 2003. Reachin API 3.2 - Programmer's Guide. Reachin AB.Google ScholarGoogle Scholar
  35. Reiner, M. 2004. The role of haptics in immersive telecommunication environments. IEEE Trans. Circ. Syst. Video Tech. 14, 3, 392--401. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Riva, G., Davide, F., and IJsselsteijn, W. A. 2003. Being There: Concepts, Effects and Measurements of User Presence in Synthetic Environments. I.O.S., Press, Amsterdam, The Netherlands, Chapter 2.Google ScholarGoogle Scholar
  37. Ruspini, D. C., Kolarov, K., and Khatib, O. 1997. The haptic display of complex graphical environments. In Proceedings of ACM SIGGRAPH. ACM, New York, NY, 345--352. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Salisbury, J. K. and Tarr, C. 1997. Haptic rendering of surfaces defined by implicit functions. In Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 61--68.Google ScholarGoogle Scholar
  39. Sallnas, E.-L., Rassmus-Grohn, K., and Sjostrom, C. 2000. Supporting presence in collaborative environments by haptic force feedback. ACM Trans. Comput.-Hum. Inter. 7, 4, 461--476. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. SensAble Technologies, Inc. 2005. OpenHaptics Toolkit Version 2.0—Programmer's Guide. SensAble Technologies, Inc.Google ScholarGoogle Scholar
  41. SenseGraphics AB. 2006. H3D API Manual for Version 1.5. SenseGraphics AB.Google ScholarGoogle Scholar
  42. Smolic, A. and Kauff, P. 2005. Interactive 3-D video representation and coding technologies. Proc. IEEE 93, 1, 98--110.Google ScholarGoogle ScholarCross RefCross Ref
  43. Thompson, T. V. and Cohen, E. 1999. Direct haptic rendering of complex trimmed NURBS models. In Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 89--96. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Thompson, T. V., Nelson, D. D., Cohen, E., and Hollerbach, J. 1997. Maneuverable NURBS models within a haptic virtual environment. In Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 37--44.Google ScholarGoogle Scholar
  45. Walker, S. P. and Salisbury, J. K. 2003. Large haptic topographic maps: MarsView and the proxy graph algorithm. In Proceedings of ACM SIGGRAPH. ACM, New York, NY, 83--92. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Witmer, B. G. and Singer, M. J. 1998. Measuring presence in virtual environments: A presence questionnaire. Presence: Teleop. Virtual Environ. 7, 3, 225--240. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Yamaguchi, T., Akabane, A., Murayama, J., and Sato, M. 2006. Automatic generation of haptic effect into published 2D graphics. In Proceedings of the EuroHaptics Conference.Google ScholarGoogle Scholar
  48. Zilles, C. B. and Salisbury, J. K. 1995. A constraint based god-object method for haptic display. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. 146--151. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Touchable 3D video system

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            • Published in

              cover image ACM Transactions on Multimedia Computing, Communications, and Applications
              ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 5, Issue 4
              October 2009
              103 pages
              ISSN:1551-6857
              EISSN:1551-6865
              DOI:10.1145/1596990
              Issue’s Table of Contents

              Copyright © 2009 ACM

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              • Published: 6 November 2009
              • Accepted: 1 February 2009
              • Revised: 1 December 2008
              • Received: 1 August 2008
              Published in tomm Volume 5, Issue 4

              Permissions

              Request permissions about this article.

              Request Permissions

              Check for updates

              Qualifiers

              • research-article
              • Research
              • Refereed

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!