Abstract
Multimedia technologies are reaching the limits of providing audio-visual media that viewers consume passively. An important factor, which will ultimately enhance the user's experience in terms of impressiveness and immersion, is interaction. Among daily life interactions, haptic interaction plays a prominent role in enhancing the quality of experience of users, and in promoting physical and emotional development. Therefore, a critical step in multimedia research is expected to bring the sense of touch, or haptics, into multimedia systems and applications. This article proposes a touchable 3D video system where viewers can actively touch a video scene through a force-feedback device, and presents the underlying technologies in three functional components: (1) contents generation, (2) contents transmission, and (3) viewing and interaction. First of all, we introduce a depth image-based haptic representation (DIBHR) method that adds haptic and heightmap images, in addition to the traditional depth image-based representation (DIBR), to encode the haptic surface properties of the video media. In this representation, the haptic image contains the stiffness, static friction, and dynamic friction, whereas the heightmap image contains roughness of the video contents. Based on this representation method, we discuss how to generate synthetic and natural (real) video media through a 3D modeling tool and a depth camera, respectively. Next, we introduce a transmission mechanism based on the MPEG-4 framework where new MPEG-4 BIFS nodes are designed to describe the haptic scene. Finally, a haptic rendering algorithm to compute the interaction force between the scene and the viewer is described. As a result, the performance of the haptic rendering algorithm is evaluated in terms of computational time and smooth contact force. It operates marginally within a 1 kHz update rate that is required to provide stable interaction force and provide smoother contact force with the depth image that has high frequency geometrical noise using a median filter.
- Alatan, A. A., Yemez, Y., Güdükbay, U., Zabulis, X., Müller, K., Erdem, Ç. E., Weigel, C., and Smolic, A. 2007. Scene representation technologies for 3DTV—A survey. IEEE Trans. Circuits Syst. Video Tech. 17, 11, 1587--1605. Google Scholar
Digital Library
- Avila, R. S. and Sobierajski, L. M. 1996. A haptic interaction method for volume visualization. In Proceedings of the IEEE Visualization Conference. 197--204. Google Scholar
Digital Library
- Bukowska, M. 2001. Winky Dink half a century later. Interaction with broadcast content: Concept development based on an interactive storytelling application for children. Tech. rep., Media Interaction Group, Philips Research Laboratories, Eindhoven, The Netherland.Google Scholar
- Cha, J., Eid, M., and El Saddik, A. 2008. DIBHR: Depth image-based haptic rendering. In Proceedings of the EuroHaptics Conference. 640--650. Google Scholar
Digital Library
- Cha, J., Kim, S., Ho, Y., and Ryu, J. 2006. 3D video player system with haptic interaction based on depth image-based representation. IEEE Trans. Consumer Elect. 52, 2, 477--484. Google Scholar
Digital Library
- Cha, J., Ryu, J., Kim, S., Eom, S., and Ahn, B. 2004. Haptic interaction in realistic multimedia broadcasting. In Proceedings of the 5th Pacific Rim Conference on Multimedia, Advances in Multimedia Information Processing. 482--490. Google Scholar
Digital Library
- Cha, J., Seo, Y., Kim, Y., and Ryu, J. 2007. An authoring/editing framework for haptic broadcasting: Passive haptic interactions using MPEG-4 BIFS. In Proceedings of the Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 274--279. Google Scholar
Digital Library
- Chorianopoulos, K. and Lekakos, G. 2007. Learn and play with interactive TV. ACM Comput. Entertain. 5, 2. Google Scholar
Digital Library
- Conti, F., Barbagli, F., Morris, D., and Sewell, C. 2005. CHAI: An open-source library for the rapid development of haptic scenes. In Proceedings of the Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.Google Scholar
- Eid, M., Andrews, S., Alamri, A., and El Saddik, A. 2008. HAMLAT: A HAML-based authoring tool for haptic application development. In Proceedings of the EuroHaptics Conference. Vol. 5024. 857--866. Google Scholar
Digital Library
- Eid, M., Orozco, M., and El Saddik, A. 2007. A guided tour in haptic audio visual environments and applications. Inter. J. Advanced Media. Comm. 1, 3, 265--297. Google Scholar
Digital Library
- El Saddik, A. 2007. The potential of haptics technologies. IEEE Instrum. Measure. Mag. 10, 1, 10--17.Google Scholar
Cross Ref
- Fehn, C., Schüür, K., Kauff, P., and Smolic, A. 2003. Meta-data requirements for EE4 in MPEG 3DAV. Pattaya, Thailand. ISO/IEC JTC1/SC29/WG11, document M9559.Google Scholar
- Gao, Z. and Gibson, I. 2005. Haptic B-spline surface sculpting with a shaped tool of implicit surface. Comput. Aid. Des. Appl. 2, 1-4, 263--272.Google Scholar
- Gaw, D., Morris, D., and Salisbury, K. 2006. Haptically annotated movies: Reaching out and touching the silver screen. In Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 287--288. Google Scholar
Digital Library
- Gibson, J. J. 1962. Observations on active touch. Psych. Rev. 69, 477--491.Google Scholar
Cross Ref
- Hale, K. S. and Stanney, K. M. 2004. Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations. IEEE Comput. Graph. Appl. 24, 2, 33--39. Google Scholar
Digital Library
- Ho, C.-H., Basdogan, C., and Srinivasan, M. A. 1999. Efficient point-based rendering techniques for haptic display of virtual objects. Presence: Teleop. Virtual Environ. 8, 5, 477--491. Google Scholar
Digital Library
- Ignatenko, A. and Konushin, A. 2003. A framework for depth image-based modeling and rendering. In Proceedings of Graphicon'03. 169--172.Google Scholar
- Ikits, M., Brederson, J. D., Hansen, C. D., and Johnson, C. R. 2003. A constraint-based technique for haptic volume exploration. In Proceedings of the IEEE Visualization Conference. 263--269. Google Scholar
Digital Library
- Kauff, P., Fehn, C., Cooke, E., and Schreer, O. 2001. Advanced incomplete 3D representation of video objects using trilinear warping for novel view synthesis. In Proceedings of the Picture Coding Symposium. 429--432.Google Scholar
- Kim, L., Sukhatme, G. S., and Desbrun, M. 2004. A haptic-rendering technique based on hybrid surface representation. IEEE Trans. Comput. Graph. Appl. 24, 2, 66--75. Google Scholar
Digital Library
- Kim, S.-M., Cha, J., Ryu, J., and Lee, K. H. 2006. Depth video enhancement for haptic interaction using a smooth surface reconstruction. IEICE Trans. Inform. Syst. E89-D, 1, 37--44. Google Scholar
Digital Library
- Lawrence, D. A., Lee, C. D., Pao, L. Y., and Novoselov, R. Y. 2000. Shock and vortex visualization using a combined visual/haptic interface. In Proceedings of the IEEE Visualization Conference. 131--137. Google Scholar
Digital Library
- Lee, M. H. and Nicholls, H. R. 1999. Tactile sensing for mechatronics--A state of the art survey. Mechatronics 9, 1, 1--31.Google Scholar
Cross Ref
- Levkovich-Maslyuk, L., Ignatenko, A., Zhirkov, A., Konushin, A., Park, I. K., Han, M., and Bayakovski, Y. 2004. Depth image-based representation and compression for static and animated 3-D objects. IEEE Trans. Circ. Syst. Video Tech. 14, 7, 1032--1045. Google Scholar
Digital Library
- Lucchese, L. and Mitra, S. K. 2001. Color image segmentation: A state-of-the-art survey. In Proceedings of the Indian National Science Academy (INSA-A). Vol. 67, 207--221.Google Scholar
- Magnenat-Thalmann, N. and Bonanni, U. 2006. Haptics in virtual reality and multimedia. IEEE Multimedia 13, 3, 6--11. Google Scholar
Digital Library
- Matusik, W. and Pfister, H. 2004. 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graph. 23, 3 (Aug.), 814--824. Google Scholar
Digital Library
- McNeely, W. A., Puterbaugh, K. D., and Troy, J. J. 1999. Six degree-of-freedom haptic rendering using voxel sampling. In Proceedings of ACM SIGGRAPH. 401--408. Google Scholar
Digital Library
- Melder, N. and Harwin, W. S. 2004. Extending the friction cone algorithm for arbitrary polygon based haptic objects. In Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 234--241. Google Scholar
Digital Library
- Michelitsch, G., Ruf, A., van Veen, H., and van Erp, J. 2002. Multi-finger haptic interaction within the MIAMM project. In Proceedings of the EuroHaptics Conference.Google Scholar
- O'Modhrain, S. and Oakley, I. 2003. Touch TV: Adding feeling to broadcast media. In Proceedings of the European Conference on Interactive Television: From Viewers to Actors. 41--47.Google Scholar
- Reachin AB. 2003. Reachin API 3.2 - Programmer's Guide. Reachin AB.Google Scholar
- Reiner, M. 2004. The role of haptics in immersive telecommunication environments. IEEE Trans. Circ. Syst. Video Tech. 14, 3, 392--401. Google Scholar
Digital Library
- Riva, G., Davide, F., and IJsselsteijn, W. A. 2003. Being There: Concepts, Effects and Measurements of User Presence in Synthetic Environments. I.O.S., Press, Amsterdam, The Netherlands, Chapter 2.Google Scholar
- Ruspini, D. C., Kolarov, K., and Khatib, O. 1997. The haptic display of complex graphical environments. In Proceedings of ACM SIGGRAPH. ACM, New York, NY, 345--352. Google Scholar
Digital Library
- Salisbury, J. K. and Tarr, C. 1997. Haptic rendering of surfaces defined by implicit functions. In Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 61--68.Google Scholar
- Sallnas, E.-L., Rassmus-Grohn, K., and Sjostrom, C. 2000. Supporting presence in collaborative environments by haptic force feedback. ACM Trans. Comput.-Hum. Inter. 7, 4, 461--476. Google Scholar
Digital Library
- SensAble Technologies, Inc. 2005. OpenHaptics Toolkit Version 2.0—Programmer's Guide. SensAble Technologies, Inc.Google Scholar
- SenseGraphics AB. 2006. H3D API Manual for Version 1.5. SenseGraphics AB.Google Scholar
- Smolic, A. and Kauff, P. 2005. Interactive 3-D video representation and coding technologies. Proc. IEEE 93, 1, 98--110.Google Scholar
Cross Ref
- Thompson, T. V. and Cohen, E. 1999. Direct haptic rendering of complex trimmed NURBS models. In Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 89--96. Google Scholar
Digital Library
- Thompson, T. V., Nelson, D. D., Cohen, E., and Hollerbach, J. 1997. Maneuverable NURBS models within a haptic virtual environment. In Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 37--44.Google Scholar
- Walker, S. P. and Salisbury, J. K. 2003. Large haptic topographic maps: MarsView and the proxy graph algorithm. In Proceedings of ACM SIGGRAPH. ACM, New York, NY, 83--92. Google Scholar
Digital Library
- Witmer, B. G. and Singer, M. J. 1998. Measuring presence in virtual environments: A presence questionnaire. Presence: Teleop. Virtual Environ. 7, 3, 225--240. Google Scholar
Digital Library
- Yamaguchi, T., Akabane, A., Murayama, J., and Sato, M. 2006. Automatic generation of haptic effect into published 2D graphics. In Proceedings of the EuroHaptics Conference.Google Scholar
- Zilles, C. B. and Salisbury, J. K. 1995. A constraint based god-object method for haptic display. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. 146--151. Google Scholar
Digital Library
Index Terms
Touchable 3D video system
Recommendations
A constraint-based god-object method for haptic display
IROS '95: Proceedings of the International Conference on Intelligent Robots and Systems-Volume 3 - Volume 3Haptic display is the process of applying forces to a human "observer" giving the sensation of touching and interacting with real physical objects. Touch is unique among the senses because it allows simultaneous exploration and manipulation of an ...
Touchable Video and Tactile Audio
ISM '09: Proceedings of the 2009 11th IEEE International Symposium on MultimediaWe propose a Haptic Audio Visual System (HAVS) which consists of a touchable video and a tactile audio. Touchable video generates tactile feedback according to the user’s real-time interactions on the video. Tactile audio provides the tactile ...
DIBHR: Depth Image-Based Haptic Rendering
EuroHaptics '08: Proceedings of the 6th international conference on Haptics: Perception, Devices and ScenariosThis paper presents Depth Image-Based Haptic Rendering (DIBHR), a haptic rendering algorithm that enables users to haptically explore 3D video media based on depth image-based representation (DIBR). The algorithm computes the shortest proxy (god-object)...






Comments