skip to main content
research-article

A Multiplexing Scheme for Multimodal Teleoperation

Authors Info & Claims
Published:11 April 2017Publication History
Skip Abstract Section

Abstract

This article proposes an application-layer multiplexing scheme for teleoperation systems with multimodal feedback (video, audio, and haptics). The available transmission resources are carefully allocated to avoid delay-jitter for the haptic signal potentially caused by the size and arrival time of the video and audio data. The multiplexing scheme gives high priority to the haptic signal and applies a preemptive-resume scheduling strategy to stream the audio and video data. The proposed approach estimates the available transmission rate in real time and adapts the video bitrate, data throughput, and force buffer size accordingly. Furthermore, the proposed scheme detects sudden transmission rate drops and applies congestion control to avoid abrupt delay increases and converge promptly to the altered transmission rate. The performance of the proposed scheme is measured objectively in terms of end-to-end signal latencies, packet rates, and peak signal-to-noise ratio (PSNR) for visual quality. Moreover, peak-delay and convergence time measurements are carried out to investigate the performance of the congestion control mode of the system.

Skip Supplemental Material Section

Supplemental Material

References

  1. Allied Vision. 1992. MAKO G-223 Gigabit Ethernet Industrial Machine Vision Camera. Retrieved March 1, 2017 from https://www.alliedvision.com.Google ScholarGoogle Scholar
  2. APPOSITE-TECH. 2005. Netropy N60 Hardware Network Emulator. Retrieved March 1, 2017 from http://www.apposite-tech.com/.Google ScholarGoogle Scholar
  3. C. Bachhuber and E. Steinbach. 2016. A system for high precision glass-to-glass delay measurements in video communication. In IEEE International Conference on Image Processing (ICIP’16). 2132--2136.Google ScholarGoogle Scholar
  4. L. S. Brakmo and L. L. Peterson. 1995. TCP Vegas: End-to-end congestion avoidance on a global internet. IEEE Journal on Selected Areas in Communications 13, 1465--1480. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. G. C. Burdea. 1996. Force and Touch Feedback for Virtual Reality. John Wiley 8 Sons, New York, NY. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. A. Capone, L. Fratta, and F. Martignon. 2004. Bandwidth estimation schemes for TCP over wireless networks. IEEE Transactions on Mobile Computing 3, 2, 129--143.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. C. Casetti, M. Gerla, S. Mascolo, M. Y. Sanadidi, and R. Wang. 2002. TCP westwood: End-to-end congestion control for wired/wireless networks. Springer-Verlag Wireless Networks 8, 5, 467--479. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Z. Cen, M. Mutka, Y. Liu, A. Goradia, and N. Xi. 2005a. QoS management of supermedia enhanced teleoperation via overlay networks. In IEEE International Conference on Intelligent Robots and Systems. 1630--1635.Google ScholarGoogle Scholar
  9. Z. Cen, M. W. Mutka, Danyu Zhu, and Ning Xi. 2005b. Improved transport service for remote sensing and control over wireless networks. In IEEE International Conference on Mobile Adhoc and Sensor Systems. 340--347.Google ScholarGoogle Scholar
  10. J. Cha, Y. Seo, Y. Kim, and J. Ryu. 2007. An authoring/editing framework for haptic broadcasting: Passive haptic interactions using MPEG-4 BIFS. In IEEE Proceedings of EuroHaptics Conference. 274--279. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. B. Cizmeci, R. Chaudhari, X. Xu, N. Alt, and E. Steinbach. 2014. A visual-haptic multiplexing scheme for teleoperation over constant-bitrate communication links. Haptics: Neuroscience, Devices, Modeling, and Applications, Lecture Notes in Computer Science, Vol. 8619, 131--138.Google ScholarGoogle ScholarCross RefCross Ref
  12. J. E. Colgate and J. M. Brown. 1994. Factors affecting the z-width of a haptic display. In IEEE International Conference on Robotics and Automation. 3205--3210.Google ScholarGoogle Scholar
  13. M. U. Demircin, P. van Beek, and Y. Altunbasak. 2008. Delay-constrained and r-d optimized transrating for high-definition video streaming over WLANs. IEEE Transactions on Multimedia 10, 6, 1155--1168. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. E. Dinc and O. B. Akan. 2015. More than the eye can see: Coherence time and coherence bandwidth of troposcatter links for mobile receivers. IEEE Vehicular Technology Magazine 10, 2, 86--92.Google ScholarGoogle ScholarCross RefCross Ref
  15. M. A. Eid, J. Cha, and A. El-Saddik. 2011. Admux: An adaptive multiplexer for haptic-audio-visual data communication. IEEE Transactions on Instrumentation and Measurement 60, 1, 21--31.Google ScholarGoogle ScholarCross RefCross Ref
  16. W. R. Ferrell. 1965. Remote manipulation with transmission delay. IEEE Transactions on Human Factors in Electronics 6, 1, 24--32.Google ScholarGoogle ScholarCross RefCross Ref
  17. FORCE DIMENSION. 2001. Omega 6, 6-DoF Haptic Device. Retrieved March 1, 2017 from http://www.forcedimension.com/.Google ScholarGoogle Scholar
  18. M. Gao, B. Cizmeci, M. Eiler, E. Steinbach, D. Zhao, and W. Gao. 2015. Macroblock level rate control for low delay H.264/AVC based video communication. In IEEE Picture Coding Symposium. 210--215.Google ScholarGoogle Scholar
  19. M. Goeller, J. Oberlaender, K. Uhl, A. Roennau, and R. Dillmann. 2012. Modular robots for on-orbit satellite servicing. In IEEE International Conference on Robotics and Biomimetics (ROBIO’12). 2018--2023.Google ScholarGoogle Scholar
  20. P. Hinterseer, S. Hirche, S. Chaudhuri, E. Steinbach, and M. Buss. 2008. Perception-based data reduction and transmission of haptic data in telepresence and teleaction systems. IEEE Transactions on Signal Processing 56, 2, 588--597. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. P. Hinterseer, E. Steinbach, S. Hirche, and M. Buss. 2005. A novel psychophysically motivated transmission approach for haptic data streams in telepresence and teleaction systems. In IEEE International Conference on Acoustics, Speech and Signal Processing. 1097--1100.Google ScholarGoogle Scholar
  22. S. Hirche, P. Hinterseer, E. Steinbach, and M. Buss. 2005. Network traffic reduction in haptic telepresence systems by deadband control. In 16th IFAC World Congress. 77--82.Google ScholarGoogle Scholar
  23. E. Isomura, S. Tasaka, and T. Nunome. 2011. QoE enhancement in audiovisual and haptic interactive IP communications by media adaptive intra-stream synchronization. In TENCON 2011 - IEEE Region 10 Conference. 1085--1089.Google ScholarGoogle Scholar
  24. ITU-T. 1993. ITU-T JPEG Standard, Digital Compression and Coding of Continuous-tone Still Images.Google ScholarGoogle Scholar
  25. ITU-T. 2005. ITU-T H.264, Advanced Video Coding for Generic Audiovisual Services.Google ScholarGoogle Scholar
  26. S. Kaede, T. Nunome, and S. Tasaka. 2015. QoE enhancement of audiovisual and haptic interactive IP communications by user-assistance. In IEEE International Conference on Computational Science and Engineering. 35--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. KUKA Robotics. 1898. KUKA Leight Weight Robot Arm. (1898). Retrieved March 1, 2017 from http://www.kuka-robotics.com/.Google ScholarGoogle Scholar
  28. M. Kuschel, P. Kremer, S. Hirche, and M. Buss. 2006. Lossy data reduction methods for haptic telepresence systems. In IEEE International Conference on Robotics and Automation. 2933--2938.Google ScholarGoogle Scholar
  29. S. Lee, S. Moon, and J. W. Kim. 2006. A network-adaptive transport scheme for haptic-based collaborative virtual environments. In Proceedings of 5th ACM SIGCOMM Workshop on Network and System Support for Games, Article No. 13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Alan Marshall, Kian Meng Yap, and Wai Yu. 2008. Providing QoS for networked peers in distributed haptic virtual environments. Hindawi, Journal of Advances in Multimedia 2008, 841590.Google ScholarGoogle Scholar
  31. L. Merritt and R. Vanam. 2004. x264: A High Performance H.264/AVC Encoder. Retrieved March 1, 2017 from http://www.videolan.org/developers/x264.html.Google ScholarGoogle Scholar
  32. H. A. Osman, M. Eid, R. Iglesias, and A. El Saddik. 2007. ALPHAN: Application layer protocol for haptic networking. In IEEE International Workshop on Haptic, Audio and Visual Environments and Games. 96--101.Google ScholarGoogle Scholar
  33. L. Ping, L. Wenjuan, and S. Zengqi. 2005. Transport layer protocol reconfiguration for network-based robot control system. In IEEE Proceedings of Networking, Sensing and Control. 1049--1053.Google ScholarGoogle Scholar
  34. PortAudio. 1999. Portable Cross Platform Audio I/O. Retrieved March 1, 2017 from http://portaudio.com/.Google ScholarGoogle Scholar
  35. J. Pyke, M. Hart, V. Popov, R. D. Harris, and S. McGrath. 2007. A tele-ultrasound system for real-time medical imaging in resource-limited settings. In IEEE International Conference of Engineering in Medicine and Biology Society. 3094--3097.Google ScholarGoogle Scholar
  36. John Ramming. 1983. JR3 6-DoF Force-Torque Sensor. Retrieved March 1, 2017 from http://www.jr3.com/.Google ScholarGoogle Scholar
  37. M. Rank, Z. Shi, H. Müller, and S. Hirche. 2010. Perception of delay in haptic telepresence systems. MIT Presence v.19, n.5 (2010), p. 389--399. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. J.-H. Ryu, J. Artigas, and C. Preusche. 2010. A passive bilateral control scheme for a teleoperator with time-varying communication delay. Mechatronics 20, 7, 812--823.Google ScholarGoogle ScholarCross RefCross Ref
  39. T. B. Sheridan. 1993. Space teleoperation through time delay: Review and prognosis. IEEE Transactions on Robotics and Automation 9, 5, 592--606.Google ScholarGoogle Scholar
  40. J. M. Silva, M. Orozco, J. Cha, A. El Saddik, and E. M. Petriu. 2013. Human perception of haptic-to-video and haptic-to-audio skew in multimedia applications. ACM Transactions on Multimedia Computing, Communications and Applications 9, 2, 9:1--9:16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. I. Stoica, S. Shenker, and H. Zhang. 1998. Core-stateless fair queueing: Achieving approximately fair bandwidth allocations in high speed networks. ACM SIGCOMM Computer Communication Review 28, 4, 118--130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Y. Uchimura, K. Ohnishi, and T. Yakoh. 2002. Bilateral robot system on the real time network structure. In IEEE International Workshop on Advanced Motion Control. 63--68.Google ScholarGoogle Scholar
  43. J. M. Valin, T. B. Terriberry, C. Montgomery, and G. Maxwell. 2010. A high-quality speech and audio codec with less than 10-ms delay. IEEE Transactions on Audio, Speech, and Language Processing 18, 1, 58--67.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. I. Vogels. 2004. Detection of temporal delays in visual-haptic interfaces. HUMAN FACTORS 46, 1, 118--134.Google ScholarGoogle ScholarCross RefCross Ref
  45. J. Walraevens. 2005. Discrete-Time Queueing Models with Priorities. Ph.D. Dissertation, (TELIN) Ghent University, Ghent, Belgium.Google ScholarGoogle Scholar
  46. E. Weber. 1851. Die lehre vom tastsinn und gemeingefuehl, auf versuche gegruendet. Vieweg: Braunschweig, Germany.Google ScholarGoogle Scholar
  47. X. Xu, B. Cizmeci, C. Schuwerk, and E. Steinbach. 2015. Haptic data reduction for time-delayed teleoperation using the time domain passivity approach. In IEEE World Haptics Conference. 512--518.Google ScholarGoogle Scholar
  48. S. Yamamoto, D. Yashiro, K. Yubai, and S. Komada. 2014. Rate control based on queuing state observer for visual-haptic communication. In IEEE International Workshop on Advanced Motion Control. 569--574.Google ScholarGoogle Scholar
  49. D. Yashiro, D. Tian, and T. Yakoh. 2013. End-to-end flow control for visual-haptic communication in the presence of bandwidth change. Wiley Electronics and Communications in Japan 96, 11, 26--34.Google ScholarGoogle ScholarCross RefCross Ref
  50. Youtube. 2016a. Combining Haptic Data Reduction with Stabilizing Control Approaches. Retrieved March 1, 2017 from https://youtu.be/0GzIm2R8ICY.Google ScholarGoogle Scholar
  51. Youtube. 2016b. Ultra-Low Delay Video Transmission. Retrieved March 1, 2017 from https://youtu.be/dl9Figh-ymk.Google ScholarGoogle Scholar
  52. Youtube. 2017. A Multiplexing Scheme for Multimodal Teleoperation. Retrieved March 1, 2017 from https://youtu.be/a1MSrvENElg.Google ScholarGoogle Scholar
  53. F. Zhang and E. Steinbach. 2011. Improved ρ-domain rate control with accurate header size estimation. In IEEE International Conference on Acoustics, Speech and Signal Processing. 813--816.Google ScholarGoogle Scholar
  54. H. Zhihai and S. K. Mitra. 2002. Optimum bit allocation and accurate rate control for video coding via ρ-domain source modeling. IEEE Transactions on Circuits and Systems for Video Technology 12, 10, 840--849. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. A Multiplexing Scheme for Multimodal Teleoperation

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!