Abstract
This article proposes an application-layer multiplexing scheme for teleoperation systems with multimodal feedback (video, audio, and haptics). The available transmission resources are carefully allocated to avoid delay-jitter for the haptic signal potentially caused by the size and arrival time of the video and audio data. The multiplexing scheme gives high priority to the haptic signal and applies a preemptive-resume scheduling strategy to stream the audio and video data. The proposed approach estimates the available transmission rate in real time and adapts the video bitrate, data throughput, and force buffer size accordingly. Furthermore, the proposed scheme detects sudden transmission rate drops and applies congestion control to avoid abrupt delay increases and converge promptly to the altered transmission rate. The performance of the proposed scheme is measured objectively in terms of end-to-end signal latencies, packet rates, and peak signal-to-noise ratio (PSNR) for visual quality. Moreover, peak-delay and convergence time measurements are carried out to investigate the performance of the congestion control mode of the system.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, A Multiplexing Scheme for Multimodal Teleoperation
- Allied Vision. 1992. MAKO G-223 Gigabit Ethernet Industrial Machine Vision Camera. Retrieved March 1, 2017 from https://www.alliedvision.com.Google Scholar
- APPOSITE-TECH. 2005. Netropy N60 Hardware Network Emulator. Retrieved March 1, 2017 from http://www.apposite-tech.com/.Google Scholar
- C. Bachhuber and E. Steinbach. 2016. A system for high precision glass-to-glass delay measurements in video communication. In IEEE International Conference on Image Processing (ICIP’16). 2132--2136.Google Scholar
- L. S. Brakmo and L. L. Peterson. 1995. TCP Vegas: End-to-end congestion avoidance on a global internet. IEEE Journal on Selected Areas in Communications 13, 1465--1480. Google Scholar
Digital Library
- G. C. Burdea. 1996. Force and Touch Feedback for Virtual Reality. John Wiley 8 Sons, New York, NY. Google Scholar
Digital Library
- A. Capone, L. Fratta, and F. Martignon. 2004. Bandwidth estimation schemes for TCP over wireless networks. IEEE Transactions on Mobile Computing 3, 2, 129--143.Google Scholar
Digital Library
- C. Casetti, M. Gerla, S. Mascolo, M. Y. Sanadidi, and R. Wang. 2002. TCP westwood: End-to-end congestion control for wired/wireless networks. Springer-Verlag Wireless Networks 8, 5, 467--479. Google Scholar
Digital Library
- Z. Cen, M. Mutka, Y. Liu, A. Goradia, and N. Xi. 2005a. QoS management of supermedia enhanced teleoperation via overlay networks. In IEEE International Conference on Intelligent Robots and Systems. 1630--1635.Google Scholar
- Z. Cen, M. W. Mutka, Danyu Zhu, and Ning Xi. 2005b. Improved transport service for remote sensing and control over wireless networks. In IEEE International Conference on Mobile Adhoc and Sensor Systems. 340--347.Google Scholar
- J. Cha, Y. Seo, Y. Kim, and J. Ryu. 2007. An authoring/editing framework for haptic broadcasting: Passive haptic interactions using MPEG-4 BIFS. In IEEE Proceedings of EuroHaptics Conference. 274--279. Google Scholar
Digital Library
- B. Cizmeci, R. Chaudhari, X. Xu, N. Alt, and E. Steinbach. 2014. A visual-haptic multiplexing scheme for teleoperation over constant-bitrate communication links. Haptics: Neuroscience, Devices, Modeling, and Applications, Lecture Notes in Computer Science, Vol. 8619, 131--138.Google Scholar
Cross Ref
- J. E. Colgate and J. M. Brown. 1994. Factors affecting the z-width of a haptic display. In IEEE International Conference on Robotics and Automation. 3205--3210.Google Scholar
- M. U. Demircin, P. van Beek, and Y. Altunbasak. 2008. Delay-constrained and r-d optimized transrating for high-definition video streaming over WLANs. IEEE Transactions on Multimedia 10, 6, 1155--1168. Google Scholar
Digital Library
- E. Dinc and O. B. Akan. 2015. More than the eye can see: Coherence time and coherence bandwidth of troposcatter links for mobile receivers. IEEE Vehicular Technology Magazine 10, 2, 86--92.Google Scholar
Cross Ref
- M. A. Eid, J. Cha, and A. El-Saddik. 2011. Admux: An adaptive multiplexer for haptic-audio-visual data communication. IEEE Transactions on Instrumentation and Measurement 60, 1, 21--31.Google Scholar
Cross Ref
- W. R. Ferrell. 1965. Remote manipulation with transmission delay. IEEE Transactions on Human Factors in Electronics 6, 1, 24--32.Google Scholar
Cross Ref
- FORCE DIMENSION. 2001. Omega 6, 6-DoF Haptic Device. Retrieved March 1, 2017 from http://www.forcedimension.com/.Google Scholar
- M. Gao, B. Cizmeci, M. Eiler, E. Steinbach, D. Zhao, and W. Gao. 2015. Macroblock level rate control for low delay H.264/AVC based video communication. In IEEE Picture Coding Symposium. 210--215.Google Scholar
- M. Goeller, J. Oberlaender, K. Uhl, A. Roennau, and R. Dillmann. 2012. Modular robots for on-orbit satellite servicing. In IEEE International Conference on Robotics and Biomimetics (ROBIO’12). 2018--2023.Google Scholar
- P. Hinterseer, S. Hirche, S. Chaudhuri, E. Steinbach, and M. Buss. 2008. Perception-based data reduction and transmission of haptic data in telepresence and teleaction systems. IEEE Transactions on Signal Processing 56, 2, 588--597. Google Scholar
Digital Library
- P. Hinterseer, E. Steinbach, S. Hirche, and M. Buss. 2005. A novel psychophysically motivated transmission approach for haptic data streams in telepresence and teleaction systems. In IEEE International Conference on Acoustics, Speech and Signal Processing. 1097--1100.Google Scholar
- S. Hirche, P. Hinterseer, E. Steinbach, and M. Buss. 2005. Network traffic reduction in haptic telepresence systems by deadband control. In 16th IFAC World Congress. 77--82.Google Scholar
- E. Isomura, S. Tasaka, and T. Nunome. 2011. QoE enhancement in audiovisual and haptic interactive IP communications by media adaptive intra-stream synchronization. In TENCON 2011 - IEEE Region 10 Conference. 1085--1089.Google Scholar
- ITU-T. 1993. ITU-T JPEG Standard, Digital Compression and Coding of Continuous-tone Still Images.Google Scholar
- ITU-T. 2005. ITU-T H.264, Advanced Video Coding for Generic Audiovisual Services.Google Scholar
- S. Kaede, T. Nunome, and S. Tasaka. 2015. QoE enhancement of audiovisual and haptic interactive IP communications by user-assistance. In IEEE International Conference on Computational Science and Engineering. 35--42. Google Scholar
Digital Library
- KUKA Robotics. 1898. KUKA Leight Weight Robot Arm. (1898). Retrieved March 1, 2017 from http://www.kuka-robotics.com/.Google Scholar
- M. Kuschel, P. Kremer, S. Hirche, and M. Buss. 2006. Lossy data reduction methods for haptic telepresence systems. In IEEE International Conference on Robotics and Automation. 2933--2938.Google Scholar
- S. Lee, S. Moon, and J. W. Kim. 2006. A network-adaptive transport scheme for haptic-based collaborative virtual environments. In Proceedings of 5th ACM SIGCOMM Workshop on Network and System Support for Games, Article No. 13. Google Scholar
Digital Library
- Alan Marshall, Kian Meng Yap, and Wai Yu. 2008. Providing QoS for networked peers in distributed haptic virtual environments. Hindawi, Journal of Advances in Multimedia 2008, 841590.Google Scholar
- L. Merritt and R. Vanam. 2004. x264: A High Performance H.264/AVC Encoder. Retrieved March 1, 2017 from http://www.videolan.org/developers/x264.html.Google Scholar
- H. A. Osman, M. Eid, R. Iglesias, and A. El Saddik. 2007. ALPHAN: Application layer protocol for haptic networking. In IEEE International Workshop on Haptic, Audio and Visual Environments and Games. 96--101.Google Scholar
- L. Ping, L. Wenjuan, and S. Zengqi. 2005. Transport layer protocol reconfiguration for network-based robot control system. In IEEE Proceedings of Networking, Sensing and Control. 1049--1053.Google Scholar
- PortAudio. 1999. Portable Cross Platform Audio I/O. Retrieved March 1, 2017 from http://portaudio.com/.Google Scholar
- J. Pyke, M. Hart, V. Popov, R. D. Harris, and S. McGrath. 2007. A tele-ultrasound system for real-time medical imaging in resource-limited settings. In IEEE International Conference of Engineering in Medicine and Biology Society. 3094--3097.Google Scholar
- John Ramming. 1983. JR3 6-DoF Force-Torque Sensor. Retrieved March 1, 2017 from http://www.jr3.com/.Google Scholar
- M. Rank, Z. Shi, H. Müller, and S. Hirche. 2010. Perception of delay in haptic telepresence systems. MIT Presence v.19, n.5 (2010), p. 389--399. Google Scholar
Digital Library
- J.-H. Ryu, J. Artigas, and C. Preusche. 2010. A passive bilateral control scheme for a teleoperator with time-varying communication delay. Mechatronics 20, 7, 812--823.Google Scholar
Cross Ref
- T. B. Sheridan. 1993. Space teleoperation through time delay: Review and prognosis. IEEE Transactions on Robotics and Automation 9, 5, 592--606.Google Scholar
- J. M. Silva, M. Orozco, J. Cha, A. El Saddik, and E. M. Petriu. 2013. Human perception of haptic-to-video and haptic-to-audio skew in multimedia applications. ACM Transactions on Multimedia Computing, Communications and Applications 9, 2, 9:1--9:16. Google Scholar
Digital Library
- I. Stoica, S. Shenker, and H. Zhang. 1998. Core-stateless fair queueing: Achieving approximately fair bandwidth allocations in high speed networks. ACM SIGCOMM Computer Communication Review 28, 4, 118--130. Google Scholar
Digital Library
- Y. Uchimura, K. Ohnishi, and T. Yakoh. 2002. Bilateral robot system on the real time network structure. In IEEE International Workshop on Advanced Motion Control. 63--68.Google Scholar
- J. M. Valin, T. B. Terriberry, C. Montgomery, and G. Maxwell. 2010. A high-quality speech and audio codec with less than 10-ms delay. IEEE Transactions on Audio, Speech, and Language Processing 18, 1, 58--67.Google Scholar
Digital Library
- I. Vogels. 2004. Detection of temporal delays in visual-haptic interfaces. HUMAN FACTORS 46, 1, 118--134.Google Scholar
Cross Ref
- J. Walraevens. 2005. Discrete-Time Queueing Models with Priorities. Ph.D. Dissertation, (TELIN) Ghent University, Ghent, Belgium.Google Scholar
- E. Weber. 1851. Die lehre vom tastsinn und gemeingefuehl, auf versuche gegruendet. Vieweg: Braunschweig, Germany.Google Scholar
- X. Xu, B. Cizmeci, C. Schuwerk, and E. Steinbach. 2015. Haptic data reduction for time-delayed teleoperation using the time domain passivity approach. In IEEE World Haptics Conference. 512--518.Google Scholar
- S. Yamamoto, D. Yashiro, K. Yubai, and S. Komada. 2014. Rate control based on queuing state observer for visual-haptic communication. In IEEE International Workshop on Advanced Motion Control. 569--574.Google Scholar
- D. Yashiro, D. Tian, and T. Yakoh. 2013. End-to-end flow control for visual-haptic communication in the presence of bandwidth change. Wiley Electronics and Communications in Japan 96, 11, 26--34.Google Scholar
Cross Ref
- Youtube. 2016a. Combining Haptic Data Reduction with Stabilizing Control Approaches. Retrieved March 1, 2017 from https://youtu.be/0GzIm2R8ICY.Google Scholar
- Youtube. 2016b. Ultra-Low Delay Video Transmission. Retrieved March 1, 2017 from https://youtu.be/dl9Figh-ymk.Google Scholar
- Youtube. 2017. A Multiplexing Scheme for Multimodal Teleoperation. Retrieved March 1, 2017 from https://youtu.be/a1MSrvENElg.Google Scholar
- F. Zhang and E. Steinbach. 2011. Improved ρ-domain rate control with accurate header size estimation. In IEEE International Conference on Acoustics, Speech and Signal Processing. 813--816.Google Scholar
- H. Zhihai and S. K. Mitra. 2002. Optimum bit allocation and accurate rate control for video coding via ρ-domain source modeling. IEEE Transactions on Circuits and Systems for Video Technology 12, 10, 840--849. Google Scholar
Digital Library
Index Terms
A Multiplexing Scheme for Multimodal Teleoperation
Recommendations
Efficient multiplexing scheme for IP packets over the advanced satellite broadcasting system
Implementing an IP-friendly transport mechanism in broadcasting systems would enable more convenient multimedia broadcasting applications. The overhead of current multiplexing methods for IP packets that are based on MPEG-2 systems is large, and they ...
Delta-Multiplexing: A Novel Technique to Improve VoIP Bandwidth Utilization between VoIP Gateways
CIT '10: Proceedings of the 2010 10th IEEE International Conference on Computer and Information TechnologyGradually, Voice over Internet Protocol (VoIP) has been dominating the telecommunications world. Unfortunately, its applications are injecting a huge number of small packets in the network, which produces high overhead and therefore wastes network ...
An Enhanced TCP Scheme for Distinguishing Non-congestion Losses from Packet Reordering over Wireless Mesh Networks
HPCC '11: Proceedings of the 2011 IEEE International Conference on High Performance Computing and CommunicationsWireless Mesh Network (WMN) is a communication networks, which aims to provide ubiquitous high bandwidth access for a large number of users. Transmission Control Protocol (TCP) is the most popular connection oriented transport layer protocol used in the ...






Comments