skip to main content
research-article

A Hardware-Efficient Architecture for Accurate Real-Time Disparity Map Estimation

Published:17 February 2015Publication History
Skip Abstract Section

Abstract

Emerging embedded vision systems utilize disparity estimation as a means to perceive depth information to intelligently interact with their host environment and take appropriate actions. Such systems demand high processing performance and accurate depth perception while requiring low energy consumption, especially when dealing with mobile and embedded applications, such as robotics, navigation, and security. The majority of real-time dedicated hardware implementations of disparity estimation systems have adopted local algorithms relying on simple cost aggregation strategies with fixed and rectangular correlation windows. However, such algorithms generally suffer from significant ambiguity along depth borders and areas with low texture. To this end, this article presents the hardware architecture of a disparity estimation system that enables good performance in both accuracy and speed. The architecture implements an adaptive support weight stereo correspondence algorithm that integrates image segmentation information in an attempt to increase the robustness of the matching process. The article also presents hardware-oriented algorithmic modifications/optimization techniques that make the algorithm hardware-friendly and suitable for efficient dedicated hardware implementation. A comparison to the literature asserts that an FPGA implementation of the proposed architecture is among the fastest implementations in terms of million disparity estimations per second (MDE/s), and with an overall accuracy of 90.21%, it presents an effective processing speed/disparity map accuracy trade-off.

References

  1. K. Ambrosch and W. Kubinger. 2010. Accurate hardware-based stereo vision. Computer Vision and Image Understanding 114, 1, 1303--1316. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. K. Ambrosch, M. Humenberger, W. Kubinger, and A. Steininger. 2009. A SAD-based stereo matching using FPGAs. In Embedded Computer Vision: Part II. London: Spinger, 121--138.Google ScholarGoogle Scholar
  3. M. Antunes, J. Barreto, C. Premebida, and U. Nunes. 2012. Can stereo vision replace a Laser Rangefinder? In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'12). 5183--5190.Google ScholarGoogle Scholar
  4. N. Baha and S. Larabi. 2011. Accurate real-time disparity map computation based on variable support window. International Journal of Artificial Intelligence and Applications 2, 3.Google ScholarGoogle ScholarCross RefCross Ref
  5. C. Banz, S. Hesselbarth, H. F. Blume, and P. Pirsch. 2011. Real-time stereo vision system using semi-global matching disparity estimation: Architecture and FPGA-implementation. Transactions on HiPEAC 5, 4.Google ScholarGoogle Scholar
  6. C. Banz, S. Hesselbarth, H. Flatt, H. Blume, and P. Pirsch. 2010. Real-time stereo vision system using semi-global matching disparity estimation: Architecture and FPGA-implementation. In Proceedings of the International Conference on Embedded Computer Systems (SAMOS'10). 93--101.Google ScholarGoogle Scholar
  7. R.-P. M. Berretty, A. K. Riemens, and P. F. Machado. 2007. Real-time embedded system for stereo video processing for multiview displays. In Proceedings of SPIE, Vol. 6490. Stereoscopic Displays and Virtual Reality Systems XIV. San Jose, CA.Google ScholarGoogle Scholar
  8. J.-Y. Bouguet. 2008. Camera Calibration Toolbox for Matlab. Retrieved August 2012, from http://www.vision.caltech.edu/bouguetj/calib_doc/index.htmlGoogle ScholarGoogle Scholar
  9. M. Z. Brown, D. Burschka, G. D. Hager, and S. Member. 2003. Advances in computational stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 993--1008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. S. Cavanag and M. Manzke. 2009. Real time disparity map estimation on the cell processor. In Proceedings of the 2009 Eurographics Ireland Workshop. 67--74.Google ScholarGoogle Scholar
  11. N. Chang, T.-M. Lin, T.-H. Tsai, Y.-C. Tseng, and T.-S. Chang. 2007. Real-time DSP implementation on local stereo matching. In Proceedings of the IEEE International Conference on Multimedia and Expo. 2090--2093.Google ScholarGoogle ScholarCross RefCross Ref
  12. N.-C. Chang, T.-H. Tsai, B.-H. Hsu, Y.-C. Chen, and T.-S. Chang. 2010. Algorithm and architecture of disparity estimation with mini-census adaptive support weight. IEEE Transactions on Circuits and Systems for Video Technology 20, 6, 792--805. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. B. Cyganek and J. P. Siebert. 2009. Introduction to 3D Computer Vision Techniques and Algorithms. John Wiley & Sons. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. A. Darabiha, J. MacLean, and J. Rose. 2006. Reconfigurable hardware implementation of a phase-correlation stereo algorithm. Machine Vision and Applications 17, 2, 116--132. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. J. Diaz, E. Ros, R. Carrillo, and A. Prieto. 2007. Real-time system for high-image resolution disparity estimation. IEEE Transactions on Image Processing 16, 1, 280--285. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. J. Ding, J. Liu, W. Zhou, H. Yu, Y. Wang, and X. Gong. 2011. Real-time stereo vision system using adaptive weight cost aggregation approach. EURASIP Journal on Image and Video Processing 1, 1--19.Google ScholarGoogle Scholar
  17. I. Ernst and H. Hirschmüller. 2008. Mutual information based semi-global stereo matching on the GPU. In Advances in Visual Computing. Lecture Notes in Computer Science, Vol. 5358. Springer, 228--239. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. S. Fahmy, P. Cheung, and W. Luk. 2005. Novel FPGA-based implementation of median and weighted median filters for image processing. In Proceedings of the International Conference on Field Programmable Logic and Applications. 142--147.Google ScholarGoogle Scholar
  19. Jianbin Fang, A. L. Varbanescu, Jie Shen, H. Sips, G. Saygili, and L. Van Der Maaten. 2012. Accelerating cost aggregation for real-time stereo matching. In Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems (ICPADS). 472--481. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. S. Foix, G. Alenya, and C. Torras. 2011. Lock-in time-of-flight (ToF) cameras: A survey. IEEE Sensors Journal 11, 9, 1917--1926.Google ScholarGoogle ScholarCross RefCross Ref
  21. S. Forstmann, Y. Kanou, Jun Ohya, S. Thuering, and A. Schmitt. 2004. Real-time stereo by using dynamic programming. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04). 29. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. J. Fowers, G. Brown, P. Cooke, and G. Stitt. 2012. A performance energy comparison of FPGAs, GPUs, and multicores for sliding-window applications. In Proceedings of the International Symposium on Field Programmable Gate Arrays. 47--56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. B. S. Freedman. 2010. Depth Mapping Using Projected Patterns. Prime Sense Ltd.Google ScholarGoogle Scholar
  24. S. Gehrig and C. Rabe. 2010. Real-time semi-global matching on the CPU. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW'10). 85--92.Google ScholarGoogle Scholar
  25. S. K. Gehrig, F. Eberli, and T. Meyer. 2009. A real-time low-power stereo vision engine using semi-global matching. In Computer Vision Systems. Lecture Notes in Computer Science, Vol. 5815. Springer, 134--143. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. C. Georgoulas and I. Andreadis. 2009. A real-time occlusion aware hardware structure for disparity map computation. In Image Analysis and Processing—ICIAP 2009. Lecture Notes in Computer Science, Vol. 5716. Springer, 721--730. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. S. Gokturk, H. Yalcin, and C. Bamji. 2004. A time-of-flight depth sensor—system description, issues and solutions. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop, Vol. 3. 35--43. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. A. Gray. 2012. GPU System Architecture. Retrieved November 15, 2012, from http://www2.epcc.ed.ac.uk/ ∼alang/GPU_training_aug12/GPU_Architecture.pdf.Google ScholarGoogle Scholar
  29. H. Hirschmüller. 2008. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2, 328--341. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. H. Hirschmüller and D. Scharstein. 2009. Evaluation of stereo matching costs on images with radiometric differences. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 9, 1582--1599. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. H. Hirschmüller, M. Buder, and I. Ernst. 2012. Memory efficient semi-global matching. ISPRS Annals of the Photogrammetry, Remote Sensing, and Spatial Information Sciences I-3, 371--376.Google ScholarGoogle Scholar
  32. H. Hirschmüller, P. R. Innocent, and J. Garibaldi. 2002. Real-time correlation-based stereo vision with reduced border errors. International Journal of Computer Vision 47, 1--3, 229--246. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. K.-Y. Hsieh, C.-H. Lai, S.-H. Lai, and J. K. Lee. 2012. Parallelization of belief propagation on cell processors for stereo vision. ACM Transactions on Embedded Computing Systems 11S, 1, Article No. 13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. M. Humenberger, C. Zinner, M. Weber, W. Kubinger, and M. Vincze. 2010. A fast stereo matching algorithm suitable for embedded real-time systems. Computer Vision and Image Understanding 114, 11, 1180--1202. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. S. Jin, J. Cho, X. D. Pham, K. M. Lee, S.-K. Park, M. Kim, and J. W. Jeon. 2010. FPGA design and implementation of a real-time stereo vision system. IEEE Transactions on Circuits and Systems for Video Technology 20, 1, 15--26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. B. Khaleghi, S. Ahuja, and Q. Wu. 2008. An improved real-time miniaturized embedded stereo vision system (MESVS-II). In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 1--8.Google ScholarGoogle Scholar
  37. K. Khoshelham. 2011. Accuracy analysis of Kinect depth data. In Proceedings of the ISPRS Workshop on Laser Scanning. 133--138.Google ScholarGoogle Scholar
  38. K. Konolige. 1997. Small vision system—hardware and implementation. In Proceedings of the International Symposium on Robotics Research. 111--116.Google ScholarGoogle Scholar
  39. K. Konolige, M. Agrawal, R. C. Bolles, C. Cowan, M. Fischler, and B. P. Gerkey. 2006. Outdoor mapping and navigation using stereo vision. In Proceedings of the International Symposium on Experimental Robotics (ISER'06). 179--190.Google ScholarGoogle Scholar
  40. J. Liu, Y. Xu, R. Klette, H. Chen, W. Rong, and T. Vaudrey. 2009. Disparity map computation on a cell processor. In Proceedings of the IASTED International Conference on Modelling, Simulation, and Identification (MSI'09).Google ScholarGoogle Scholar
  41. W. J. MacLean. 2005. An evaluation of the suitability of FPGAs for embedded vision systems. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 131. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Microsoft Kinect. 2010. Kinect for Xbox One. Retrieved December 15, 2012, from http://www.xbox.com/en-us/kinect/.Google ScholarGoogle Scholar
  43. I. H. Moring. 1989. Acquisition of three-dimensional image data by a scanning laser range finder. Optical Engineering 28, 8, 897--902.Google ScholarGoogle ScholarCross RefCross Ref
  44. K. M. Muhlmann. 2001. Calculating dense disparity maps from color stereo images, an efficient implementation. In Proceedings of the IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV'01). 30--36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. D. H. Park, H. S. Ko, J. G. Kim, and J. D. Cho. 2011. Real time rectification using differentially encoded lookup table. In Proceedings of the 5th International Conference on Ubiquitous Information Management and Communication (ICUIMC'11). 1--4. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. P. Steingrube, S. K. Gehrig, and U. Franke. 2009. Performance evaluation of stereo algorithms for automotive applications. In Computer Vision Systems. Lecture Notes in Computer Science, Vol. 5815. Springer, 285--294. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. H. Sunyoto, W. van der Mark, and D. Gavrila. 2004. A comparative study of fast dense stereo vision algorithms. In Proceedings of the IEEE Intelligent Vehicles Symposium. 319--324.Google ScholarGoogle Scholar
  48. D. Szeliski and R. Scharstein. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47, 7--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Tokyo Electron Device Limited. 2010. Kintex-7 FPGA Display Kit. Retrieved September 15, 2012, from http://solutions.inrevium.com/products/kits/consumer/tb-7k-acdc.html.Google ScholarGoogle Scholar
  50. F. Tombari, S. Mattoccia, and L. D. Stefano. 2007. Segmentation-based adaptive support for accurate stereo correspondence. In Advances in Image and Video Technology. Lecture Notes in Computer Science, Vol. 4872. Springer, 427--438. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. F. Tombari, S. Mattoccia, L. D. Stefano, and E. Addimanda. 2008. Classification and evaluation of cost aggregation methods for stereo correspondence. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. 1--8.Google ScholarGoogle Scholar
  52. C. Ttofis and T. Theocharides. 2012. Towards accurate hardware stereo correspondence: A real-time FPGA implementation of a segmentation-based adaptive support weight algorithm. In Proceedings of the Design, Automation, and Test in Europe Conference and Exhibition (DATE'12). 703--708. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. C. Ttofis, S. Hadjitheophanous, A. Georghiades, and T. Theocharides. 2013. Edge-directed hardware architecture for real-time disparity map computation. IEEE Transactions on Computers 62, 4, 690--704. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. T. Vaudrey, C. Rabe, R. Klette, and J. Milburn. 2008. Differences between stereo and motion behaviour on synthetic and real-world stereo sequences. In Proceedings of the 23rd International Conference on Image and Vision Computing. 1--6.Google ScholarGoogle Scholar
  55. L. Wang, M. Liao, M. Gong, R. Yang, and D. Nister. 2006. High-quality real-time stereo using adaptive cost aggregation and dynamic programming. In Proceedings of the 3rd International Symposium on 3D Data Processing, Visualization, and Transmission. 798--805. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. X. Xiang, M. Zhang, G. Li, Y. He, and Z. Pan. 2012. Real-time stereo matching based on fast belief propagation. Machine Vision and Applications 23, 6, 1219--1227.Google ScholarGoogle ScholarCross RefCross Ref
  57. Xilinx Inc. 2012. Kintex-7 FPGA Family. Retrieved September 15, 2012, from http://www.xilinx.com/ products/silicon-devices/fpga/kintex-7/.Google ScholarGoogle Scholar
  58. Q. Yang, L. Wang, R. Yang, S. Wang, M. Liao, and D. Nister. 2006. Real-time global stereo matching using hierarchical belief propagation. In Proceedings of the British Machine Vision Conference.Google ScholarGoogle Scholar
  59. R. Yang and M. Pollefeys. 2005. A versatile stereo implementation on commodity graphics hardware. Real-Time Imaging 11, 1, 7--18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. K.-J. Yoon and I.-S. Kweon. 2006. Adaptive support-weight approach for correspondence search. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 4, 650--656. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. C. Zinner, M. Humenberger, K. Ambrosch, and W. Kubinger. 2008. An optimized software-based implementation of a census-based stereo matching algorithm. In Advances in Visual Computing. Lecture Notes in Computer Science, Vol. 5358. Springer, 216--227. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. C. Zinner, M. Humenberger, K. Ambrosch, and W. Kubinger. 2008. An optimized software-based implementation of a census-based stereo matching algorithm. In Proceedings of the 4th International Symposium on Advances in Visual Computing (ISVC'08). 216--227. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. A Hardware-Efficient Architecture for Accurate Real-Time Disparity Map Estimation

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!