Abstract
Emerging embedded vision systems utilize disparity estimation as a means to perceive depth information to intelligently interact with their host environment and take appropriate actions. Such systems demand high processing performance and accurate depth perception while requiring low energy consumption, especially when dealing with mobile and embedded applications, such as robotics, navigation, and security. The majority of real-time dedicated hardware implementations of disparity estimation systems have adopted local algorithms relying on simple cost aggregation strategies with fixed and rectangular correlation windows. However, such algorithms generally suffer from significant ambiguity along depth borders and areas with low texture. To this end, this article presents the hardware architecture of a disparity estimation system that enables good performance in both accuracy and speed. The architecture implements an adaptive support weight stereo correspondence algorithm that integrates image segmentation information in an attempt to increase the robustness of the matching process. The article also presents hardware-oriented algorithmic modifications/optimization techniques that make the algorithm hardware-friendly and suitable for efficient dedicated hardware implementation. A comparison to the literature asserts that an FPGA implementation of the proposed architecture is among the fastest implementations in terms of million disparity estimations per second (MDE/s), and with an overall accuracy of 90.21%, it presents an effective processing speed/disparity map accuracy trade-off.
- K. Ambrosch and W. Kubinger. 2010. Accurate hardware-based stereo vision. Computer Vision and Image Understanding 114, 1, 1303--1316. Google Scholar
Digital Library
- K. Ambrosch, M. Humenberger, W. Kubinger, and A. Steininger. 2009. A SAD-based stereo matching using FPGAs. In Embedded Computer Vision: Part II. London: Spinger, 121--138.Google Scholar
- M. Antunes, J. Barreto, C. Premebida, and U. Nunes. 2012. Can stereo vision replace a Laser Rangefinder? In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'12). 5183--5190.Google Scholar
- N. Baha and S. Larabi. 2011. Accurate real-time disparity map computation based on variable support window. International Journal of Artificial Intelligence and Applications 2, 3.Google Scholar
Cross Ref
- C. Banz, S. Hesselbarth, H. F. Blume, and P. Pirsch. 2011. Real-time stereo vision system using semi-global matching disparity estimation: Architecture and FPGA-implementation. Transactions on HiPEAC 5, 4.Google Scholar
- C. Banz, S. Hesselbarth, H. Flatt, H. Blume, and P. Pirsch. 2010. Real-time stereo vision system using semi-global matching disparity estimation: Architecture and FPGA-implementation. In Proceedings of the International Conference on Embedded Computer Systems (SAMOS'10). 93--101.Google Scholar
- R.-P. M. Berretty, A. K. Riemens, and P. F. Machado. 2007. Real-time embedded system for stereo video processing for multiview displays. In Proceedings of SPIE, Vol. 6490. Stereoscopic Displays and Virtual Reality Systems XIV. San Jose, CA.Google Scholar
- J.-Y. Bouguet. 2008. Camera Calibration Toolbox for Matlab. Retrieved August 2012, from http://www.vision.caltech.edu/bouguetj/calib_doc/index.htmlGoogle Scholar
- M. Z. Brown, D. Burschka, G. D. Hager, and S. Member. 2003. Advances in computational stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 993--1008. Google Scholar
Digital Library
- S. Cavanag and M. Manzke. 2009. Real time disparity map estimation on the cell processor. In Proceedings of the 2009 Eurographics Ireland Workshop. 67--74.Google Scholar
- N. Chang, T.-M. Lin, T.-H. Tsai, Y.-C. Tseng, and T.-S. Chang. 2007. Real-time DSP implementation on local stereo matching. In Proceedings of the IEEE International Conference on Multimedia and Expo. 2090--2093.Google Scholar
Cross Ref
- N.-C. Chang, T.-H. Tsai, B.-H. Hsu, Y.-C. Chen, and T.-S. Chang. 2010. Algorithm and architecture of disparity estimation with mini-census adaptive support weight. IEEE Transactions on Circuits and Systems for Video Technology 20, 6, 792--805. Google Scholar
Digital Library
- B. Cyganek and J. P. Siebert. 2009. Introduction to 3D Computer Vision Techniques and Algorithms. John Wiley & Sons. Google Scholar
Digital Library
- A. Darabiha, J. MacLean, and J. Rose. 2006. Reconfigurable hardware implementation of a phase-correlation stereo algorithm. Machine Vision and Applications 17, 2, 116--132. Google Scholar
Digital Library
- J. Diaz, E. Ros, R. Carrillo, and A. Prieto. 2007. Real-time system for high-image resolution disparity estimation. IEEE Transactions on Image Processing 16, 1, 280--285. Google Scholar
Digital Library
- J. Ding, J. Liu, W. Zhou, H. Yu, Y. Wang, and X. Gong. 2011. Real-time stereo vision system using adaptive weight cost aggregation approach. EURASIP Journal on Image and Video Processing 1, 1--19.Google Scholar
- I. Ernst and H. Hirschmüller. 2008. Mutual information based semi-global stereo matching on the GPU. In Advances in Visual Computing. Lecture Notes in Computer Science, Vol. 5358. Springer, 228--239. Google Scholar
Digital Library
- S. Fahmy, P. Cheung, and W. Luk. 2005. Novel FPGA-based implementation of median and weighted median filters for image processing. In Proceedings of the International Conference on Field Programmable Logic and Applications. 142--147.Google Scholar
- Jianbin Fang, A. L. Varbanescu, Jie Shen, H. Sips, G. Saygili, and L. Van Der Maaten. 2012. Accelerating cost aggregation for real-time stereo matching. In Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems (ICPADS). 472--481. Google Scholar
Digital Library
- S. Foix, G. Alenya, and C. Torras. 2011. Lock-in time-of-flight (ToF) cameras: A survey. IEEE Sensors Journal 11, 9, 1917--1926.Google Scholar
Cross Ref
- S. Forstmann, Y. Kanou, Jun Ohya, S. Thuering, and A. Schmitt. 2004. Real-time stereo by using dynamic programming. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04). 29. Google Scholar
Digital Library
- J. Fowers, G. Brown, P. Cooke, and G. Stitt. 2012. A performance energy comparison of FPGAs, GPUs, and multicores for sliding-window applications. In Proceedings of the International Symposium on Field Programmable Gate Arrays. 47--56. Google Scholar
Digital Library
- B. S. Freedman. 2010. Depth Mapping Using Projected Patterns. Prime Sense Ltd.Google Scholar
- S. Gehrig and C. Rabe. 2010. Real-time semi-global matching on the CPU. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW'10). 85--92.Google Scholar
- S. K. Gehrig, F. Eberli, and T. Meyer. 2009. A real-time low-power stereo vision engine using semi-global matching. In Computer Vision Systems. Lecture Notes in Computer Science, Vol. 5815. Springer, 134--143. Google Scholar
Digital Library
- C. Georgoulas and I. Andreadis. 2009. A real-time occlusion aware hardware structure for disparity map computation. In Image Analysis and Processing—ICIAP 2009. Lecture Notes in Computer Science, Vol. 5716. Springer, 721--730. Google Scholar
Digital Library
- S. Gokturk, H. Yalcin, and C. Bamji. 2004. A time-of-flight depth sensor—system description, issues and solutions. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop, Vol. 3. 35--43. Google Scholar
Digital Library
- A. Gray. 2012. GPU System Architecture. Retrieved November 15, 2012, from http://www2.epcc.ed.ac.uk/ ∼alang/GPU_training_aug12/GPU_Architecture.pdf.Google Scholar
- H. Hirschmüller. 2008. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2, 328--341. Google Scholar
Digital Library
- H. Hirschmüller and D. Scharstein. 2009. Evaluation of stereo matching costs on images with radiometric differences. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 9, 1582--1599. Google Scholar
Digital Library
- H. Hirschmüller, M. Buder, and I. Ernst. 2012. Memory efficient semi-global matching. ISPRS Annals of the Photogrammetry, Remote Sensing, and Spatial Information Sciences I-3, 371--376.Google Scholar
- H. Hirschmüller, P. R. Innocent, and J. Garibaldi. 2002. Real-time correlation-based stereo vision with reduced border errors. International Journal of Computer Vision 47, 1--3, 229--246. Google Scholar
Digital Library
- K.-Y. Hsieh, C.-H. Lai, S.-H. Lai, and J. K. Lee. 2012. Parallelization of belief propagation on cell processors for stereo vision. ACM Transactions on Embedded Computing Systems 11S, 1, Article No. 13. Google Scholar
Digital Library
- M. Humenberger, C. Zinner, M. Weber, W. Kubinger, and M. Vincze. 2010. A fast stereo matching algorithm suitable for embedded real-time systems. Computer Vision and Image Understanding 114, 11, 1180--1202. Google Scholar
Digital Library
- S. Jin, J. Cho, X. D. Pham, K. M. Lee, S.-K. Park, M. Kim, and J. W. Jeon. 2010. FPGA design and implementation of a real-time stereo vision system. IEEE Transactions on Circuits and Systems for Video Technology 20, 1, 15--26. Google Scholar
Digital Library
- B. Khaleghi, S. Ahuja, and Q. Wu. 2008. An improved real-time miniaturized embedded stereo vision system (MESVS-II). In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 1--8.Google Scholar
- K. Khoshelham. 2011. Accuracy analysis of Kinect depth data. In Proceedings of the ISPRS Workshop on Laser Scanning. 133--138.Google Scholar
- K. Konolige. 1997. Small vision system—hardware and implementation. In Proceedings of the International Symposium on Robotics Research. 111--116.Google Scholar
- K. Konolige, M. Agrawal, R. C. Bolles, C. Cowan, M. Fischler, and B. P. Gerkey. 2006. Outdoor mapping and navigation using stereo vision. In Proceedings of the International Symposium on Experimental Robotics (ISER'06). 179--190.Google Scholar
- J. Liu, Y. Xu, R. Klette, H. Chen, W. Rong, and T. Vaudrey. 2009. Disparity map computation on a cell processor. In Proceedings of the IASTED International Conference on Modelling, Simulation, and Identification (MSI'09).Google Scholar
- W. J. MacLean. 2005. An evaluation of the suitability of FPGAs for embedded vision systems. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 131. Google Scholar
Digital Library
- Microsoft Kinect. 2010. Kinect for Xbox One. Retrieved December 15, 2012, from http://www.xbox.com/en-us/kinect/.Google Scholar
- I. H. Moring. 1989. Acquisition of three-dimensional image data by a scanning laser range finder. Optical Engineering 28, 8, 897--902.Google Scholar
Cross Ref
- K. M. Muhlmann. 2001. Calculating dense disparity maps from color stereo images, an efficient implementation. In Proceedings of the IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV'01). 30--36. Google Scholar
Digital Library
- D. H. Park, H. S. Ko, J. G. Kim, and J. D. Cho. 2011. Real time rectification using differentially encoded lookup table. In Proceedings of the 5th International Conference on Ubiquitous Information Management and Communication (ICUIMC'11). 1--4. Google Scholar
Digital Library
- P. Steingrube, S. K. Gehrig, and U. Franke. 2009. Performance evaluation of stereo algorithms for automotive applications. In Computer Vision Systems. Lecture Notes in Computer Science, Vol. 5815. Springer, 285--294. Google Scholar
Digital Library
- H. Sunyoto, W. van der Mark, and D. Gavrila. 2004. A comparative study of fast dense stereo vision algorithms. In Proceedings of the IEEE Intelligent Vehicles Symposium. 319--324.Google Scholar
- D. Szeliski and R. Scharstein. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47, 7--42. Google Scholar
Digital Library
- Tokyo Electron Device Limited. 2010. Kintex-7 FPGA Display Kit. Retrieved September 15, 2012, from http://solutions.inrevium.com/products/kits/consumer/tb-7k-acdc.html.Google Scholar
- F. Tombari, S. Mattoccia, and L. D. Stefano. 2007. Segmentation-based adaptive support for accurate stereo correspondence. In Advances in Image and Video Technology. Lecture Notes in Computer Science, Vol. 4872. Springer, 427--438. Google Scholar
Digital Library
- F. Tombari, S. Mattoccia, L. D. Stefano, and E. Addimanda. 2008. Classification and evaluation of cost aggregation methods for stereo correspondence. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. 1--8.Google Scholar
- C. Ttofis and T. Theocharides. 2012. Towards accurate hardware stereo correspondence: A real-time FPGA implementation of a segmentation-based adaptive support weight algorithm. In Proceedings of the Design, Automation, and Test in Europe Conference and Exhibition (DATE'12). 703--708. Google Scholar
Digital Library
- C. Ttofis, S. Hadjitheophanous, A. Georghiades, and T. Theocharides. 2013. Edge-directed hardware architecture for real-time disparity map computation. IEEE Transactions on Computers 62, 4, 690--704. Google Scholar
Digital Library
- T. Vaudrey, C. Rabe, R. Klette, and J. Milburn. 2008. Differences between stereo and motion behaviour on synthetic and real-world stereo sequences. In Proceedings of the 23rd International Conference on Image and Vision Computing. 1--6.Google Scholar
- L. Wang, M. Liao, M. Gong, R. Yang, and D. Nister. 2006. High-quality real-time stereo using adaptive cost aggregation and dynamic programming. In Proceedings of the 3rd International Symposium on 3D Data Processing, Visualization, and Transmission. 798--805. Google Scholar
Digital Library
- X. Xiang, M. Zhang, G. Li, Y. He, and Z. Pan. 2012. Real-time stereo matching based on fast belief propagation. Machine Vision and Applications 23, 6, 1219--1227.Google Scholar
Cross Ref
- Xilinx Inc. 2012. Kintex-7 FPGA Family. Retrieved September 15, 2012, from http://www.xilinx.com/ products/silicon-devices/fpga/kintex-7/.Google Scholar
- Q. Yang, L. Wang, R. Yang, S. Wang, M. Liao, and D. Nister. 2006. Real-time global stereo matching using hierarchical belief propagation. In Proceedings of the British Machine Vision Conference.Google Scholar
- R. Yang and M. Pollefeys. 2005. A versatile stereo implementation on commodity graphics hardware. Real-Time Imaging 11, 1, 7--18. Google Scholar
Digital Library
- K.-J. Yoon and I.-S. Kweon. 2006. Adaptive support-weight approach for correspondence search. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 4, 650--656. Google Scholar
Digital Library
- C. Zinner, M. Humenberger, K. Ambrosch, and W. Kubinger. 2008. An optimized software-based implementation of a census-based stereo matching algorithm. In Advances in Visual Computing. Lecture Notes in Computer Science, Vol. 5358. Springer, 216--227. Google Scholar
Digital Library
- C. Zinner, M. Humenberger, K. Ambrosch, and W. Kubinger. 2008. An optimized software-based implementation of a census-based stereo matching algorithm. In Proceedings of the 4th International Symposium on Advances in Visual Computing (ISVC'08). 216--227. Google Scholar
Digital Library
Index Terms
A Hardware-Efficient Architecture for Accurate Real-Time Disparity Map Estimation
Recommendations
Hardware Acceleration for an Accurate Stereo Vision System Using Mini-Census Adaptive Support Region
Special Issue on Real-Time and Embedded Technology and Applications, Domain-Specific Multicore Computing, Cross-Layer Dependable Embedded Systems, and Application of Concurrency to System Design (ACSD'13)Domain of stereo vision is highly important in the fields of autonomous cars, video tolling, robotics, and aerial surveys. The specific feature of this domain is that we should handle not only the pixel-by-pixel 2D processing in one image but also the ...
Low-cost FPGA stereo vision system for real time disparity maps calculation
Several applications demand efficient hardware implementations of stereo vision systems in order to furnish real time three-dimensional measurements. This paper proposes a complete fast low-cost stereo vision system that performs stereo image ...
VHDL description of a synthetizable and reconfigurable real-time stereo vision processor
ISPRA'05: Proceedings of the 4th WSEAS International Conference on Signal Processing, Robotics and AutomationThis paper describes a reconfigurable digital architecture to compute dense disparity maps at video-rate for stereo vision. The processor architecture is described in synthetizable VHDL and, by means of the reconfigurability, the hardware requirements ...






Comments