Abstract
Increased mobile autonomy is a vital requisite for future planetary exploration rovers. Stereo vision is a key enabling technology in this regard, as it can passively reconstruct in three dimensions the surroundings of a rover and facilitate the selection of science targets and the planning of safe routes. Nonetheless, accurate dense stereo algorithms are computationally demanding. When executed on the low-performance, radiation-hardened CPUs typically installed on rovers, slow stereo processing severely limits the driving speed and hence the science that can be conducted in situ. Aiming to decrease execution time while increasing the accuracy of stereo vision embedded in future rovers, this article proposes HW/SW co-design and acceleration on resource-constrained, space-grade FPGAs. In a top-down approach, we develop a stereo algorithm based on the space sweep paradigm, design its parallel HW architecture, implement it with VHDL, and demonstrate feasible solutions even on small-sized devices with our multi-FPGA partitioning methodology. To meet all cost, accuracy, and speed requirements set by the European Space Agency for this system, we customize our HW/SW co-processor by design space exploration and testing on a Mars-like dataset. Implemented on Xilinx Virtex technology, or European NG-MEDIUM devices, the FPGA kernel processes a 1,120 × 1,120 stereo pair in 1.7s−3.1s, utilizing only 5.4−9.3 LUT6 and 200−312 RAMB18. The proposed system exhibits up to 32× speedup over desktop CPUs, or 2,810× over space-grade LEON3, and achieves a mean reconstruction error less than 2cm up to 4m depth. Excluding errors exceeding 2cm (which are less than 4% of the total), the mean error is under 8mm.
- Kristian Ambrosch and Wilfried Kubinger. 2010. Accurate hardware-based stereo vision. Comput. Vis. Image Underst. 114, 11 (2010), 1303--1316. Google Scholar
Digital Library
- Max Bajracharya, Mark W. Maimone, and Daniel Helmick. 2008. Autonomy for mars rovers: Past, present, and future. Computer 41, 12 (Dec. 2008), 44--50. Google Scholar
Digital Library
- Christian Banz et al. 2010. Real-time stereo vision system using semi-global matching disparity estimation: Architecture and FPGA-implementation. In Proceedings of the International Conference on Embedded Computer Systems: Architectures, Modeling 8 Simulation. 93--101.Google Scholar
- Jungwook Choi and Rob A. Rutenbar. 2016. Video-rate stereo matching using Markov random field TRW-S inference on a hybrid CPU+FPGA computing platform. IEEE Trans. Circuits Syst. Vid. Technol. 26, 2 (2016), 385--398. Google Scholar
Digital Library
- Giuseppe Cocorullo, Pasquale Corsonello, Fabio Frustaci, and Stefania Perri. 2016. An efficient hardware-oriented stereo matching algorithm. Microprocess. Microsyst. 46, A (2016), 21--33. Google Scholar
Digital Library
- Robert T. Collins. 1996. A space-sweep approach to true multi-image matching. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’96). 358--363. Google Scholar
Digital Library
- Ahmad Darabiha, W. James MacLean, and Jonathan Rose. 2006. Reconfigurable hardware implementation of a phase-correlation stereo algorithm. Mach. Vis. Appl. 17, 2 (May 2006), 116--132. Google Scholar
Digital Library
- Alex Ellery. 2015. Planetary Rovers: Robotic Exploration of the Solar System. Springer, Berlin. Google Scholar
Digital Library
- David Gallup, Jan-Michael Frahm, Philippos Mordohai, Qingxiong Yang, and Marc Pollefeys. 2007. Real-time plane-sweeping stereo with multiple sweeping directions. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’07). 1--8.Google Scholar
Cross Ref
- David Gallup, Jan-Michael Frahm, Philippos Mordohai, and Marc Pollefeys. 2008. Variable baseline/resolution stereo. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’08). 1--8.Google Scholar
Cross Ref
- Pierre Greisen, Simon Heinzle, Markus Gross, and Andreas P. Burg. 2011. An FPGA-based processing pipeline for high-definition stereo video. EURASIP J. Image Vid. Process. 2011, 1 (2011), 18.Google Scholar
Cross Ref
- Heiko Hirschmüller and Daniel Scharstein. 2009. Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell. 31, 9 (2009), 1582--1599. Google Scholar
Digital Library
- Minxi Jin and Tsutomu Maruyama. 2014. Fast and accurate stereo vision system on FPGA. ACM Trans. Reconfig. Technol. Syst. 7, 1 (Feb. 2014), 1--24. Google Scholar
Digital Library
- Seunghun Jin et al. 2010. FPGA design and implementation of a real-time stereo vision system. IEEE Trans. Circ. Syst. Vid. Technol. 20, 1 (Jan. 2010), 15--26. Google Scholar
Digital Library
- Ioannis Kostavelis et al. 2014. SPARTAN: Developing a vision system for future autonomous space exploration robots. J. Field Robot. 31, 1 (2014), 107--140.Google Scholar
Cross Ref
- Joel Le Mauff. 2018. From eFPGA cores to RHBD System-On-Chip FPGA (NanoXplore’s presentation of the NG-MEDIUM rad-hard FPGA). Retrieved from https://indico.esa.int/event/232/contributions/2137/attachments/1820/2121/2018-04_NX-From_eFPGA_cores_to_RHBH_SoC_FPGAs-JLM-v2.pdf.Google Scholar
- Jaedon Lee, Youngsam Shin, Won-Jong Lee, Soojung Ryu, and Jeongwook Kim. 2013. Real-time ray tracing on coarse-grained reconfigurable processor. In Proceedings of the IEEE International Conference on Field-Programmable Technology (FPT’13). 192--197.Google Scholar
Cross Ref
- George Lentaris, Dionysios Diamantopoulos, Kostas Siozios, Dimitrios Soudris, and Marcos Avilés Rodrigálvarez. 2012. Hardware implementation of stereo correspondence algorithm for the ExoMars mission. In Proceedings of the IEEE International Conference on Field Programmable Logic and Applications (FPL’12). 667--670.Google Scholar
Cross Ref
- George Lentaris, Konstantinos Maragos, Ioannis Stratakos, Lazaros Papadopoulos, Odysseas Papanikolaou, Dimitrios Soudris, Manolis Lourakis, Xenophon Zabulis, David Gonzalez-Arjona, and Gianluca Furano. 2018. High performance embedded computing in space: Evaluation of platforms for vision-based navigation. J. Aerosp. Inf. Syst. 15, 4 (Apr. 2018), 178--192.Google Scholar
Cross Ref
- George Lentaris, Ioannis Stamoulias, Dimitrios Soudris, and Manolis Lourakis. 2016. HW/SW co-design and FPGA acceleration of visual odometry algorithms for rover navigation on mars. IEEE Trans. Circuits Syst. Vid. Technol. 26, 8 (Aug. 2016), 1563--1577.Google Scholar
Digital Library
- Jing Liu, Chunpeng Li, Feng Mei, and Zhaoqi Wang. 2015. 3D entity-based stereo matching with ground control points and joint second-order smoothness prior. Vis. Comput. 31, 9 (Sept. 2015), 1253--1269. Google Scholar
Digital Library
- Manolis Lourakis and Xenophon Zabulis. 2013. Accurate scale factor estimation in 3D reconstruction. In Proceedings of the International Conference on Computer Analysis of Images and Patterns (CAIP’13). Springer, Berlin, 498--506. Google Scholar
Digital Library
- Mark W. Maimone, P. Chris Leger, and Jeffrey J. Biesiadecki. 2007. Overview of the Mars exploration rovers’ autonomous mobility and vision capabilities. In Proceedings of the International Conference on Robotics and Automation (ICRA’07).Google Scholar
- Larry Matthies et al. 2007. Computer vision on mars. Int. J. Comput. Vis. 75, 1 (2007), 67--92. Google Scholar
Digital Library
- Richard H. Maurer, Martin E. Fraeman, Mark N. Martin, and David R. Roth. 2008. Harsh environments: Space radiation environment, effects, and mitigation. Johns Hopkins APL Techn. Digest 28, 1 (2008), 17--29.Google Scholar
- Hans P. Moravec. 1977. Towards automatic visual obstacle avoidance. In Proceedings of the International Joint Conference on AI (IJCAI’77). 584--594. Google Scholar
Digital Library
- Mikhail G. Mozerov and Joost van de Weijer. 2015. Accurate stereo matching by two-step energy minimization. IEEE Trans. Image Process. 24, 3 (Mar. 2015), 1153--1163.Google Scholar
Digital Library
- Kyprianos Papadimitriou, Sotiris Thomas, and Apostolos Dollas. 2013. An FPGA-based real-time system for 3D stereo matching, combining absolute differences and census with aggregation and belief propagation. In Proceedings of the IFIP/IEEE International Conference on VLSI-SoC. Springer, 168--187.Google Scholar
- Madaín Pérez-Patricio and Abiel Aguilar-González. 2015. FPGA implementation of an efficient similarity-based adaptive window algorithm for real-time stereo matching. J. Real-Time Image Process. (Sept. 2015), 1–17.Google Scholar
- Marc Pollefeys et al. 2008. Detailed real-time urban 3D reconstruction from video. Int. J. Comput. Vision 78, 2--3 (Jul. 2008), 143--167. Google Scholar
Digital Library
- Marc Pollefeys and Sudipta Sinha. 2004. Iso-disparity surfaces for general stereo configurations. In Proceedings of the European Conference on Computer Vision ECCV, Vol. III. Springer, 509--520.Google Scholar
Cross Ref
- Kyle Rupnow, Yun Liang, Yinan Li, Dongbo Min, Minh Do, and Deming Chen. 2011. High level synthesis of stereo matching: Productivity, performance, and software constraints. In Proceedings of the 2011 International Conference on Field-Programmable Technology (FPT’11). IEEE, 1--8.Google Scholar
Cross Ref
- Daniel Scharstein and Richard Szeliski. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 1-3 (Apr. 2002), 7--42. Google Scholar
Digital Library
- Daniel Scharstein and Richard Szeliski. 2015. Middlebury Stereo Evaluation, Version 2. Retrieved November 21, 2017 from http://vision.middlebury.edu/stereo/eval/.Google Scholar
- Thomas Schöps, Torsten Sattler, Christian Häne, and Marc Pollefeys. 2017. Large-scale outdoor 3D reconstruction on a mobile device. Comput. Vis. Image Underst. 157 (2017), 151--166. Google Scholar
Digital Library
- Yi Shan et al. 2014. Hardware acceleration for an accurate stereo vision system using mini-census adaptive support region. ACM Trans. Embed. Comput. Syst. 13, 4 (Apr. 2014), 132:1--132:24. Google Scholar
Digital Library
- Synopsys. 2017. HAPS Family of Physical Prototyping Solutions. Retrieved July 10, 2010 from https://www.synopsys.com/verification/prototyping/haps.html.Google Scholar
- Matteo Tomasi, Mauricio Vanegas, Francisco Barranco, Javier Diaz, and Eduardo Ros. 2012. Real-time architecture for a robust multi-scale stereo engine on FPGA. IEEE Trans. VLSI Syst. 20, 12 (Dec. 2012), 2208--2219. Google Scholar
Digital Library
- Christos Ttofis, Christos Kyrkou, and Theocharis Theocharides. 2015. A hardware-efficient architecture for accurate real-time disparity map estimation. ACM Trans. Embed. Comput. Syst. 14, 2 (Feb. 2015), 36:1--36:26. Google Scholar
Digital Library
- George Vogiatzis, Carlos Hernández Esteban, Philip H. S. Torr, and Roberto Cipolla. 2007. Multiview stereo via volumetric graph-cuts and occlusion robust photo-consistency. IEEE Trans. Pattern Anal. Mach. Intell. 29, 12 (Dec. 2007), 2241--2246. Google Scholar
Digital Library
- Wenqiang Wang, Jing Yan, Ningyi Xu, Yu Wang, and Feng-Hsiung Hsu. 2015. Real-time high-quality stereo vision system in FPGA. IEEE Trans. Circuits Syst. Video Technol. 25, 10 (Oct. 2015), 1696--1708.Google Scholar
Digital Library
- Qingxiong Yang et al. 2009. Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling. IEEE Trans. Pattern Anal. Mach. Intell. 31, 3 (Mar. 2009), 492--504. Google Scholar
Digital Library
- Qingqing Yang, Pan Ji, Dongxiao Li, Shaojun Yao, and Ming Zhang. 2014. Fast stereo matching using adaptive guided filtering. Image Vis. Comput. 32, 3 (Mar. 2014), 202--211. Google Scholar
Digital Library
- Ruigang Yang and Marc Pollefeys. 2003. Multi-resolution real-time stereo on commodity graphics hardware. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’03). 211--217. Google Scholar
Digital Library
- Kuk-Jin Yoon and In So Kweon. 2006. Adaptive support-weight approach for correspondence search. IEEE Trans. Pattern Anal. Mach. Intell. 28, 4 (Apr. 2006), 650--656. Google Scholar
Digital Library
- Xenophon Zabulis, Georgios Kordelas, Karsten Müller, and Aljoscha Smolic. 2006. Increasing the accuracy of the space-sweeping approach to stereo reconstruction, using spherical backprojection surfaces. In Proceedings of the International Conference on Image Processing (ICIP’06). 2965--2968.Google Scholar
Cross Ref
- Yunlong Zhan, Yuzhang Gu, Kui Huang, Cheng Zhang, and Keli Hu. 2016. Accurate image-guided stereo matching with efficient matching cost and disparity refinement. IEEE Trans. Circ. Syst. Video Technol. 26, 9 (Sept. 2016), 1632--1645. Google Scholar
Digital Library
- Paolo Zicari, Stefania Perri, Pasquale Corsonello, and Giuseppe Cocorullo. 2012. Low-cost FPGA stereo vision system for real time disparity maps calculation. Microprocess. Microsyst. 36, 4 (2012), 281--288. Google Scholar
Digital Library
Index Terms
Single- and Multi-FPGA Acceleration of Dense Stereo Vision for Planetary Rovers
Recommendations
FPGA based real-time on-road stereo vision system
Accurate and Real-Time stereo vision is an essential need for many computer vision applications, such as On-Road stereo vision system. In this paper, a fast and accurate system for On-Road stereo vision application is presented. In order to achieve this ...
Low-cost FPGA stereo vision system for real time disparity maps calculation
Several applications demand efficient hardware implementations of stereo vision systems in order to furnish real time three-dimensional measurements. This paper proposes a complete fast low-cost stereo vision system that performs stereo image ...
FPGA Design and Implementation of a Real-Time Stereo Vision System
Stereo vision is a well-known ranging method because it resembles the basic mechanism of the human eye. However, the computational complexity and large amount of data access make real-time processing of stereo vision challenging because of the inherent ...






Comments