skip to main content
research-article

Single- and Multi-FPGA Acceleration of Dense Stereo Vision for Planetary Rovers

Authors Info & Claims
Published:18 March 2019Publication History
Skip Abstract Section

Abstract

Increased mobile autonomy is a vital requisite for future planetary exploration rovers. Stereo vision is a key enabling technology in this regard, as it can passively reconstruct in three dimensions the surroundings of a rover and facilitate the selection of science targets and the planning of safe routes. Nonetheless, accurate dense stereo algorithms are computationally demanding. When executed on the low-performance, radiation-hardened CPUs typically installed on rovers, slow stereo processing severely limits the driving speed and hence the science that can be conducted in situ. Aiming to decrease execution time while increasing the accuracy of stereo vision embedded in future rovers, this article proposes HW/SW co-design and acceleration on resource-constrained, space-grade FPGAs. In a top-down approach, we develop a stereo algorithm based on the space sweep paradigm, design its parallel HW architecture, implement it with VHDL, and demonstrate feasible solutions even on small-sized devices with our multi-FPGA partitioning methodology. To meet all cost, accuracy, and speed requirements set by the European Space Agency for this system, we customize our HW/SW co-processor by design space exploration and testing on a Mars-like dataset. Implemented on Xilinx Virtex technology, or European NG-MEDIUM devices, the FPGA kernel processes a 1,120 × 1,120 stereo pair in 1.7s−3.1s, utilizing only 5.4−9.3 LUT6 and 200−312 RAMB18. The proposed system exhibits up to 32× speedup over desktop CPUs, or 2,810× over space-grade LEON3, and achieves a mean reconstruction error less than 2cm up to 4m depth. Excluding errors exceeding 2cm (which are less than 4% of the total), the mean error is under 8mm.

References

  1. Kristian Ambrosch and Wilfried Kubinger. 2010. Accurate hardware-based stereo vision. Comput. Vis. Image Underst. 114, 11 (2010), 1303--1316. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Max Bajracharya, Mark W. Maimone, and Daniel Helmick. 2008. Autonomy for mars rovers: Past, present, and future. Computer 41, 12 (Dec. 2008), 44--50. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Christian Banz et al. 2010. Real-time stereo vision system using semi-global matching disparity estimation: Architecture and FPGA-implementation. In Proceedings of the International Conference on Embedded Computer Systems: Architectures, Modeling 8 Simulation. 93--101.Google ScholarGoogle Scholar
  4. Jungwook Choi and Rob A. Rutenbar. 2016. Video-rate stereo matching using Markov random field TRW-S inference on a hybrid CPU+FPGA computing platform. IEEE Trans. Circuits Syst. Vid. Technol. 26, 2 (2016), 385--398. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Giuseppe Cocorullo, Pasquale Corsonello, Fabio Frustaci, and Stefania Perri. 2016. An efficient hardware-oriented stereo matching algorithm. Microprocess. Microsyst. 46, A (2016), 21--33. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Robert T. Collins. 1996. A space-sweep approach to true multi-image matching. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’96). 358--363. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Ahmad Darabiha, W. James MacLean, and Jonathan Rose. 2006. Reconfigurable hardware implementation of a phase-correlation stereo algorithm. Mach. Vis. Appl. 17, 2 (May 2006), 116--132. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Alex Ellery. 2015. Planetary Rovers: Robotic Exploration of the Solar System. Springer, Berlin. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. David Gallup, Jan-Michael Frahm, Philippos Mordohai, Qingxiong Yang, and Marc Pollefeys. 2007. Real-time plane-sweeping stereo with multiple sweeping directions. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’07). 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  10. David Gallup, Jan-Michael Frahm, Philippos Mordohai, and Marc Pollefeys. 2008. Variable baseline/resolution stereo. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’08). 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  11. Pierre Greisen, Simon Heinzle, Markus Gross, and Andreas P. Burg. 2011. An FPGA-based processing pipeline for high-definition stereo video. EURASIP J. Image Vid. Process. 2011, 1 (2011), 18.Google ScholarGoogle ScholarCross RefCross Ref
  12. Heiko Hirschmüller and Daniel Scharstein. 2009. Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell. 31, 9 (2009), 1582--1599. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Minxi Jin and Tsutomu Maruyama. 2014. Fast and accurate stereo vision system on FPGA. ACM Trans. Reconfig. Technol. Syst. 7, 1 (Feb. 2014), 1--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Seunghun Jin et al. 2010. FPGA design and implementation of a real-time stereo vision system. IEEE Trans. Circ. Syst. Vid. Technol. 20, 1 (Jan. 2010), 15--26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Ioannis Kostavelis et al. 2014. SPARTAN: Developing a vision system for future autonomous space exploration robots. J. Field Robot. 31, 1 (2014), 107--140.Google ScholarGoogle ScholarCross RefCross Ref
  16. Joel Le Mauff. 2018. From eFPGA cores to RHBD System-On-Chip FPGA (NanoXplore’s presentation of the NG-MEDIUM rad-hard FPGA). Retrieved from https://indico.esa.int/event/232/contributions/2137/attachments/1820/2121/2018-04_NX-From_eFPGA_cores_to_RHBH_SoC_FPGAs-JLM-v2.pdf.Google ScholarGoogle Scholar
  17. Jaedon Lee, Youngsam Shin, Won-Jong Lee, Soojung Ryu, and Jeongwook Kim. 2013. Real-time ray tracing on coarse-grained reconfigurable processor. In Proceedings of the IEEE International Conference on Field-Programmable Technology (FPT’13). 192--197.Google ScholarGoogle ScholarCross RefCross Ref
  18. George Lentaris, Dionysios Diamantopoulos, Kostas Siozios, Dimitrios Soudris, and Marcos Avilés Rodrigálvarez. 2012. Hardware implementation of stereo correspondence algorithm for the ExoMars mission. In Proceedings of the IEEE International Conference on Field Programmable Logic and Applications (FPL’12). 667--670.Google ScholarGoogle ScholarCross RefCross Ref
  19. George Lentaris, Konstantinos Maragos, Ioannis Stratakos, Lazaros Papadopoulos, Odysseas Papanikolaou, Dimitrios Soudris, Manolis Lourakis, Xenophon Zabulis, David Gonzalez-Arjona, and Gianluca Furano. 2018. High performance embedded computing in space: Evaluation of platforms for vision-based navigation. J. Aerosp. Inf. Syst. 15, 4 (Apr. 2018), 178--192.Google ScholarGoogle ScholarCross RefCross Ref
  20. George Lentaris, Ioannis Stamoulias, Dimitrios Soudris, and Manolis Lourakis. 2016. HW/SW co-design and FPGA acceleration of visual odometry algorithms for rover navigation on mars. IEEE Trans. Circuits Syst. Vid. Technol. 26, 8 (Aug. 2016), 1563--1577.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Jing Liu, Chunpeng Li, Feng Mei, and Zhaoqi Wang. 2015. 3D entity-based stereo matching with ground control points and joint second-order smoothness prior. Vis. Comput. 31, 9 (Sept. 2015), 1253--1269. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Manolis Lourakis and Xenophon Zabulis. 2013. Accurate scale factor estimation in 3D reconstruction. In Proceedings of the International Conference on Computer Analysis of Images and Patterns (CAIP’13). Springer, Berlin, 498--506. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Mark W. Maimone, P. Chris Leger, and Jeffrey J. Biesiadecki. 2007. Overview of the Mars exploration rovers’ autonomous mobility and vision capabilities. In Proceedings of the International Conference on Robotics and Automation (ICRA’07).Google ScholarGoogle Scholar
  24. Larry Matthies et al. 2007. Computer vision on mars. Int. J. Comput. Vis. 75, 1 (2007), 67--92. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Richard H. Maurer, Martin E. Fraeman, Mark N. Martin, and David R. Roth. 2008. Harsh environments: Space radiation environment, effects, and mitigation. Johns Hopkins APL Techn. Digest 28, 1 (2008), 17--29.Google ScholarGoogle Scholar
  26. Hans P. Moravec. 1977. Towards automatic visual obstacle avoidance. In Proceedings of the International Joint Conference on AI (IJCAI’77). 584--594. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Mikhail G. Mozerov and Joost van de Weijer. 2015. Accurate stereo matching by two-step energy minimization. IEEE Trans. Image Process. 24, 3 (Mar. 2015), 1153--1163.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Kyprianos Papadimitriou, Sotiris Thomas, and Apostolos Dollas. 2013. An FPGA-based real-time system for 3D stereo matching, combining absolute differences and census with aggregation and belief propagation. In Proceedings of the IFIP/IEEE International Conference on VLSI-SoC. Springer, 168--187.Google ScholarGoogle Scholar
  29. Madaín Pérez-Patricio and Abiel Aguilar-González. 2015. FPGA implementation of an efficient similarity-based adaptive window algorithm for real-time stereo matching. J. Real-Time Image Process. (Sept. 2015), 1–17.Google ScholarGoogle Scholar
  30. Marc Pollefeys et al. 2008. Detailed real-time urban 3D reconstruction from video. Int. J. Comput. Vision 78, 2--3 (Jul. 2008), 143--167. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Marc Pollefeys and Sudipta Sinha. 2004. Iso-disparity surfaces for general stereo configurations. In Proceedings of the European Conference on Computer Vision ECCV, Vol. III. Springer, 509--520.Google ScholarGoogle ScholarCross RefCross Ref
  32. Kyle Rupnow, Yun Liang, Yinan Li, Dongbo Min, Minh Do, and Deming Chen. 2011. High level synthesis of stereo matching: Productivity, performance, and software constraints. In Proceedings of the 2011 International Conference on Field-Programmable Technology (FPT’11). IEEE, 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  33. Daniel Scharstein and Richard Szeliski. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 1-3 (Apr. 2002), 7--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Daniel Scharstein and Richard Szeliski. 2015. Middlebury Stereo Evaluation, Version 2. Retrieved November 21, 2017 from http://vision.middlebury.edu/stereo/eval/.Google ScholarGoogle Scholar
  35. Thomas Schöps, Torsten Sattler, Christian Häne, and Marc Pollefeys. 2017. Large-scale outdoor 3D reconstruction on a mobile device. Comput. Vis. Image Underst. 157 (2017), 151--166. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Yi Shan et al. 2014. Hardware acceleration for an accurate stereo vision system using mini-census adaptive support region. ACM Trans. Embed. Comput. Syst. 13, 4 (Apr. 2014), 132:1--132:24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Synopsys. 2017. HAPS Family of Physical Prototyping Solutions. Retrieved July 10, 2010 from https://www.synopsys.com/verification/prototyping/haps.html.Google ScholarGoogle Scholar
  38. Matteo Tomasi, Mauricio Vanegas, Francisco Barranco, Javier Diaz, and Eduardo Ros. 2012. Real-time architecture for a robust multi-scale stereo engine on FPGA. IEEE Trans. VLSI Syst. 20, 12 (Dec. 2012), 2208--2219. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Christos Ttofis, Christos Kyrkou, and Theocharis Theocharides. 2015. A hardware-efficient architecture for accurate real-time disparity map estimation. ACM Trans. Embed. Comput. Syst. 14, 2 (Feb. 2015), 36:1--36:26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. George Vogiatzis, Carlos Hernández Esteban, Philip H. S. Torr, and Roberto Cipolla. 2007. Multiview stereo via volumetric graph-cuts and occlusion robust photo-consistency. IEEE Trans. Pattern Anal. Mach. Intell. 29, 12 (Dec. 2007), 2241--2246. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Wenqiang Wang, Jing Yan, Ningyi Xu, Yu Wang, and Feng-Hsiung Hsu. 2015. Real-time high-quality stereo vision system in FPGA. IEEE Trans. Circuits Syst. Video Technol. 25, 10 (Oct. 2015), 1696--1708.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Qingxiong Yang et al. 2009. Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling. IEEE Trans. Pattern Anal. Mach. Intell. 31, 3 (Mar. 2009), 492--504. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Qingqing Yang, Pan Ji, Dongxiao Li, Shaojun Yao, and Ming Zhang. 2014. Fast stereo matching using adaptive guided filtering. Image Vis. Comput. 32, 3 (Mar. 2014), 202--211. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Ruigang Yang and Marc Pollefeys. 2003. Multi-resolution real-time stereo on commodity graphics hardware. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’03). 211--217. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Kuk-Jin Yoon and In So Kweon. 2006. Adaptive support-weight approach for correspondence search. IEEE Trans. Pattern Anal. Mach. Intell. 28, 4 (Apr. 2006), 650--656. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Xenophon Zabulis, Georgios Kordelas, Karsten Müller, and Aljoscha Smolic. 2006. Increasing the accuracy of the space-sweeping approach to stereo reconstruction, using spherical backprojection surfaces. In Proceedings of the International Conference on Image Processing (ICIP’06). 2965--2968.Google ScholarGoogle ScholarCross RefCross Ref
  47. Yunlong Zhan, Yuzhang Gu, Kui Huang, Cheng Zhang, and Keli Hu. 2016. Accurate image-guided stereo matching with efficient matching cost and disparity refinement. IEEE Trans. Circ. Syst. Video Technol. 26, 9 (Sept. 2016), 1632--1645. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Paolo Zicari, Stefania Perri, Pasquale Corsonello, and Giuseppe Cocorullo. 2012. Low-cost FPGA stereo vision system for real time disparity maps calculation. Microprocess. Microsyst. 36, 4 (2012), 281--288. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Single- and Multi-FPGA Acceleration of Dense Stereo Vision for Planetary Rovers

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!