skip to main content
research-article

Sparse LIDAR Measurement Fusion with Joint Updating Cost for Fast Stereo Matching

Authors Info & Claims
Published:27 January 2022Publication History
Skip Abstract Section

Abstract

The complementary virtues of active and passive depth sensors inspire the LIDAR-Stereo fusion for enhancing the accuracy of stereo matching. However, most of the fusion based stereo matching algorithms have exploited dense LIDAR priors with single fusion methodology. In this paper, we intend to break these fetters, utilizing sparse LIDAR priors with multi-step fusion strategy for obtaining accurate disparity estimation more efficiently. At first, random sparse sampling LIDAR depth measurements are provided in Naive Fusion for updating the matching cost of Semi-Global Matching (SGM). Then Neighborhood Based Fusion is performed based on the former step for further updating the cost. Subsequently, Diffusion Based Fusion is utilized to update both the cost and disparities. At last, Tree Filtering is applied for removing speckle outliers and smoothing disparities. Performance evaluations on various stereo data sets demonstrate that the proposed algorithm outperforms other most challenging stereo matching algorithms significantly with approximately real-time implementation efficiency. Furthermore, it is worth pointing out that our proposal surprisingly possesses one of the top ten performances on Middlebury v.3 online evaluation system even if it has not been adopted any learning-based techniques.

REFERENCES

  1. [1] Scharstein Daniel and Szeliski Richard. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In International Journal of Computer Vision (IJCV), Vol. 47. Springer, 742. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Minglun Gong, Ruigang Yang, Liang Wang, and Mingwei Gong.2007. A performance study on different cost aggregation approaches used in real-time stereo matching. In International Journal of Computer Vision (IJCV), Vol. 75. Springer, 283296. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Tombari Federico, Mattoccia Stefano, Di Stefano Luigi, and Addimanda Elisa.2008. Classification and evaluation of cost aggregation methods for stereo correspondence. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). IEEE, 18.Google ScholarGoogle Scholar
  4. [4] Kuk-Jin Yoon and So Kweon In.2006. Adaptive support-weight approach for correspondence search. In IEEE Transactions on Pattern Analysis (TPAMI), Vol. 28. IEEE, 650656. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Dongbo Min, Jiangbo Lu, and Minh N Do.2013. Joint histogram based cost aggregation for stereo matching. In IEEE Transactions on Pattern Analysis (TPAMI), Vol. 35. IEEE, 25392545. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. [6] Bleyer Michael and Breiteneder Christian.2013. Stereo matching-state-of-the-art and research challenges. In Chapter 6 of Advanced Topics in Computer Vision. Springer, 143179.Google ScholarGoogle Scholar
  7. [7] Qingxiong Yang.2012. A non-local cost aggregation method for stereo matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 14021409. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Qingqing Yang.2015. Local smoothness enforced cost volume regularization for fast stereo correspondence. In IEEE Signal Processing Letters (SPL), Vol. 22. IEEE, 14291433.Google ScholarGoogle Scholar
  9. [9] Besse Frederic, Rother Carsten, Fitzgibbon Andrew, and Kautz Jan.2012. PMBP: PatchMatch belief propagation for correspondence field estimation. In British Machine Vision Conference (BMVC’12). BMVA, 111.Google ScholarGoogle Scholar
  10. [10] Chi Zhang, Zhiwei Li, Rui Cai, Hongyang Chao, and Yong Rui.2014. As-rigid-as-possible stereo under second order smoothness priors. In European Conference on Computer Vision (ECCV’14). Springer, 112126.Google ScholarGoogle Scholar
  11. [11] Tatsunori Taniai, Yasuyuki Matsushita, and Takeshi Naemura.2014. Graph cut based continuous stereo matching using locally shared labels. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). IEEE, 16131620. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Tatsunori Taniai, Yasuyuki Matsushita, and Takeshi Naemura.2018. Continuous 3D label stereo matching using local expansion moves. In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Vol. 40. IEEE, 27252739.Google ScholarGoogle Scholar
  13. [13] Peng Yao, Hua Zhang, Yanbing Xue, and Shengyong Chen.2018. AGO: Accelerating global optimization for accurate stereo matching. In International Conference on Multimedia Modeling (MMM’18). Springer, 6780.Google ScholarGoogle Scholar
  14. [14] Peng Yao, Hua Zhang, Yanbing Xue, and Shengyong Chen.2018. MSCS: MeshStereo with cross-scale cost filtering for fast stereo matching. In IET Computer Vision (IET-CVI), Vol. 12. IET, 908918.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Hirschmuller Heiko.2005. Accurate and efficient stereo processing by semi-global matching and mutual information. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’05). IEEE, 807814. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Hirschmuller Heiko.2008. Stereo processing by semi-global matching and mutual information. In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Vol. 30. IEEE, 328341. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Facciolo Gabriele, de Franchis Carlo, and Meinhardt Enric.2015. MGM: A significantly more global matching for stereo vision. In British Machine Vision Conference (BMVC’15). BMVA, 112.Google ScholarGoogle Scholar
  18. [18] Peng Yao, Hua Zhang, Yanbing Xue, and Shengyong Chen.2019. As-global-as-possible stereo matching with adaptive smoothness prior. In IET Image Processing (IET-IPR), Vol. 13. IET, 98107.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Zbontar Jure and LeCun Yann.2015. Computing the stereo matching cost with a convolutional neural network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). IEEE, 15921599.Google ScholarGoogle Scholar
  20. [20] Zbontar Jure and LeCun Yann.2016. Stereo matching by training a convolutional neural network to compare image patches. In Journal of Machine Learning Research (JMLR), Vol. 17. MIT Press, 132. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Seki Akihito and Pollefeys Marc.2016. Patch based confidence prediction for dense disparity map. In British Machine Vision Conference (BMVC’16). BMVA, 113.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Seki Akihito and Pollefeys Marc.2017. SGM-nets: Semi-global matching with neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 66406649.Google ScholarGoogle Scholar
  23. [23] Gidaris Spyros and Komodakis Nikos.2017. Detect, replace, refine: Deep structured prediction for pixel wise labeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 71877196.Google ScholarGoogle Scholar
  24. [24] Batsos Konstantinos and Mordohai. Philippos2018. RecResNet: A recurrent residual CNN architecture for disparity map enhancement. In IEEE International Conference on 3D Vision (3DV’18). IEEE, 238247.Google ScholarGoogle Scholar
  25. [25] Mayer Nikolaus, Ilg Eddy, Hausser Philip, and Fischer Philipp.2016. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). IEEE, 40404048.Google ScholarGoogle Scholar
  26. [26] Jiahao Pang, Wenxiu Sun, S. J. Ren Jimmy, Chengxi Yang, and Qiong Yan.2017. Cascade residual learning: A two-stage convolutional neural network for stereo matching. In IEEE International Conference on Computer Vision (ICCV’17). IEEE, 878886.Google ScholarGoogle Scholar
  27. [27] Zhengfa Liang, Yiliu Feng, Yulan Guo, Hengzhu Liu, Wei Chen, Linbo Qiao, Li Zhou, and Jianfeng Zhang.2018. Learning for disparity estimation through feature constancy. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). IEEE, 28112820.Google ScholarGoogle Scholar
  28. [28] Kendall Alex, Martirosyan Hayk, Dasgupta Saumitro, Henry Peter, Kennedy Ryan, Bachrach Abraham, Bry Adam, and Research Skydio.2017. End-to-end learning of geometry and context for deep stereo regression. In IEEE International Conference on Computer Vision (ICCV’17). IEEE, 6675.Google ScholarGoogle Scholar
  29. [29] Jiejie Zhu, Liang Wang, Ruigang Yang, and James Davis.2008. Fusion of time-of-flight depth and stereo for high accuracy depth maps. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). IEEE, 18.Google ScholarGoogle Scholar
  30. [30] Vineet Gandhi, Cech Jan, and Radu Horaud.2012. High-resolution depth maps based on ToF-stereo fusion. In IEEE International Conference on Robotics and Automation (ICRA’12). IEEE, 47424749.Google ScholarGoogle Scholar
  31. [31] Hernnn Badino, Daniel F. Huber, and Takeo Kanade.2011. Integrating LIDAR into stereo for fast and improved disparity computation. In IEEE International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT’11). IEEE, 405412. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Maddern Will and Newman Paul M..2016. Real-time probabilistic fusion of sparse 3D LIDAR and dense stereo. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’16). IEEE, 21812188.Google ScholarGoogle Scholar
  33. [33] Poggi Matteo, Pallotti Davide, Tosi Fabio, and Mattoccia Stefano.2019. Guided stereo matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19). IEEE, 979988.Google ScholarGoogle Scholar
  34. [34] Xuelian Cheng, Yiran Zhong, Yuchao Dai, Pan Ji, and Hongdong Li.2019. Noise-aware unsupervised deep LIDAR-Stereo fusion. In IEEE Computer Vision and Pattern Recognition (CVPR’19). IEEE, 63396348.Google ScholarGoogle Scholar
  35. [35] Fischer Jan, Arbeiter Georg, and Verl Alexander.2011. Combination of time-of-flight depth and stereo using semi-global optimization. In IEEE International Conference on Robotics and Automation (ICRA’11). IEEE, 35483553.Google ScholarGoogle Scholar
  36. [36] Shivakumar Shreyas S., Mohta Kartik, Pfrommer Bernd, Kumar Vijay, and Taylor Camillo J..2019. Real time dense depth estimation by fusing stereo with sparse depth measurements. In IEEE International Conference on Robotics and Automation (ICRA’19). IEEE, pp. 64826488.Google ScholarGoogle Scholar
  37. [37] Batsos Konstantinos, Cai Changjiang, and Mordohai. Philippos2018. CBMV: A coalesced bidirectional matching volume for disparity estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). IEEE, 20602069.Google ScholarGoogle Scholar
  38. [38] Scharstein Daniel and Pal Chris.2007. Learning conditional random fields for stereo. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’07). IEEE, 18.Google ScholarGoogle Scholar
  39. [39] Hirschmuller Heiko and Scharstein Daniel.2007. Evaluation of cost functions for stereo matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’07). IEEE, 18.Google ScholarGoogle Scholar
  40. [40] Drory Amnon, Haubold Carsten, Avidan Shai, and Hamprecht Fred A..2014. Semi-global matching: A principled derivation in terms of message passing. In German Conference of Pattern Recognition (GCPR’14). Springer, 4353.Google ScholarGoogle Scholar
  41. [41] Haeusler Ralf, Nair Rahul, and Kondermann Daniel.2013. Ensemble learning for confidence measures in stereo vision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’13). IEEE, 305312. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Spyropoulos Aristotle, Komodakis Nikos, and Mordohai Philippos.2014. Learning to detect ground control points for improving the accuracy of stereo matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). IEEE, 16211628. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. [43] Spyropoulos Aristotle, Komodakis Nikos, and Mordohai Philippos.2016. Correctness prediction, accuracy improvement and generalization of stereo matching using supervised learning. In International Journal of Computer Vision (IJCV), Vol. 118. Springer, 300318. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. [44] Min-Gyu Park and Kuk-Jin Yoon.2015. Leveraging stereo matching with learning-based confidence measures. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). IEEE, 101109.Google ScholarGoogle Scholar
  45. [45] Poggi Matteo and Mattoccia Stefano.2016. Learning a general-purpose confidence measure based on O(1) features and a smarter aggregation strategy for semi global matching. In IEEE International Conference on 3D Vision (3DV’16). IEEE, 509518.Google ScholarGoogle Scholar
  46. [46] Peng Yao and Jieqing Feng.2021. Ensemble learning with advanced fast image filtering features for semi-global matching. In Machine Vision and Applications (MVA), Vol. 32. Springer, 115.Google ScholarGoogle Scholar
  47. [47] Jihao Yin, Hongmei Zhu, Ding Yuan, and Tianfan Xue.2017. Sparse representation over discriminative dictionary for stereo matching. In Pattern Recognition (PR), Vol. 71. Elsevier, 278289.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Geiger Andreas, Roser Martin, and Urtasun Raquel.2010. Efficient large-scale stereo matching. In Asian Conference on Computer Vision (ACCV’10). Springer, 2538. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. [49] Shengman Zhang, Jie Shan, Zhichao Zhang, Jixing Yan, and Yaolin Hou.2016. Integrating smartphone images and airborne LIDAR data for complete urban building modelling. In International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS’16). ISPRS, 741747.Google ScholarGoogle Scholar
  50. [50] Tingting Cui, Shunping Ji, Jie Shan, Jianya Gong, and Kejian Liu.2017. Line-based registration of panoramic images and LiDAR point clouds for mobile mapping. In Sensors (Switzerland), Vol. 1. MDPI, 120.Google ScholarGoogle Scholar
  51. [51] Mengdie Zhou, Zhizhong Kang, Zhen Wang, and Min Kong.2020. Airborne LIDAR point cloud classification fusion with DIM point cloud. In International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS’20), Vol. 43, No. B2. ISPRS, 375382.Google ScholarGoogle Scholar
  52. [52] Zheng Gong, Haojia Lin, Zhan Dedong, Zhipeng Luo, John Zelek, Yiping Chen, Abdul Nurunnabi, Cheng Wang, and Jonathan Li.2020. A frustum-based probabilistic framework for 3D object detection by fusion of LiDAR and camera data. In ISPRS Journal of Photogrammetry and Remote Sensing (ISPRS-JPRS), Vol. 159. ISPRS, 90100.Google ScholarGoogle ScholarCross RefCross Ref
  53. [53] Shanxiong Chen, Wenzhong Shi, Mingting Zhou, Min Zhang, and Pengfei Chen.2020. Automatic building extraction via adaptive iterative segmentation with LiDAR data and high spatial resolution imagery fusion. In IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (JST-AEORS), Vol. 13. IEEE, 20812095.Google ScholarGoogle ScholarCross RefCross Ref
  54. [54] Premebida Cristiano, Garrote Luis, Asvadi Alireza, Ribeiro A. Pedro, and Urbano Nunes.2016. High-Resolution LIDAR-Based depth mapping using bilateral filter. In IEEE Intelligent Transportation Systems (ITSC’16). IEEE, 24692474.Google ScholarGoogle Scholar
  55. [55] Zabih Ramin and Woodfill John.1994. Non-parametric local transforms for computing visual correspondence. In European Conference on Computer Vision (ECCV’94). Springer, 151158. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. [56] Perona Pietro and Malik Jitendra.1990. Scale-space and edge detection using anisotropic diffusion. In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Vol. 12. IEEE, 629639. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. [57] Qingxiong Yang.2015. Stereo matching using tree filtering. In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Vol. 37. IEEE, 834846.Google ScholarGoogle Scholar
  58. [58] Scharstein Daniel, Hirschmuller Heiko, Kitajima York, Krathwohl Greg, Nesic Nera, Wang Xi, and Westling Porter.2014. High-resolution stereo datasets withsubpixel-accurate ground truth. In German Conference on Pattern Recognition (GCPR’14). Springer, 3142.Google ScholarGoogle Scholar
  59. [59] Geiger Andreas, Lenz Philip, and Urtasun Raquel.2012. Are we ready for autonomous driving? The KITTI Vision Benchmark Suite. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. [60] Menze Moritz, Heipke Christian, and Geiger Andreas.2015. Joint 3D estimation of vehicles and scene flow. In ISPRS Workshop on Image Sequence Analysis (ISA’15). ISPRS, 110.Google ScholarGoogle Scholar

Index Terms

  1. Sparse LIDAR Measurement Fusion with Joint Updating Cost for Fast Stereo Matching

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 1
        January 2022
        517 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3505205
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 27 January 2022
        • Accepted: 1 June 2021
        • Revised: 1 January 2021
        • Received: 1 July 2020
        Published in tomm Volume 18, Issue 1

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed
      • Article Metrics

        • Downloads (Last 12 months)215
        • Downloads (Last 6 weeks)2

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!