Abstract
The complementary virtues of active and passive depth sensors inspire the LIDAR-Stereo fusion for enhancing the accuracy of stereo matching. However, most of the fusion based stereo matching algorithms have exploited dense LIDAR priors with single fusion methodology. In this paper, we intend to break these fetters, utilizing sparse LIDAR priors with multi-step fusion strategy for obtaining accurate disparity estimation more efficiently. At first, random sparse sampling LIDAR depth measurements are provided in Naive Fusion for updating the matching cost of Semi-Global Matching (SGM). Then Neighborhood Based Fusion is performed based on the former step for further updating the cost. Subsequently, Diffusion Based Fusion is utilized to update both the cost and disparities. At last, Tree Filtering is applied for removing speckle outliers and smoothing disparities. Performance evaluations on various stereo data sets demonstrate that the proposed algorithm outperforms other most challenging stereo matching algorithms significantly with approximately real-time implementation efficiency. Furthermore, it is worth pointing out that our proposal surprisingly possesses one of the top ten performances on Middlebury v.3 online evaluation system even if it has not been adopted any learning-based techniques.
- [1] . 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In International Journal of Computer Vision (IJCV), Vol. 47. Springer, 7–42. Google Scholar
Digital Library
- [2] .2007. A performance study on different cost aggregation approaches used in real-time stereo matching. In International Journal of Computer Vision (IJCV), Vol. 75. Springer, 283–296. Google Scholar
Digital Library
- [3] .2008. Classification and evaluation of cost aggregation methods for stereo correspondence. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). IEEE, 1–8.Google Scholar
- [4] .2006. Adaptive support-weight approach for correspondence search. In IEEE Transactions on Pattern Analysis (TPAMI), Vol. 28. IEEE, 650–656. Google Scholar
Digital Library
- [5] .2013. Joint histogram based cost aggregation for stereo matching. In IEEE Transactions on Pattern Analysis (TPAMI), Vol. 35. IEEE, 2539–2545. Google Scholar
Digital Library
- [6] .2013. Stereo matching-state-of-the-art and research challenges. In Chapter 6 of Advanced Topics in Computer Vision. Springer, 143–179.Google Scholar
- [7] .2012. A non-local cost aggregation method for stereo matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 1402–1409. Google Scholar
Digital Library
- [8] .2015. Local smoothness enforced cost volume regularization for fast stereo correspondence. In IEEE Signal Processing Letters (SPL), Vol. 22. IEEE, 1429–1433.Google Scholar
- [9] .2012. PMBP: PatchMatch belief propagation for correspondence field estimation. In British Machine Vision Conference (BMVC’12). BMVA, 1–11.Google Scholar
- [10] .2014. As-rigid-as-possible stereo under second order smoothness priors. In European Conference on Computer Vision (ECCV’14). Springer, 112–126.Google Scholar
- [11] .2014. Graph cut based continuous stereo matching using locally shared labels. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). IEEE, 1613–1620. Google Scholar
Digital Library
- [12] .2018. Continuous 3D label stereo matching using local expansion moves. In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Vol. 40. IEEE, 2725–2739.Google Scholar
- [13] .2018. AGO: Accelerating global optimization for accurate stereo matching. In International Conference on Multimedia Modeling (MMM’18). Springer, 67–80.Google Scholar
- [14] .2018. MSCS: MeshStereo with cross-scale cost filtering for fast stereo matching. In IET Computer Vision (IET-CVI), Vol. 12. IET, 908–918.Google Scholar
Cross Ref
- [15] .2005. Accurate and efficient stereo processing by semi-global matching and mutual information. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’05). IEEE, 807–814. Google Scholar
Digital Library
- [16] .2008. Stereo processing by semi-global matching and mutual information. In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Vol. 30. IEEE, 328–341. Google Scholar
Digital Library
- [17] .2015. MGM: A significantly more global matching for stereo vision. In British Machine Vision Conference (BMVC’15). BMVA, 1–12.Google Scholar
- [18] .2019. As-global-as-possible stereo matching with adaptive smoothness prior. In IET Image Processing (IET-IPR), Vol. 13. IET, 98–107.Google Scholar
Cross Ref
- [19] .2015. Computing the stereo matching cost with a convolutional neural network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). IEEE, 1592–1599.Google Scholar
- [20] .2016. Stereo matching by training a convolutional neural network to compare image patches. In Journal of Machine Learning Research (JMLR), Vol. 17. MIT Press, 1–32. Google Scholar
Digital Library
- [21] .2016. Patch based confidence prediction for dense disparity map. In British Machine Vision Conference (BMVC’16). BMVA, 1–13.Google Scholar
Cross Ref
- [22] .2017. SGM-nets: Semi-global matching with neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 6640–6649.Google Scholar
- [23] .2017. Detect, replace, refine: Deep structured prediction for pixel wise labeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 7187–7196.Google Scholar
- [24] 2018. RecResNet: A recurrent residual CNN architecture for disparity map enhancement. In IEEE International Conference on 3D Vision (3DV’18). IEEE, 238–247.Google Scholar
- [25] .2016. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). IEEE, 4040–4048.Google Scholar
- [26] .2017. Cascade residual learning: A two-stage convolutional neural network for stereo matching. In IEEE International Conference on Computer Vision (ICCV’17). IEEE, 878–886.Google Scholar
- [27] .2018. Learning for disparity estimation through feature constancy. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). IEEE, 2811–2820.Google Scholar
- [28] .2017. End-to-end learning of geometry and context for deep stereo regression. In IEEE International Conference on Computer Vision (ICCV’17). IEEE, 66–75.Google Scholar
- [29] .2008. Fusion of time-of-flight depth and stereo for high accuracy depth maps. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). IEEE, 1–8.Google Scholar
- [30] .2012. High-resolution depth maps based on ToF-stereo fusion. In IEEE International Conference on Robotics and Automation (ICRA’12). IEEE, 4742–4749.Google Scholar
- [31] .2011. Integrating LIDAR into stereo for fast and improved disparity computation. In IEEE International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT’11). IEEE, 405–412. Google Scholar
Digital Library
- [32] .2016. Real-time probabilistic fusion of sparse 3D LIDAR and dense stereo. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’16). IEEE, 2181–2188.Google Scholar
- [33] .2019. Guided stereo matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19). IEEE, 979–988.Google Scholar
- [34] .2019. Noise-aware unsupervised deep LIDAR-Stereo fusion. In IEEE Computer Vision and Pattern Recognition (CVPR’19). IEEE, 6339–6348.Google Scholar
- [35] .2011. Combination of time-of-flight depth and stereo using semi-global optimization. In IEEE International Conference on Robotics and Automation (ICRA’11). IEEE, 3548–3553.Google Scholar
- [36] .2019. Real time dense depth estimation by fusing stereo with sparse depth measurements. In IEEE International Conference on Robotics and Automation (ICRA’19). IEEE, pp. 6482–6488.Google Scholar
- [37] 2018. CBMV: A coalesced bidirectional matching volume for disparity estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). IEEE, 2060–2069.Google Scholar
- [38] .2007. Learning conditional random fields for stereo. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’07). IEEE, 1–8.Google Scholar
- [39] .2007. Evaluation of cost functions for stereo matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’07). IEEE, 1–8.Google Scholar
- [40] .2014. Semi-global matching: A principled derivation in terms of message passing. In German Conference of Pattern Recognition (GCPR’14). Springer, 43–53.Google Scholar
- [41] .2013. Ensemble learning for confidence measures in stereo vision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’13). IEEE, 305–312. Google Scholar
Digital Library
- [42] .2014. Learning to detect ground control points for improving the accuracy of stereo matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). IEEE, 1621–1628. Google Scholar
Digital Library
- [43] .2016. Correctness prediction, accuracy improvement and generalization of stereo matching using supervised learning. In International Journal of Computer Vision (IJCV), Vol. 118. Springer, 300–318. Google Scholar
Digital Library
- [44] .2015. Leveraging stereo matching with learning-based confidence measures. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). IEEE, 101–109.Google Scholar
- [45] .2016. Learning a general-purpose confidence measure based on O(1) features and a smarter aggregation strategy for semi global matching. In IEEE International Conference on 3D Vision (3DV’16). IEEE, 509–518.Google Scholar
- [46] .2021. Ensemble learning with advanced fast image filtering features for semi-global matching. In Machine Vision and Applications (MVA), Vol. 32. Springer, 1–15.Google Scholar
- [47] .2017. Sparse representation over discriminative dictionary for stereo matching. In Pattern Recognition (PR), Vol. 71. Elsevier, 278–289.Google Scholar
Cross Ref
- [48] .2010. Efficient large-scale stereo matching. In Asian Conference on Computer Vision (ACCV’10). Springer, 25–38. Google Scholar
Digital Library
- [49] .2016. Integrating smartphone images and airborne LIDAR data for complete urban building modelling. In International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS’16). ISPRS, 741–747.Google Scholar
- [50] .2017. Line-based registration of panoramic images and LiDAR point clouds for mobile mapping. In Sensors (Switzerland), Vol. 1. MDPI, 1–20.Google Scholar
- [51] .2020. Airborne LIDAR point cloud classification fusion with DIM point cloud. In International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS’20), Vol. 43, No. B2. ISPRS, 375–382.Google Scholar
- [52] .2020. A frustum-based probabilistic framework for 3D object detection by fusion of LiDAR and camera data. In ISPRS Journal of Photogrammetry and Remote Sensing (ISPRS-JPRS), Vol. 159. ISPRS, 90–100.Google Scholar
Cross Ref
- [53] .2020. Automatic building extraction via adaptive iterative segmentation with LiDAR data and high spatial resolution imagery fusion. In IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (JST-AEORS), Vol. 13. IEEE, 2081–2095.Google Scholar
Cross Ref
- [54] .2016. High-Resolution LIDAR-Based depth mapping using bilateral filter. In IEEE Intelligent Transportation Systems (ITSC’16). IEEE, 2469–2474.Google Scholar
- [55] .1994. Non-parametric local transforms for computing visual correspondence. In European Conference on Computer Vision (ECCV’94). Springer, 151–158. Google Scholar
Digital Library
- [56] .1990. Scale-space and edge detection using anisotropic diffusion. In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Vol. 12. IEEE, 629–639. Google Scholar
Digital Library
- [57] .2015. Stereo matching using tree filtering. In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Vol. 37. IEEE, 834–846.Google Scholar
- [58] .2014. High-resolution stereo datasets withsubpixel-accurate ground truth. In German Conference on Pattern Recognition (GCPR’14). Springer, 31–42.Google Scholar
- [59] .2012. Are we ready for autonomous driving? The KITTI Vision Benchmark Suite. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 1–8. Google Scholar
Digital Library
- [60] .2015. Joint 3D estimation of vehicles and scene flow. In ISPRS Workshop on Image Sequence Analysis (ISA’15). ISPRS, 1–10.Google Scholar
Index Terms
Sparse LIDAR Measurement Fusion with Joint Updating Cost for Fast Stereo Matching
Recommendations
Stereo vision using two PTZ cameras
The research of traditional stereo vision is mainly based on static cameras. As PTZ (Pan-Tilt-Zoom) cameras are able to obtain multi-view-angle and multi-resolution information, they have received more and more concern in both research and real ...
Stereo fusion
A stereo fusion system that combines binocular and refractive stereo is presented.Our stereo fusion outperforms traditional binocular and refractive stereo.An efficient calibration method for refractive stereo is proposed. Display Omitted The ...
Binocular structured light stereo matching approach for dense facial disparity map
AI'11: Proceedings of the 24th international conference on Advances in Artificial IntelligenceBinocular stereo vision technology shows a particular interesting for face recognition, in which the accurate stereo matching is the key issue for obtaining dense disparity map used for exploiting 3D shape information of object. This paper proposed a ...






Comments