skip to main content
research-article

Quality Estimation and Optimization of Adaptive Stereo Matching Algorithms for Smart Vehicles

Authors Info & Claims
Published:10 February 2020Publication History
Skip Abstract Section

Abstract

Stereo matching is a promising approach for smart vehicles to find the depth of nearby objects. Transforming a traditional stereo matching algorithm to its adaptive version has potential advantages to achieve the maximum quality (depth accuracy) in a best-effort manner. However, it is very challenging to support this adaptive feature, since (1) the internal mechanism of adaptive stereo matching (ASM) has to be accurately modeled, and (2) scheduling ASM tasks on multiprocessors to generate the maximum quality is difficult under strict real-time constraints of smart vehicles. In this article, we propose a framework for constructing an ASM application and optimizing its output quality on smart vehicles. First, we empirically convert stereo matching into ASM by exploiting its inherent characteristics of disparity–cycle correspondence and introduce an exponential quality model that accurately represents the quality–cycle relationship. Second, with the explicit quality model, we propose an efficient quadratic programming-based dynamic voltage/frequency scaling (DVFS) algorithm to decide the optimal operating strategy, which maximizes the output quality under timing, energy, and temperature constraints. Third, we propose two novel methods to efficiently estimate the parameters of the quality model, namely location similarity-based feature point thresholding and street scenario-confined CNN prediction. Results show that our DVFS algorithm achieves at least 1.61 times quality improvement compared to the state-of-the-art techniques, and average parameter estimation for the quality model achieves 96.35% accuracy on the straight road.

References

  1. Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16). 265--283.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. H. Aydin, R. Melhem, D. Mosse, and P. Mejia-Alvarez. 2001. Optimal reward-based scheduling for periodic real-time tasks. IEEE Trans. Comput. 50, 2 (Feb 2001), 111--130. DOI:https://doi.org/10.1109/12.908988Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. G. Bradski. 2000. The OpenCV library. Dr. Dobb’s J. Softw. Tools (2000).Google ScholarGoogle Scholar
  4. Vinay K. Chippa, Kaushik Roy, Srimat T. Chakradhar, and Anand Raghunathan. 2013. Managing the quality vs. efficiency trade-off using dynamic effort scaling. ACM Trans. Embed. Comput. Syst. 12, 2s, Article 90 (May 2013), 23 pages. DOI:https://doi.org/10.1145/2465787.2465792Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. J. Chung, J. W. S. Liu, and K. Lin. 1990. Scheduling periodic jobs that allow imprecise results. IEEE Trans. Comput. 39, 9 (Sep. 1990), 1156--1174. DOI:https://doi.org/10.1109/12.57057Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. R. Danescu, F. Oniga, and S. Nedevschi. 2011. Modeling and tracking the driving environment with a particle-based occupancy grid. IEEE Trans. Intell. Transport. Syst. 12, 4 (Dec. 2011), 1331--1342. DOI:https://doi.org/10.1109/TITS.2011.2158097Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning. 1--16.Google ScholarGoogle Scholar
  8. A. Ess, B. Leibe, K. Schindler, and L. van Gool. 2009. Robust multiperson tracking from a mobile platform. IEEE Trans. Pattern Anal. Mach. Intell. 31, 10 (Oct. 2009), 1831--1846. DOI:https://doi.org/10.1109/TPAMI.2009.109Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Andreas Geiger, Philip Lenz, and Raquel Urtasun. 2012. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. 3354--3361. DOI:https://doi.org/10.1109/CVPR.2012.6248074Google ScholarGoogle ScholarCross RefCross Ref
  10. Andreas Geiger, Martin Roser, and Raquel Urtasun. 2011. Efficient large-scale stereo matching. In Proceedings of the Annual Conference on Computer Vision (ACCV’10), Ron Kimmel, Reinhard Klette, and Akihiro Sugimoto (Eds.). Springer, Berlin, 25--38.Google ScholarGoogle ScholarCross RefCross Ref
  11. V. Hanumaiah and S. Vrudhula. 2012. Temperature-aware DVFS for hard real-time applications on multicore processors. IEEE Trans. Comput. 61, 10 (Oct. 2012), 1484--1494. DOI:https://doi.org/10.1109/TC.2011.156Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. A. Hosni, C. Rhemann, M. Bleyer, C. Rother, and M. Gelautz. 2013. Fast cost-volume filtering for visual correspondence and beyond. IEEE Trans. Pattern Anal. Mach. Intell. 35, 2 (Feb. 2013), 504--511. DOI:https://doi.org/10.1109/TPAMI.2012.156Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Albert S. Huang, Abraham Bachrach, Peter Henry, Michael Krainin, Daniel Maturana, Dieter Fox, and Nicholas Roy. 2011. Visual odometry and mapping for autonomous flight using an RGB-D camera. In Proceedings of the International Symposium on Robotics Research (ISRR’11). 466--474. DOI:https://doi.org/10.1109/CVPR.2015.7298644Google ScholarGoogle Scholar
  14. H. Huang, V. Chaturvedi, G. Quan, J. Fan, and M. Qiu. 2014. Throughput maximization for periodic real-time systems under the maximal temperature constraint. ACM Trans. Embed. Comput. Syst. 13, 2s, Article 70 (Jan. 2014), 22 pages. DOI:https://doi.org/10.1145/2544375.2544390Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Wei Huang, Shougata Ghosh, Sivakumar Velusamy, Karthik Sankaranarayanan, Kevin Skadron, and Mircea Stan. 2006. HotSpot: A compact thermal modeling methodology for early-stage VLSI design. IEEE Trans. VLSI 14, 5 (2006), 501--513. DOI:https://doi.org/10.1109/TVLSI.2006.876103Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Juan Luis Jerez, George A. Constantinides, and Eric C. Kerrigan. 2011. An FPGA implementation of a sparse quadratic programming solver for constrained predictive control. In Proceedings of the ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA’11).Google ScholarGoogle Scholar
  17. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25. 1097--1105.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Kuk-Jin Yoon and In So Kweon. 2006. Adaptive support-weight approach for correspondence search. IEEE Trans. Pattern Anal. Mach. Intell. 28, 4 (Apr. 2006), 650--656. DOI:https://doi.org/10.1109/TPAMI.2006.70Google ScholarGoogle Scholar
  19. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (Nov. 1998), 2278--2324. DOI:https://doi.org/10.1109/5.726791Google ScholarGoogle ScholarCross RefCross Ref
  20. Weiping Liao, Lei He, and Kevin M. Lepak. 2005. Temperature and supply Voltage aware performance and power modeling at microarchitecture level. IEEE Trans. Comput.-Aid. Des. Integr. Circ. Syst. 24, 08 (2005), 1042--1053. DOI:https://doi.org/10.1109/TCAD.2005.850860Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Jiling Liu, Yong Zhang, and Xueguang Dong. 2015. Local stereo matching based on the improved matching cost function and the adaptive window. In Proceedings of the 2015 8th International Congress on Image and Signal Processing (CISP’15). IEEE, 287--292.Google ScholarGoogle ScholarCross RefCross Ref
  22. Tong Liu, Xiyuan Peng, and Qiao Li-yan. 2016. Window-based three-dimensional aggregation for stereo matching. IEEE Sign. Process. Lett. 23, 7 (2016), 1--1. DOI:https://doi.org/10.1109/LSP.2016.2578944Google ScholarGoogle ScholarCross RefCross Ref
  23. Frank D. Macías-Escrivá, Rodolfo E. Haber, Raúl M. del Toro, and Vicente Hernández. 2013. Self-adaptive systems: A survey of current approaches, research challenges and applications. Expert Syst. Appl. 40, 18 (2013), 7267--7279. DOI:https://doi.org/10.1016/j.eswa.2013.07.033Google ScholarGoogle Scholar
  24. Sarah Martull, Martin Peris, and Kazuhiro Fukui. 2012. Realistic CG stereo image dataset with ground truth disparity maps. In Proceedings of the International Conference on Pattern Recognition (ICPR) Workshop (TrakMark’12), Vol. 111. 117--118.Google ScholarGoogle Scholar
  25. Lei Mo, Angeliki Kritikakou, and Olivier Sentieys. 2017. Decomposed task mapping to maximize QoS in energy-constrained real-time multicores. In Proceedings of the 2017 IEEE International Conference on Computer Design (ICCD’17). 493--500. DOI:https://doi.org/10.1109/ICCD.2017.86Google ScholarGoogle ScholarCross RefCross Ref
  26. Andrew Nelson, Benny Akesson, Anca Molnos, Sjoerd Te Pas, and Kees Goossens. 2012. Power versus quality trade-offs for adaptive real-time applications. In Proceedings of the 2012 IEEE 10th Symposium on Embedded Systems for Real-time Multimedia. IEEE, 75--84.Google ScholarGoogle ScholarCross RefCross Ref
  27. O. Ozturk, M. Kandemir, and G. Chen. 2013. Compiler-directed energy reduction using dynamic voltage scaling and voltage islands for embedded systems. IEEE Trans. Comput. 62, 2 (Feb. 2013), 268--278. DOI:https://doi.org/10.1109/TC.2011.229Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Sylvain Paris, Pierre Kornprobst, Jack Tumblin, Frédo Durand, et al. 2009. Bilateral filtering: Theory and applications. Found. Trends Comput. Graph. Vis. 4, 1 (2009), 1--73.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Bidyut Kr. Patra, Raimo Launonen, Ville Ollikainen, and Sukumar Nandi. 2015. A new similarity measure using Bhattacharyya coefficient for collaborative filtering in sparse data. Knowl.-Based Syst. 82, C (2015), 163--177. DOI:10.1016/j.knosys.2015.03.001Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. M. Perrollaz, J. Yoder, A. Negre, A. Spalanzani, and C. Laugier. 2012. A visibility-based approach for occupancy grid computation in disparity space. IEEE Trans. Intell. Transport. Syst. 13, 3 (Sep. 2012), 1383--1393. DOI:https://doi.org/10.1109/TITS.2012.2188393Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Oscar Rahnama, Duncan P. Frost, Ondrej Miksik, and Philip H. S. Torr. 2018. Real-time dense stereo matching with ELAS on FPGA-accelerated embedded devices. IEEE Robot. Autom. Lett. 3, 3 (2018), 2008--2015. DOI:10.1109/LRA.2018.2800786Google ScholarGoogle ScholarCross RefCross Ref
  32. Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the 2011 International Conference on Computer Vision. 2564--2571. DOI:https://doi.org/10.1109/ICCV.2011.6126544Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Cosmin Rusu, Rami Melhem, and Daniel Mossé. 2003. Maximizing rewards for real-time applications with energy constraints. ACM Trans. Embed. Comput. Syst. 2, 4 (Nov. 2003), 537--559. DOI:https://doi.org/10.1145/950162.950166Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Daniel Scharstein and Richard Szeliski. 2003. High-accuracy stereo depth maps using structured light. In Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'03). IEEE Computer Society, 195--202.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Daniel Scharstein, Richard Szeliski, and Ramin Zabih. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithm. Int. J. Comput. Vis. 47, 1 (2002), 7--42. DOI:10.1023/A:1014573219977Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Stephan Schraml, Ahmed Nabil Belbachir, and Horst Bischof. 2015. Event-driven stereo matching for real-time 3D panoramic vision. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15), 466--474.Google ScholarGoogle ScholarCross RefCross Ref
  37. Wenjie Song, Yi Yang, Mengyin Fu, Yujun Li, and Meiling Wang. 2018. Lane detection and classification for forward collision warning system based on stereo vision. IEEE Sens. J. 18, 12 (Jun. 2018), 5151--5163. DOI:https://doi.org/10.1109/JSEN.2018.2832291Google ScholarGoogle ScholarCross RefCross Ref
  38. Rajeev Thakur. 2016. Scanning LIDAR in advanced driver assistance systems and beyond: Building a road map for next-generation LIDAR technology. IEEE Cons. Electr. Mag. 5, 3 (2016), 48--54. DOI:10.1109/MCE.2016.2556878Google ScholarGoogle ScholarCross RefCross Ref
  39. Sihan Wen. 2017. Convolutional neural network and adaptive guided image filter based stereo matching. In Proceedings of the IEEE International Conference on Imaging Systems and Techniques (IST’17). 1--6. DOI:https://doi.org/10.1109/IST.2017.8261530Google ScholarGoogle ScholarCross RefCross Ref
  40. Heng Yu, Yajun Ha, and Bharadwaj Veeravalli. 2013. Quality-driven dynamic scheduling for real-time adaptive applications on multiprocessor systems. IEEE Trans. Comput. 62, 10 (2013), 2026--2040. DOI:https://doi.org/10.1109/TC.2012.194Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Heng Yu, Yajun Ha, and Jing Wang. 2017. Quality optimization of resilient applications under temperature constraints. In Proceedings of the Computing Frontiers Conference (CF’17). ACM, New York, NY, 9--16. DOI:https://doi.org/10.1145/3075564.3075577Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Heng Yu, Bharadwaj Veeravalli, Yajun Ha, and Shaobo Luo. 2013. Dynamic scheduling of imprecise-computation tasks on real-time embedded multiprocessors. In Proceedings of the 2013 IEEE 16th International Conference on Computational Science and Engineering. 770--777.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Q. Zhang, F. Yuan, R. Ye, and Q. Xu. 2014. ApproxIt: An approximate computing framework for iterative methods. In Proceedings of the Design Automation Conference. 1--6.Google ScholarGoogle Scholar
  44. Sushu Zhang and Karam S. Chatha. 2010. Thermal aware task sequencing on embedded processors. In Proceedings of the 47th Design Automation Conference. ACM, 585--590.Google ScholarGoogle Scholar
  45. Junlong Zhou, Jianming Yan, Tongquan Wei, Mingsong Chen, and Xiaobo Sharon Hu. 2017. Energy-adaptive scheduling of imprecise computation tasks for QoS optimization in real-time MPSoC systems. DOI:https://doi.org/10.23919/DATE.2017.7927212Google ScholarGoogle Scholar
  46. Yongmei Zhou and Jingfei Jiang. 2015. An FPGA-based accelerator implementation for deep convolutional neural networks. In Proceedings of the 2015 4th International Conference on Computer Science and Network Technology (ICCSNT’15), 829--832.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Quality Estimation and Optimization of Adaptive Stereo Matching Algorithms for Smart Vehicles

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!