Abstract
Stereo matching is a promising approach for smart vehicles to find the depth of nearby objects. Transforming a traditional stereo matching algorithm to its adaptive version has potential advantages to achieve the maximum quality (depth accuracy) in a best-effort manner. However, it is very challenging to support this adaptive feature, since (1) the internal mechanism of adaptive stereo matching (ASM) has to be accurately modeled, and (2) scheduling ASM tasks on multiprocessors to generate the maximum quality is difficult under strict real-time constraints of smart vehicles. In this article, we propose a framework for constructing an ASM application and optimizing its output quality on smart vehicles. First, we empirically convert stereo matching into ASM by exploiting its inherent characteristics of disparity–cycle correspondence and introduce an exponential quality model that accurately represents the quality–cycle relationship. Second, with the explicit quality model, we propose an efficient quadratic programming-based dynamic voltage/frequency scaling (DVFS) algorithm to decide the optimal operating strategy, which maximizes the output quality under timing, energy, and temperature constraints. Third, we propose two novel methods to efficiently estimate the parameters of the quality model, namely location similarity-based feature point thresholding and street scenario-confined CNN prediction. Results show that our DVFS algorithm achieves at least 1.61 times quality improvement compared to the state-of-the-art techniques, and average parameter estimation for the quality model achieves 96.35% accuracy on the straight road.
- Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16). 265--283.Google Scholar
Digital Library
- H. Aydin, R. Melhem, D. Mosse, and P. Mejia-Alvarez. 2001. Optimal reward-based scheduling for periodic real-time tasks. IEEE Trans. Comput. 50, 2 (Feb 2001), 111--130. DOI:https://doi.org/10.1109/12.908988Google Scholar
Digital Library
- G. Bradski. 2000. The OpenCV library. Dr. Dobb’s J. Softw. Tools (2000).Google Scholar
- Vinay K. Chippa, Kaushik Roy, Srimat T. Chakradhar, and Anand Raghunathan. 2013. Managing the quality vs. efficiency trade-off using dynamic effort scaling. ACM Trans. Embed. Comput. Syst. 12, 2s, Article 90 (May 2013), 23 pages. DOI:https://doi.org/10.1145/2465787.2465792Google Scholar
Digital Library
- J. Chung, J. W. S. Liu, and K. Lin. 1990. Scheduling periodic jobs that allow imprecise results. IEEE Trans. Comput. 39, 9 (Sep. 1990), 1156--1174. DOI:https://doi.org/10.1109/12.57057Google Scholar
Digital Library
- R. Danescu, F. Oniga, and S. Nedevschi. 2011. Modeling and tracking the driving environment with a particle-based occupancy grid. IEEE Trans. Intell. Transport. Syst. 12, 4 (Dec. 2011), 1331--1342. DOI:https://doi.org/10.1109/TITS.2011.2158097Google Scholar
Digital Library
- Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning. 1--16.Google Scholar
- A. Ess, B. Leibe, K. Schindler, and L. van Gool. 2009. Robust multiperson tracking from a mobile platform. IEEE Trans. Pattern Anal. Mach. Intell. 31, 10 (Oct. 2009), 1831--1846. DOI:https://doi.org/10.1109/TPAMI.2009.109Google Scholar
Digital Library
- Andreas Geiger, Philip Lenz, and Raquel Urtasun. 2012. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. 3354--3361. DOI:https://doi.org/10.1109/CVPR.2012.6248074Google Scholar
Cross Ref
- Andreas Geiger, Martin Roser, and Raquel Urtasun. 2011. Efficient large-scale stereo matching. In Proceedings of the Annual Conference on Computer Vision (ACCV’10), Ron Kimmel, Reinhard Klette, and Akihiro Sugimoto (Eds.). Springer, Berlin, 25--38.Google Scholar
Cross Ref
- V. Hanumaiah and S. Vrudhula. 2012. Temperature-aware DVFS for hard real-time applications on multicore processors. IEEE Trans. Comput. 61, 10 (Oct. 2012), 1484--1494. DOI:https://doi.org/10.1109/TC.2011.156Google Scholar
Digital Library
- A. Hosni, C. Rhemann, M. Bleyer, C. Rother, and M. Gelautz. 2013. Fast cost-volume filtering for visual correspondence and beyond. IEEE Trans. Pattern Anal. Mach. Intell. 35, 2 (Feb. 2013), 504--511. DOI:https://doi.org/10.1109/TPAMI.2012.156Google Scholar
Digital Library
- Albert S. Huang, Abraham Bachrach, Peter Henry, Michael Krainin, Daniel Maturana, Dieter Fox, and Nicholas Roy. 2011. Visual odometry and mapping for autonomous flight using an RGB-D camera. In Proceedings of the International Symposium on Robotics Research (ISRR’11). 466--474. DOI:https://doi.org/10.1109/CVPR.2015.7298644Google Scholar
- H. Huang, V. Chaturvedi, G. Quan, J. Fan, and M. Qiu. 2014. Throughput maximization for periodic real-time systems under the maximal temperature constraint. ACM Trans. Embed. Comput. Syst. 13, 2s, Article 70 (Jan. 2014), 22 pages. DOI:https://doi.org/10.1145/2544375.2544390Google Scholar
Digital Library
- Wei Huang, Shougata Ghosh, Sivakumar Velusamy, Karthik Sankaranarayanan, Kevin Skadron, and Mircea Stan. 2006. HotSpot: A compact thermal modeling methodology for early-stage VLSI design. IEEE Trans. VLSI 14, 5 (2006), 501--513. DOI:https://doi.org/10.1109/TVLSI.2006.876103Google Scholar
Digital Library
- Juan Luis Jerez, George A. Constantinides, and Eric C. Kerrigan. 2011. An FPGA implementation of a sparse quadratic programming solver for constrained predictive control. In Proceedings of the ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA’11).Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25. 1097--1105.Google Scholar
Digital Library
- Kuk-Jin Yoon and In So Kweon. 2006. Adaptive support-weight approach for correspondence search. IEEE Trans. Pattern Anal. Mach. Intell. 28, 4 (Apr. 2006), 650--656. DOI:https://doi.org/10.1109/TPAMI.2006.70Google Scholar
- Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (Nov. 1998), 2278--2324. DOI:https://doi.org/10.1109/5.726791Google Scholar
Cross Ref
- Weiping Liao, Lei He, and Kevin M. Lepak. 2005. Temperature and supply Voltage aware performance and power modeling at microarchitecture level. IEEE Trans. Comput.-Aid. Des. Integr. Circ. Syst. 24, 08 (2005), 1042--1053. DOI:https://doi.org/10.1109/TCAD.2005.850860Google Scholar
Digital Library
- Jiling Liu, Yong Zhang, and Xueguang Dong. 2015. Local stereo matching based on the improved matching cost function and the adaptive window. In Proceedings of the 2015 8th International Congress on Image and Signal Processing (CISP’15). IEEE, 287--292.Google Scholar
Cross Ref
- Tong Liu, Xiyuan Peng, and Qiao Li-yan. 2016. Window-based three-dimensional aggregation for stereo matching. IEEE Sign. Process. Lett. 23, 7 (2016), 1--1. DOI:https://doi.org/10.1109/LSP.2016.2578944Google Scholar
Cross Ref
- Frank D. Macías-Escrivá, Rodolfo E. Haber, Raúl M. del Toro, and Vicente Hernández. 2013. Self-adaptive systems: A survey of current approaches, research challenges and applications. Expert Syst. Appl. 40, 18 (2013), 7267--7279. DOI:https://doi.org/10.1016/j.eswa.2013.07.033Google Scholar
- Sarah Martull, Martin Peris, and Kazuhiro Fukui. 2012. Realistic CG stereo image dataset with ground truth disparity maps. In Proceedings of the International Conference on Pattern Recognition (ICPR) Workshop (TrakMark’12), Vol. 111. 117--118.Google Scholar
- Lei Mo, Angeliki Kritikakou, and Olivier Sentieys. 2017. Decomposed task mapping to maximize QoS in energy-constrained real-time multicores. In Proceedings of the 2017 IEEE International Conference on Computer Design (ICCD’17). 493--500. DOI:https://doi.org/10.1109/ICCD.2017.86Google Scholar
Cross Ref
- Andrew Nelson, Benny Akesson, Anca Molnos, Sjoerd Te Pas, and Kees Goossens. 2012. Power versus quality trade-offs for adaptive real-time applications. In Proceedings of the 2012 IEEE 10th Symposium on Embedded Systems for Real-time Multimedia. IEEE, 75--84.Google Scholar
Cross Ref
- O. Ozturk, M. Kandemir, and G. Chen. 2013. Compiler-directed energy reduction using dynamic voltage scaling and voltage islands for embedded systems. IEEE Trans. Comput. 62, 2 (Feb. 2013), 268--278. DOI:https://doi.org/10.1109/TC.2011.229Google Scholar
Digital Library
- Sylvain Paris, Pierre Kornprobst, Jack Tumblin, Frédo Durand, et al. 2009. Bilateral filtering: Theory and applications. Found. Trends Comput. Graph. Vis. 4, 1 (2009), 1--73.Google Scholar
Digital Library
- Bidyut Kr. Patra, Raimo Launonen, Ville Ollikainen, and Sukumar Nandi. 2015. A new similarity measure using Bhattacharyya coefficient for collaborative filtering in sparse data. Knowl.-Based Syst. 82, C (2015), 163--177. DOI:10.1016/j.knosys.2015.03.001Google Scholar
Digital Library
- M. Perrollaz, J. Yoder, A. Negre, A. Spalanzani, and C. Laugier. 2012. A visibility-based approach for occupancy grid computation in disparity space. IEEE Trans. Intell. Transport. Syst. 13, 3 (Sep. 2012), 1383--1393. DOI:https://doi.org/10.1109/TITS.2012.2188393Google Scholar
Digital Library
- Oscar Rahnama, Duncan P. Frost, Ondrej Miksik, and Philip H. S. Torr. 2018. Real-time dense stereo matching with ELAS on FPGA-accelerated embedded devices. IEEE Robot. Autom. Lett. 3, 3 (2018), 2008--2015. DOI:10.1109/LRA.2018.2800786Google Scholar
Cross Ref
- Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the 2011 International Conference on Computer Vision. 2564--2571. DOI:https://doi.org/10.1109/ICCV.2011.6126544Google Scholar
Digital Library
- Cosmin Rusu, Rami Melhem, and Daniel Mossé. 2003. Maximizing rewards for real-time applications with energy constraints. ACM Trans. Embed. Comput. Syst. 2, 4 (Nov. 2003), 537--559. DOI:https://doi.org/10.1145/950162.950166Google Scholar
Digital Library
- Daniel Scharstein and Richard Szeliski. 2003. High-accuracy stereo depth maps using structured light. In Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'03). IEEE Computer Society, 195--202.Google Scholar
Digital Library
- Daniel Scharstein, Richard Szeliski, and Ramin Zabih. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithm. Int. J. Comput. Vis. 47, 1 (2002), 7--42. DOI:10.1023/A:1014573219977Google Scholar
Digital Library
- Stephan Schraml, Ahmed Nabil Belbachir, and Horst Bischof. 2015. Event-driven stereo matching for real-time 3D panoramic vision. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15), 466--474.Google Scholar
Cross Ref
- Wenjie Song, Yi Yang, Mengyin Fu, Yujun Li, and Meiling Wang. 2018. Lane detection and classification for forward collision warning system based on stereo vision. IEEE Sens. J. 18, 12 (Jun. 2018), 5151--5163. DOI:https://doi.org/10.1109/JSEN.2018.2832291Google Scholar
Cross Ref
- Rajeev Thakur. 2016. Scanning LIDAR in advanced driver assistance systems and beyond: Building a road map for next-generation LIDAR technology. IEEE Cons. Electr. Mag. 5, 3 (2016), 48--54. DOI:10.1109/MCE.2016.2556878Google Scholar
Cross Ref
- Sihan Wen. 2017. Convolutional neural network and adaptive guided image filter based stereo matching. In Proceedings of the IEEE International Conference on Imaging Systems and Techniques (IST’17). 1--6. DOI:https://doi.org/10.1109/IST.2017.8261530Google Scholar
Cross Ref
- Heng Yu, Yajun Ha, and Bharadwaj Veeravalli. 2013. Quality-driven dynamic scheduling for real-time adaptive applications on multiprocessor systems. IEEE Trans. Comput. 62, 10 (2013), 2026--2040. DOI:https://doi.org/10.1109/TC.2012.194Google Scholar
Digital Library
- Heng Yu, Yajun Ha, and Jing Wang. 2017. Quality optimization of resilient applications under temperature constraints. In Proceedings of the Computing Frontiers Conference (CF’17). ACM, New York, NY, 9--16. DOI:https://doi.org/10.1145/3075564.3075577Google Scholar
Digital Library
- Heng Yu, Bharadwaj Veeravalli, Yajun Ha, and Shaobo Luo. 2013. Dynamic scheduling of imprecise-computation tasks on real-time embedded multiprocessors. In Proceedings of the 2013 IEEE 16th International Conference on Computational Science and Engineering. 770--777.Google Scholar
Digital Library
- Q. Zhang, F. Yuan, R. Ye, and Q. Xu. 2014. ApproxIt: An approximate computing framework for iterative methods. In Proceedings of the Design Automation Conference. 1--6.Google Scholar
- Sushu Zhang and Karam S. Chatha. 2010. Thermal aware task sequencing on embedded processors. In Proceedings of the 47th Design Automation Conference. ACM, 585--590.Google Scholar
- Junlong Zhou, Jianming Yan, Tongquan Wei, Mingsong Chen, and Xiaobo Sharon Hu. 2017. Energy-adaptive scheduling of imprecise computation tasks for QoS optimization in real-time MPSoC systems. DOI:https://doi.org/10.23919/DATE.2017.7927212Google Scholar
- Yongmei Zhou and Jingfei Jiang. 2015. An FPGA-based accelerator implementation for deep convolutional neural networks. In Proceedings of the 2015 4th International Conference on Computer Science and Network Technology (ICCSNT’15), 829--832.Google Scholar
Cross Ref
Index Terms
Quality Estimation and Optimization of Adaptive Stereo Matching Algorithms for Smart Vehicles
Recommendations
Embedded Real-time Stereo Estimation via Semi-global Matching on the GPU
Dense, robust and real-time computation of depth information from stereo-camera systems is a computationally demanding requirement for robotics, advanced driver assistance systems (ADAS) and autonomous vehicles. Semi-Global Matching (SGM) is a widely ...
Research on Multi-sensor Fusion Perception Method of Vehicle-infrastructure Collaboration for Smart Automobiles
ICCIR '21: Proceedings of the 2021 1st International Conference on Control and Intelligent RoboticsTraffic congestion should be solved, it reduce traffic safety potential risks, and improve people's travel efficiency. The paper based on intelligent car's own operation characteristics (Smart car perception is generally only 300 ~ 500 meters, there is ...
Adaptive descriptor-based robust stereo matching under radiometric changes
An effective matching scheme for robust stereo matching under various radiometric changes.A content adaptive descriptor-based approach to effectively reflect image contents.Entropy-based energy function guiding for weighting the elements of the ...






Comments