skip to main content
research-article

Online Correction of Camera Poses for the Surround-view System: A Sparse Direct Approach

Authors Info & Claims
Published:04 March 2022Publication History
Skip Abstract Section

Abstract

The surround-view module is an indispensable component of a modern advanced driving assistance system. By calibrating the intrinsics and extrinsics of the surround-view cameras accurately, a top-down surround-view can be generated from raw fisheye images. However, poses of these cameras sometimes may change. At present, how to correct poses of cameras in a surround-view system online without re-calibration is still an open issue. To settle this problem, we introduce the sparse direct framework and propose a novel optimization scheme of a cascade structure. This scheme is actually composed of two levels of optimization and two corresponding photometric error based models are proposed. The model for the first-level optimization is called the ground model, as its photometric errors are measured on the ground plane. For the second level of the optimization, it’s based on the so-called ground-camera model, in which photometric errors are computed on the imaging planes. With these models, the pose correction task is formulated as a nonlinear least-squares problem to minimize photometric errors in overlapping regions of adjacent bird’s-eye-view images. With a cascade structure of these two levels of optimization, an appropriate balance between the speed and the accuracy can be achieved. Experiments show that our method can effectively eliminate the misalignment caused by cameras’ moderate pose changes in the surround-view system. Source code and test cases are available online at https://cslinzhang.github.io/CamPoseCorrection/.

REFERENCES

  1. [1] Battiti R.. 1992. First- and second-order methods for learning: Between steepest descent and Newton’s method. Neural Computation 4, 2 (1992), 141166.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Bay H., Tuytelaars T., and Gool L. V.. 2006. SURF: Speeded up robust features. In Proc. European Conf. Comput. Vis.404417.Google ScholarGoogle Scholar
  3. [3] Choi K., Jung H., and Suhr J.. 2018. Automatic calibration of an around view monitor system exploiting lane markings. Sensors 18, 9 (2018), 2956:1–26.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Collado J., Hilario C., Escalera A., and Armingol J.. 2006. Self-calibration of an on-board stereo-vision system for driver assistance systems. In Proc. Int’l IEEE Conf. Intell. Vehicles Symposium. 156162.Google ScholarGoogle Scholar
  5. [5] Dang T. and Hoffmann C.. 2006. Tracking camera parameters of an active stereo rig. In Joint DAGM Symposium. 627-636.Google ScholarGoogle Scholar
  6. [6] Dennis J. E. and Schnabel R. B.. 1983. Numerical methods for unconstrained optimization and nonlinear equations. Prentice Hall, Inc. 28, 3 (1983), 417419.Google ScholarGoogle Scholar
  7. [7] Du F. and Brady M.. 1993. Self-calibration of the intrinsic parameters of cameras for active vision systems. In Proc. IEEE Int’l Conf. Comput. Vis. Pattern Recognit.477482.Google ScholarGoogle Scholar
  8. [8] Engel J., Koltun V., and Cremers D.. 2018. Direct sparse odometry. IEEE Trans. Pattern Analysis and Machine Intell. 40, 3 (2018), 611625.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Engel J., Schöps T., and Cremers D.. 2014. LSD-SLAM: Large-scale direct monocular SLAM. In Proc. European Conf. Comput. Vis.834849.Google ScholarGoogle Scholar
  10. [10] Forster C., Pizzoli M., and Scaramuzza D.. 2014. SVO: Fast semi-direct monocular visual odometry. In Proc. IEEE Int’l Conf. Robotics and Automation. 1522.Google ScholarGoogle Scholar
  11. [11] Gressmann M., Palm G., and Löhlein O.. 2011. Surround view pedestrian detection using heterogeneous classifier cascades. In Proc. Int’l IEEE Conf. Intell. Transportation Systems. 13171324.Google ScholarGoogle Scholar
  12. [12] Hansen P., Alismail H., Rander P., and Browning B.. 2012. Online continuous stereo extrinsic parameter estimation. In Proc. IEEE Int’l Conf. Comput. Vis. Pattern Recognit.10591066.Google ScholarGoogle Scholar
  13. [13] Hartley R. and Zisserman A.. 2003. Multiple View Geometry in Computer Vision (2 ed.). Cambridge University Press, USA.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Heng L., Bürki M., Lee G. H., Furgale P., Siegwart R., and Pollefeys M.. 2014. Infrastructure-based calibration of a multi-camera rig. In Proc. IEEE Int’l Conf. Robotics and Automation. 49124919.Google ScholarGoogle Scholar
  15. [15] Heng L., Li B., and Pollefeys M.. 2013. CamOdoCal: Automatic intrinsic and extrinsic calibration of a rig with multiple generic cameras and odometry. In Proc. IEEE/RSJ Int’l Conf. Intell. Robots and Systems. 17931800.Google ScholarGoogle Scholar
  16. [16] Hoffman W. C.. 1966. The Lie algebra of visual perception. J. Mathematical Psychology 3, 1 (1966), 6598.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Hold S., Görmer S., Kummert A., Meuter M., and Muller-Schneiders S.. 2009. A novel approach for the online initial calibration of extrinsic parameters for a car-mounted camera. In Proc. Int’l IEEE Conf. Intell. Transportation Systems. 420425.Google ScholarGoogle Scholar
  18. [18] Hou C., Ai H., and Lao S.. 2007. Multiview pedestrian detection based on vector boosting. In Proc. Asian Conf. Comput. Vis.1822.Google ScholarGoogle Scholar
  19. [19] Irani M. and Anandan P.. 1999. About direct methods. In Proc. Int’l Workshop on Vis. Algorithms. 267277.Google ScholarGoogle Scholar
  20. [20] Klette R., Koschan A., and Schluns K.. 1998. Computer Vision: Three-dimensional Data from Images. Springer, Singapore.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Knorr M., Niehsen W., and Stiller C.. 2013. Online extrinsic multi-camera calibration using ground plane induced homographies. In Proc. IEEE Intell. Vehicles Symposium. 236241.Google ScholarGoogle Scholar
  22. [22] Lébraly Pierre, Royer Eric, Ait-Aider Omar, Deymier Clément, and Dhome Michel. 2011. Fast calibration of embedded non-overlapping cameras. In Proc. IEEE Int’l Conf. Robotics and Automation. 221227.Google ScholarGoogle Scholar
  23. [23] Li L., Zhang L., Li X., Liu X., Shen Y., and Xiong L.. 2017. Vision-based parking-slot detection: A benchmark and a learning-based approach. In Proc. IEEE Int’l Conf. Multimedia and Expo. 649654.Google ScholarGoogle Scholar
  24. [24] Lin C. and Wang M.. 2012. A vision based top-view transformation model for a vehicle parking assistant. Sensors 12, 4 (2012), 44314446.Google ScholarGoogle Scholar
  25. [25] Ling Y. and Shen S.. 2016. High-precision online markerless stereo extrinsic calibration. In Proc. IEEE/RSJ Int’l Conf. Intell. Robots and Systems. 17711778.Google ScholarGoogle Scholar
  26. [26] Liu X., Zhang L., Shen Y., Zhang S., and Zhao S.. 2019. Online camera pose optimization for the surround-view system. In Proc. ACM Int’l Conf. Multimedia. 383391.Google ScholarGoogle Scholar
  27. [27] Lourakis M. I. A.. 2019. Sparse non-linear least squares optimization for geometric vision. In Proc. European Conf. Comput. Vis.4356.Google ScholarGoogle Scholar
  28. [28] Lowe D. G.. 2004. Distinctive image features from scale-invariant keypoints. Int’l J. Comput. Vis. 60, 2 (2004), 91110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Lucas B. D. and Kanade T.. 1981. An iterative image registration technique with an application to stereo vision. In Proc. Int’l Joint Conf. Artificial Intell.674679.Google ScholarGoogle Scholar
  30. [30] Moré J. J.. 1978. The Levenberg-Marquardt algorithm: Implementation and theory. In Numerical Analysis.Google ScholarGoogle Scholar
  31. [31] Mur-Artal R. and Tardós J. D.. 2017. ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Trans. Robotics 33, 5 (2017), 12551262.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Nedevschi S., Vancea C., Marita T., and Graf T.. 2007. Online extrinsic parameters calibration for stereo vision systems used in far-range detection vehicle applications. IEEE Trans. Intell. Transportation Systems 8, 4 (2007).Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Newcombe R. A., Lovegrove S. J., and Davison A. J.. 2011. DTAM: Dense tracking and mapping in real-time. In Proc. IEEE Int’l Conf. Comput. Vis.23202327.Google ScholarGoogle Scholar
  34. [34] Nielsen F.. 2005. Surround video: A multihead camera approach. The Visual Computer 21, 1-2 (2005), 92103.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Nocedal J.. 1992. Theory of algorithms for unconstrained optimization. Acta Numerica 1, 8 (1992), 199242.Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Mur-Artal J. M. M. Montiel R. and Tardós J. D.. 2015. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Trans. Robotics 31, 5 (2015), 11471163.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Rublee E., Rabaud V., Konolige K., and Bradski G.. 2011. ORB: An efficient alternative to SIFT or SURF. Proc. IEEE Int’l Conf. Comput. Vis. (2011), 25642571.Google ScholarGoogle Scholar
  38. [38] Schneider S., Luettel T., and Wuensche H.. 2013. Odometry-based online extrinsic sensor calibration. In Proc. IEEE/RSJ Int’l Conf. Intell. Robots and Systems. 12871292.Google ScholarGoogle Scholar
  39. [39] Shao X., Liu X., Zhang L., Zhao S., Shen Y., and Yang Y.. 2019. Revisit surround-view camera system calibration. In Proc. IEEE Int’l Conf. Multimedia and Expo. 14861491.Google ScholarGoogle Scholar
  40. [40] Wedderburn R. W. M.. 1974. Quasi-likelihood functions, generalized linear models, and the Gauss-Newton method. Biometrika 61, 3 (1974), 439447.Google ScholarGoogle Scholar
  41. [41] Xu J., Chen G., and Xie M.. 2000. Vision-guided automatic parking for smart car. In Proc. IEEE Intell. Vehicles Symposium. 725730.Google ScholarGoogle Scholar
  42. [42] Zhang L., Huang J., Li X., and Xiong L.. 2018. Vision-based parking-slot detection: A DCNN-based approach and a large-scale benchmark dataset. IEEE Trans. Image Processing 27, 11 (2018), 53505364.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. [43] Zhang Z.. 1999. Flexible camera calibration by viewing a plane from unknown orientations. In Proc. IEEE Int’l Conf. Comput. Vis.666673.Google ScholarGoogle Scholar
  44. [44] Zhao K., Iurgel U., Meuter M., and Pauli J.. 2014. An automatic online camera calibration system for vehicular applications. In Proc. Int’l IEEE Conf. Intell. Transportation Systems. 14901492.Google ScholarGoogle Scholar
  45. [45] Zhu H., Yang J., and Liu Z.. 2009. Fisheye camera calibration with two pairs of vanishing points. In Proc. Int’l Conf. Inf. Tech. Comput. Sci.321324.Google ScholarGoogle Scholar

Index Terms

  1. Online Correction of Camera Poses for the Surround-view System: A Sparse Direct Approach

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 4
        November 2022
        497 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3514185
        • Editor:
        • Abdulmotaleb El Saddik
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 4 March 2022
        • Accepted: 1 December 2021
        • Revised: 1 October 2021
        • Received: 1 May 2021
        Published in tomm Volume 18, Issue 4

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed
      • Article Metrics

        • Downloads (Last 12 months)310
        • Downloads (Last 6 weeks)11

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!