skip to main content
research-article

Robust Real-Time Super-Resolution on FPGA and an Application to Video Enhancement

Published:01 September 2009Publication History
Skip Abstract Section

Abstract

The high density image sensors of state-of-the-art imaging systems provide outputs with high spatial resolution, but require long exposure times. This limits their applicability, due to the motion blur effect. Recent technological advances have lead to adaptive image sensors that can combine several pixels together in real time to form a larger pixel. Larger pixels require shorter exposure times and produce high-frame-rate samples with reduced motion blur. This work proposes combining an FPGA with an adaptive image sensor to produce an output of high resolution both in space and time. The FPGA is responsible for the spatial resolution enhancement of the high-frame-rate samples using super-resolution (SR) techniques in real time. To achieve it, this article proposes utilizing the Iterative Back Projection (IBP) SR algorithm. The original IBP method is modified to account for the presence of noise, leading to an algorithm more robust to noise. An FPGA implementation of this algorithm is presented. The proposed architecture can serve as a general purpose real-time resolution enhancement system, and its performance is evaluated under various noise levels.

References

  1. Angelopoulou, M. E., Bouganis, C.-S., and Cheung, P. Y. K. 2008a. Video enhancement on an adaptive image sensor. In Proceedings of the IEEE International Conference on Image Processing (ICIP). 681--684.Google ScholarGoogle Scholar
  2. Angelopoulou, M. E., Bouganis, C.-S., Cheung, P. Y. K., and Constantinides, G. A. 2008b. FPGA-based real-time super-resolution on an adaptive image sensor. In Proceedings of the International Workshop on Applied Reconfigurable Computing (ARC). 125--136. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Arulampalam, M. S., Maskell, S., Gordon, N., and Clapp, T. 2002. A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans. Sig. Proc. 50, 2, 174--188. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Baker, S. and Kanade, T. 2002. Limits on super-resolution and how to break them. IEEE Trans. Patt. Anal. Mach. Intell. 24, 9, 1167--1183. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Ben-Ezra, M. and Nayar, S. K. 2004. Motion-based motion deblurring. IEEE Trans. Patt. Anal. Mach. Intell. 26, 6, 689--698. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Bonato, V., Marques, E., and Constantinides, G. A. 2007. A floating-point extended kalman filter implementation for autonomous mobile robots. In Proceedings of the International Conference on Field Programmable Logic and Applications (FPL). 576--579.Google ScholarGoogle Scholar
  7. Bouguet, J.-Y. 2002. Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of the Algorithm. Microprocessor Research Labs, Intel Corporation.Google ScholarGoogle Scholar
  8. Brown, L. G. 1992. A survey of image registration techniques. ACM Comput. Surv. 24, 4, 325--376. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Chen, T., Catrysse, P., Gamal, A. E., and Wandell, B. 2000. How small should pixel size be? In Proceedings of the SPIE Sensors and Camera Systems for Scientific, Industrial and Digital Photography Applications. Vol. 3965. 451--459.Google ScholarGoogle Scholar
  10. Constandinou, T. G., Degenaar, P., and Toumazou, C. 2006. An adaptable foveating vision chip. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS). 3566--3569.Google ScholarGoogle Scholar
  11. Farrell, J., Xiao, F., and Kavusi, S. 2006. Resolution and light sensitivity tradeoff with pixel size. In Proceedings of the SPIE Electronic Imaging ’06 Conference. Vol. 6069. 211--218.Google ScholarGoogle Scholar
  12. Farsiu, S., Robinson, D., Elad, M., and Milanfar, P. 2004. Advances and challenges in super-resolution. Int. J. Imag. Syst. Tech. 14, 2, 47--57.Google ScholarGoogle ScholarCross RefCross Ref
  13. Foveon, Inc. 2007. Foveon X3 Image Sensor. www.foveon.com.Google ScholarGoogle Scholar
  14. Gamal, A. E. and Eltoukhy, H. 2005. CMOS image sensors. IEEE Circ. Devic. Mag. 21, 3, 6--20.Google ScholarGoogle ScholarCross RefCross Ref
  15. Irani, M. and Peleg, S. 1991. Improving resolution by image registration. Graph. Mod. Image Proc. 53, 3, 231--239. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Liu, Y., Bouganis, C.-S., and Cheung, P. Y. K. 2007. Efficient mapping of a Kalman filter into an FPGA using Taylor expansion. In Proceedings of the International Conference on Field Programmable Logic and Applications (FPL). 345--350.Google ScholarGoogle Scholar
  17. Mendis, S., Kemeny, S., Gee, R., Pain, B., Staller, C., Kim, Q., and Fossum, E. 1997. CMOS active pixel image sensors for highly integrated imaging systems. IEEE J. Solid-State Circ. 32, 2, 187--197.Google ScholarGoogle ScholarCross RefCross Ref
  18. Park, S. C., Park, M. K., and Kang, M. G. 2003. Super-resolution image reconstruction: a technical overview. IEEE Sig. Proc. Mag. 20, 3, 21--36.Google ScholarGoogle ScholarCross RefCross Ref
  19. Shechtman, E., Caspi, Y., and Irani, M. 2005. Space-time super-resolution. IEEE Trans. Patt. Anal. Mach. Intell. 27, 4, 531--545. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Shi, J. and Tomasi, C. 1994. Good features to track. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 593--600.Google ScholarGoogle Scholar
  21. Sroubek, F., Cristobal, G., and Flusser, J. 2007. A unified approach to super-resolution and multichannel blind deconvolution. IEEE Trans. Image Proc. 16, 9, 2322--2332. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Stark, H. and Oskoui, P. 1989. High-resolution image recovery from image-plane arrays, using convex projections. J. Opt. Soc. Amer. A 6, 11, 1715--1726.Google ScholarGoogle ScholarCross RefCross Ref
  23. Stern, A., Porat, Y., Ben-Dor, A., and Kopeika, N. S. 2001. Enhanced-resolution image restoration from a sequence of low-frequency vibrated images by use of convex projections. Appl. Opt. 40, 26, 4706--4715.Google ScholarGoogle ScholarCross RefCross Ref
  24. Thrun, S., Burgard, W., and Fox, D. 2005. Probabilistic Robotics. MIT Press, Cambridge, MA.Google ScholarGoogle Scholar
  25. Tian, H., Fowler, B., and Gamal, A. E. 2001. Analysis of temporal noise in CMOS photodiode active pixel sensor. IEEE J. Solid-State Circ. 36, 1, 92--101.Google ScholarGoogle ScholarCross RefCross Ref
  26. Wan, E. A. and Merwe, R. V. D. 2000. The unscented Kalman filter for nonlinear estimation. In Proceedings of the IEEE Adaptive Systems for Signal Processing, Communications, and Control Symposium (AS-SPCC). 153--158.Google ScholarGoogle Scholar
  27. Welch, G. and Bishop, G. 2006. An introduction to the Kalman filter. http://www.cs.unc.edu/welch/media/pdf/kalman_intro.pdf.Google ScholarGoogle Scholar
  28. Youla, D. C. and Webb, H. 1982. Image restoration by the method of convex projections: part 1-theory. IEEE Trans. Med. Imag. MI-1, 2, 81--94.Google ScholarGoogle ScholarCross RefCross Ref
  29. Zitová, B. and Flusser, J. 2003. Image registration methods: A survey. Imag. Vis. Comput. 21, 11, 977--1000.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Robust Real-Time Super-Resolution on FPGA and an Application to Video Enhancement

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Reconfigurable Technology and Systems
        ACM Transactions on Reconfigurable Technology and Systems  Volume 2, Issue 4
        September 2009
        134 pages
        ISSN:1936-7406
        EISSN:1936-7414
        DOI:10.1145/1575779
        Issue’s Table of Contents

        Copyright © 2009 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 1 September 2009
        • Accepted: 1 October 2008
        • Revised: 1 September 2008
        • Received: 1 May 2008
        Published in trets Volume 2, Issue 4

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!