Abstract
The high density image sensors of state-of-the-art imaging systems provide outputs with high spatial resolution, but require long exposure times. This limits their applicability, due to the motion blur effect. Recent technological advances have lead to adaptive image sensors that can combine several pixels together in real time to form a larger pixel. Larger pixels require shorter exposure times and produce high-frame-rate samples with reduced motion blur. This work proposes combining an FPGA with an adaptive image sensor to produce an output of high resolution both in space and time. The FPGA is responsible for the spatial resolution enhancement of the high-frame-rate samples using super-resolution (SR) techniques in real time. To achieve it, this article proposes utilizing the Iterative Back Projection (IBP) SR algorithm. The original IBP method is modified to account for the presence of noise, leading to an algorithm more robust to noise. An FPGA implementation of this algorithm is presented. The proposed architecture can serve as a general purpose real-time resolution enhancement system, and its performance is evaluated under various noise levels.
- Angelopoulou, M. E., Bouganis, C.-S., and Cheung, P. Y. K. 2008a. Video enhancement on an adaptive image sensor. In Proceedings of the IEEE International Conference on Image Processing (ICIP). 681--684.Google Scholar
- Angelopoulou, M. E., Bouganis, C.-S., Cheung, P. Y. K., and Constantinides, G. A. 2008b. FPGA-based real-time super-resolution on an adaptive image sensor. In Proceedings of the International Workshop on Applied Reconfigurable Computing (ARC). 125--136. Google Scholar
Digital Library
- Arulampalam, M. S., Maskell, S., Gordon, N., and Clapp, T. 2002. A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans. Sig. Proc. 50, 2, 174--188. Google Scholar
Digital Library
- Baker, S. and Kanade, T. 2002. Limits on super-resolution and how to break them. IEEE Trans. Patt. Anal. Mach. Intell. 24, 9, 1167--1183. Google Scholar
Digital Library
- Ben-Ezra, M. and Nayar, S. K. 2004. Motion-based motion deblurring. IEEE Trans. Patt. Anal. Mach. Intell. 26, 6, 689--698. Google Scholar
Digital Library
- Bonato, V., Marques, E., and Constantinides, G. A. 2007. A floating-point extended kalman filter implementation for autonomous mobile robots. In Proceedings of the International Conference on Field Programmable Logic and Applications (FPL). 576--579.Google Scholar
- Bouguet, J.-Y. 2002. Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of the Algorithm. Microprocessor Research Labs, Intel Corporation.Google Scholar
- Brown, L. G. 1992. A survey of image registration techniques. ACM Comput. Surv. 24, 4, 325--376. Google Scholar
Digital Library
- Chen, T., Catrysse, P., Gamal, A. E., and Wandell, B. 2000. How small should pixel size be? In Proceedings of the SPIE Sensors and Camera Systems for Scientific, Industrial and Digital Photography Applications. Vol. 3965. 451--459.Google Scholar
- Constandinou, T. G., Degenaar, P., and Toumazou, C. 2006. An adaptable foveating vision chip. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS). 3566--3569.Google Scholar
- Farrell, J., Xiao, F., and Kavusi, S. 2006. Resolution and light sensitivity tradeoff with pixel size. In Proceedings of the SPIE Electronic Imaging ’06 Conference. Vol. 6069. 211--218.Google Scholar
- Farsiu, S., Robinson, D., Elad, M., and Milanfar, P. 2004. Advances and challenges in super-resolution. Int. J. Imag. Syst. Tech. 14, 2, 47--57.Google Scholar
Cross Ref
- Foveon, Inc. 2007. Foveon X3 Image Sensor. www.foveon.com.Google Scholar
- Gamal, A. E. and Eltoukhy, H. 2005. CMOS image sensors. IEEE Circ. Devic. Mag. 21, 3, 6--20.Google Scholar
Cross Ref
- Irani, M. and Peleg, S. 1991. Improving resolution by image registration. Graph. Mod. Image Proc. 53, 3, 231--239. Google Scholar
Digital Library
- Liu, Y., Bouganis, C.-S., and Cheung, P. Y. K. 2007. Efficient mapping of a Kalman filter into an FPGA using Taylor expansion. In Proceedings of the International Conference on Field Programmable Logic and Applications (FPL). 345--350.Google Scholar
- Mendis, S., Kemeny, S., Gee, R., Pain, B., Staller, C., Kim, Q., and Fossum, E. 1997. CMOS active pixel image sensors for highly integrated imaging systems. IEEE J. Solid-State Circ. 32, 2, 187--197.Google Scholar
Cross Ref
- Park, S. C., Park, M. K., and Kang, M. G. 2003. Super-resolution image reconstruction: a technical overview. IEEE Sig. Proc. Mag. 20, 3, 21--36.Google Scholar
Cross Ref
- Shechtman, E., Caspi, Y., and Irani, M. 2005. Space-time super-resolution. IEEE Trans. Patt. Anal. Mach. Intell. 27, 4, 531--545. Google Scholar
Digital Library
- Shi, J. and Tomasi, C. 1994. Good features to track. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 593--600.Google Scholar
- Sroubek, F., Cristobal, G., and Flusser, J. 2007. A unified approach to super-resolution and multichannel blind deconvolution. IEEE Trans. Image Proc. 16, 9, 2322--2332. Google Scholar
Digital Library
- Stark, H. and Oskoui, P. 1989. High-resolution image recovery from image-plane arrays, using convex projections. J. Opt. Soc. Amer. A 6, 11, 1715--1726.Google Scholar
Cross Ref
- Stern, A., Porat, Y., Ben-Dor, A., and Kopeika, N. S. 2001. Enhanced-resolution image restoration from a sequence of low-frequency vibrated images by use of convex projections. Appl. Opt. 40, 26, 4706--4715.Google Scholar
Cross Ref
- Thrun, S., Burgard, W., and Fox, D. 2005. Probabilistic Robotics. MIT Press, Cambridge, MA.Google Scholar
- Tian, H., Fowler, B., and Gamal, A. E. 2001. Analysis of temporal noise in CMOS photodiode active pixel sensor. IEEE J. Solid-State Circ. 36, 1, 92--101.Google Scholar
Cross Ref
- Wan, E. A. and Merwe, R. V. D. 2000. The unscented Kalman filter for nonlinear estimation. In Proceedings of the IEEE Adaptive Systems for Signal Processing, Communications, and Control Symposium (AS-SPCC). 153--158.Google Scholar
- Welch, G. and Bishop, G. 2006. An introduction to the Kalman filter. http://www.cs.unc.edu/welch/media/pdf/kalman_intro.pdf.Google Scholar
- Youla, D. C. and Webb, H. 1982. Image restoration by the method of convex projections: part 1-theory. IEEE Trans. Med. Imag. MI-1, 2, 81--94.Google Scholar
Cross Ref
- Zitová, B. and Flusser, J. 2003. Image registration methods: A survey. Imag. Vis. Comput. 21, 11, 977--1000.Google Scholar
Cross Ref
Index Terms
Robust Real-Time Super-Resolution on FPGA and an Application to Video Enhancement
Recommendations
An FPGA-based design for real-time Super Resolution Reconstruction
ICDSC '18: Proceedings of the 12th International Conference on Distributed Smart CamerasSince several decades, the camera spatial resolution is gradually increasing with the CMOS technology evolution. The image sensors provide more and more pixels, generating new constraints for the suitable optics. As an alternative, promising solutions ...
Spatial-Spectral Deep Residual Network for Hyperspectral Image Super-Resolution
AbstractRecently, single hyperspectral image super-resolution (SR) methods based on deep learning have been extensively studied. However, there has been limited technical development focusing on single hyperspectral image super-resolution due to the high-...
Face image super-resolution through locality-induced support regression
In this paper we propose a novel face image super-resolution (SR) method named Locality-induced Support Regression (LiSR). Given a low-resolution (LR) input patch, we learn a mapping function between the local support LR and high-resolution (HR) patch ...






Comments