skip to main content
research-article

Image/Video Restoration via Multiplanar Autoregressive Model and Low-Rank Optimization

Authors Info & Claims
Published:16 December 2019Publication History
Skip Abstract Section

Abstract

In this article, we introduce an image/video restoration approach by utilizing the high-dimensional similarity in images/videos. After grouping similar patches from neighboring frames, we propose to build a multiplanar autoregressive (AR) model to exploit the correlation in cross-dimensional planes of the patch group, which has long been neglected by previous AR models. To further utilize the nonlocal self-similarity in images/videos, a joint multiplanar AR and low-rank based approach is proposed (MARLow) to reconstruct patch groups more effectively. Moreover, for video restoration, the temporal smoothness of the restored video is constrained by the Markov random field (MRF), where MRF encodes a priori knowledge about consistency of patches from neighboring frames. Specifically, we treat different restoration results (from different patch groups) of a certain patch as labels of an MRF, and temporal consistency among these restored patches is imposed. The proposed method is also suitable for other restoration applications such as interpolation and text removal. Extensive experimental results demonstrate that the proposed approach obtains encouraging performance comparing with state-of-the-art methods.

References

  1. Adobe Research. n.d. Content-Aware Fill. Retrieved October 23, 2019 from https://research.adobe.com/project/content-aware-fill/.Google ScholarGoogle Scholar
  2. Y. Boykov, O. Veksler, and R. Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11 (Nov. 2001), 1222--1239. DOI:https://doi.org/10.1109/34.969114Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2005. A review of image denoising algorithms, with a new one. Multiscale Modeling 8 Simulation 4, 2 (2005), 490--530.Google ScholarGoogle Scholar
  4. Jian-Feng Cai, Emmanuel J. Candès, and Zuowei Shen. 2010. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization 20, 4 (2010), 1956--1982.Google ScholarGoogle ScholarCross RefCross Ref
  5. Emmanuel J. Candès and Benjamin Recht. 2009. Exact matrix completion via convex optimization. Foundations of Computational Mathematics 9, 6 (2009), 717--772. DOI:https://doi.org/10.1007/s10208-009-9045-5Google ScholarGoogle ScholarCross RefCross Ref
  6. Y.-L. Chen, C.-T. Hsu, and H.-Y. M. Liao. 2014. Simultaneous tensor decomposition and completion using factor priors. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 3 (March 2014), 577--591. DOI:https://doi.org/10.1109/TPAMI.2013.164Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. G. Chierchia, N. Pustelnik, B. Pesquet-Popescu, and J.-C. Pesquet. 2014. A nonlocal structure tensor-based approach for multicomponent image recovery problems. IEEE Transactions on Image Processing 23, 12 (Dec. 2014), 5531--5544. DOI:https://doi.org/10.1109/TIP.2014.2364141Google ScholarGoogle ScholarCross RefCross Ref
  8. K. Dabov, A. Foi, and K. Egiazarian. 2007. Video denoising by sparse 3D transform-domain collaborative filtering. In Proceedings of the European Signal Processing Conference. 145--149.Google ScholarGoogle Scholar
  9. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing 16, 8 (Aug. 2007), 2080--2095. DOI:https://doi.org/10.1109/TIP.2007.901238Google ScholarGoogle ScholarCross RefCross Ref
  10. Weisheng Dong, Guangming Shi, and Xin Li. 2013. Nonlocal image restoration with bilateral variance estimation: A low-rank approach. IEEE Transactions on Image Processing 22, 2 (Feb. 2013), 700--711. DOI:https://doi.org/10.1109/TIP.2012.2221729Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. W. Dong, L. Zhang, R. Lukac, and G. Shi. 2013. Sparse representation based image interpolation with nonlocal autoregressive modeling. IEEE Transactions on Image Processing 22, 4 (April 2013), 1382--1394. DOI:https://doi.org/10.1109/TIP.2012.2231086Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. J.-J. Fadili, J.-L. Starck, and F. Murtagh. 2009. Inpainting and zooming using sparse representations. Computer Journal 52, 1 (2009), 64--79.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. W. B. Goh, M. N. Chong, S. Kalra, and D. Krishnan. 1996. Bi-directional 3D auto-regressive model approach to motion picture restoration. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP ’96), Vol. 4. 2275--2278. DOI:https://doi.org/10.1109/ICASSP.1996.545876Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Mohammad Golbabaee and Pierre Vandergheynst. 2012. Hyperspectral image compressed sensing via low-rank and joint-sparse matrix recovery. In Proceedings of the 2012 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’12). IEEE, Los Alamitos, CA, 2741--2744.Google ScholarGoogle ScholarCross RefCross Ref
  15. Liangtian He and Yilun Wang. 2014. Iterative support detection-based split Bregman method for wavelet frame-based image inpainting. IEEE Transactions on Image Processing 23, 12 (Dec. 2014), 5470--5485. DOI:https://doi.org/10.1109/TIP.2014.2362051Google ScholarGoogle ScholarCross RefCross Ref
  16. F. Heide, W. Heidrich, and G. Wetzstein. 2015. Fast and flexible convolutional sparse coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 5135--5143. DOI:https://doi.org/10.1109/CVPR.2015.7299149Google ScholarGoogle Scholar
  17. Wenrui Hu, Dacheng Tao, Wensheng Zhang, Yuan Xie, and Yehui Yang. 2015. A new low-rank tensor model for video completion. arXiv:1509.02027.Google ScholarGoogle Scholar
  18. Yao Hu, Debing Zhang, Jieping Ye, Xuelong Li, and Xiaofei He. 2013. Fast and accurate matrix completion via truncated nuclear norm regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 9 (2013), 2117--2130. DOI:https://doi.org/10.1109/TPAMI.2012.271Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Yao Hu, Chen Zhao, Deng Cai, Xiaofei He, and Xuelong Li. 2016. Atom decomposition with adaptive basis selection strategy for matrix completion. ACM Transactions on Multimedia Computing, Communications, and Applications 12, 3 (June 2016), Article 43, 25 pages. DOI:https://doi.org/10.1145/2903716Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. J. B. Huang, A. Singh, and N. Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 5197--5206. DOI:https://doi.org/10.1109/CVPR.2015.7299156Google ScholarGoogle Scholar
  21. V. Jakhetiya, W. Lin, S. P. Jaiswal, S. C. Guntuku, and O. C. Au. 2017. Maximum a posterior and perceptually motivated reconstruction algorithm: A generic framework. IEEE Transactions on Multimedia 19, 1 (Jan. 2017), 93--106.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Hui Ji, Sibin Huang, Zuowei Shen, and Yuhong Xu. 2011. Robust video restoration by joint sparse and low rank matrix approximation. SIAM Journal on Imaging Sciences 4, 4 (2011), 1122--1142.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Hui Ji, Chaoqiang Liu, Zuowei Shen, and Yuhong Xu. 2010. Robust video denoising using low rank matrix completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’10). 1791--1798. DOI:https://doi.org/10.1109/CVPR.2010.5539849Google ScholarGoogle ScholarCross RefCross Ref
  24. A. Kokaram and P. Rayner. 1994. Detection and interpolation of replacement noise in motion picture sequences using 3D autoregressive modelling. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS’94), Vol. 3. 21--24. DOI:https://doi.org/10.1109/ISCAS.1994.409091Google ScholarGoogle ScholarCross RefCross Ref
  25. Chao Li, Lili Guo, and Andrzej Cichocki. 2014. Multi-tensor completion for estimating missing values in video data. In Proceedings of the 2014 Joint 7th International Conference on Soft Computing and Intelligent Systems (SCIS’14) and the 15th International Symposium on Advanced Intelligent Systems (ISIS’14). IEEE, Los Alamitos, CA, 1339--1342.Google ScholarGoogle ScholarCross RefCross Ref
  26. Chao Li, Qibin Zhao, Junhua Li, Andrzej Cichocki, and Lili Guo. 2015. Multi-Tensor Completion with Common Structures. Retrieved October 23, 2019 from http://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/9437.Google ScholarGoogle ScholarCross RefCross Ref
  27. Mading Li, Jiaying Liu, Zhiwei Xiong, Xiaoyan Sun, and Zongming Guo. 2016. MARLow: A joint multiplanar autoregressive and low-rank approach for image completion. In Proceedings of the European Conference on Computer Vision (ECCV’16). 819--834. DOI:https://doi.org/10.1007/978-3-319-46478-7_50Google ScholarGoogle ScholarCross RefCross Ref
  28. X. Li and M. T. Orchard. 2001. New edge-directed interpolation. IEEE Transactions on Image Processing 10, 10 (Oct. 2001), 1521--1527. DOI:https://doi.org/10.1109/83.951537Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia. 2015. Video super-resolution via deep draft-ensemble learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). 531--539. DOI:https://doi.org/10.1109/ICCV.2015.68Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Ji Liu, P. Musialski, P. Wonka, and J. Ye. 2009. Tensor completion for estimating missing values in visual data. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’09). 2114--2121. DOI:https://doi.org/10.1109/ICCV.2009.5459463Google ScholarGoogle Scholar
  31. J. Liu, P. Musialski, P. Wonka, and J. Ye. 2013. Tensor completion for estimating missing values in visual data. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1 (2013), 208--220. DOI:https://doi.org/10.1109/TPAMI.2012.39Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Luca Lorenzi, Farid Melgani, and Grégoire Mercier. 2013. Missing-area reconstruction in multispectral images under a compressive sensing perspective. IEEE Transactions on Geoscience and Remote Sensing 51, 7 (2013), 3998--4008.Google ScholarGoogle ScholarCross RefCross Ref
  33. M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian. 2012. Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms. IEEE Transactions on Image Processing 21, 9 (Sept. 2012), 3952--3966. DOI:https://doi.org/10.1109/TIP.2012.2199324Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. M. Maggioni, V. Katkovnik, K. Egiazarian, and A. Foi. 2013. Nonlocal transform-domain filter for volumetric data denoising and reconstruction. IEEE Transactions on Image Processing 22, 1 (Jan. 2013), 119--133. DOI:https://doi.org/10.1109/TIP.2012.2210725Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. 2009. Non-local sparse models for image restoration. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’09). 2272--2279. DOI:https://doi.org/10.1109/ICCV.2009.5459452Google ScholarGoogle Scholar
  36. J. Mairal, M. Elad, and G. Sapiro. 2008. Sparse representation for color image restoration. IEEE Transactions on Image Processing 17, 1 (Jan. 2008), 53--69.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. D. Martin, C. Fowlkes, D. Tal, and J. Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the International Conference on Computer Vision (ICCV’01), Vol. 2. 416--423.Google ScholarGoogle Scholar
  38. S. Ono, T. Miyata, and I. Yamada. 2014. Cartoon-texture image decomposition using blockwise low-rank texture characterization. IEEE Transactions on Image Processing 23, 3 (March 2014), 1128--1142. DOI:https://doi.org/10.1109/TIP.2014.2299067Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Holger Rauhut, Karin Schnass, and Pierre Vandergheynst. 2008. Compressed sensing and redundant dictionaries. IEEE Transactions on Information Theory 54, 5 (2008), 2210--2219.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Justin Romberg. 2009. Compressive sensing by random convolution. SIAM Journal on Imaging Sciences 2, 4 (2009), 1098--1128.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Stefan Roth and Michael J. Black. 2009. Fields of experts. International Journal of Computer Vision 82, 2 (2009), 205--229.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Tijana Ružić, Bruno Cornelis, Ljiljana Platiša, Aleksandra Pižurica, Ann Dooms, Wilfried Philips, Maximiliaan Martens, Marc De Mey, and Ingrid Daubechies. 2011. Virtual restoration of the Ghent Altarpiece using crack detection and inpainting. In Proceedings of the International Conference on Advanced Concepts for Intelligent Vision Systems. 417--428.Google ScholarGoogle ScholarCross RefCross Ref
  43. Huanfeng Shen, Xinghua Li, Liangpei Zhang, Dacheng Tao, and Chao Zeng. 2014. Compressed sensing-based inpainting of aqua moderate resolution imaging spectroradiometer band 6 using adaptive spectrum-weighted sparse Bayesian dictionary learning. IEEE Transactions on Geoscience and Remote Sensing 52, 2 (2014), 894--906.Google ScholarGoogle ScholarCross RefCross Ref
  44. L. Sun and J. Hays. 2012. Super-resolution from Internet-scale scene matching. In Proceedings of the 2012 IEEE International Conference on Computational Photography (ICCP’12). 1--12. DOI:https://doi.org/10.1109/ICCPhot.2012.6215221Google ScholarGoogle Scholar
  45. Kenny Kal Vin Toh and Nor Ashidi Mat Isa. 2010. Noise adaptive fuzzy switching median filter for salt-and-pepper noise reduction. IEEE Signal Processing Letters 17, 3 (2010), 281--284.Google ScholarGoogle ScholarCross RefCross Ref
  46. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). 9446--9454.Google ScholarGoogle Scholar
  47. H. Wang, Y. Cen, Z. He, R. Zhao, Y. Cen, and F. Zhang. 2017. Robust generalized low-rank decomposition of multimatrices for image recovery. IEEE Transactions on Multimedia 19, 5 (May 2017), 969--983.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Hua Wang, Feiping Nie, and Heng Huang. 2014. Low-Rank Tensor Completion with Spatio-Temporal Consistency. Retrieved October 23, 2019 from http://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8580.Google ScholarGoogle ScholarCross RefCross Ref
  49. Debing Zhang, Yao Hu, Jieping Ye, Xuelong Li, and Xiaofei He. 2012. Matrix completion by truncated nuclear norm regularization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). 2192--2199. DOI:https://doi.org/10.1109/CVPR.2012.6247927Google ScholarGoogle Scholar
  50. Jian Zhang, Debin Zhao, and Wen Gao. 2014. Group-based sparse representation for image restoration. IEEE Transactions on Image Processing 23, 8 (Aug. 2014), 3336--3351. DOI:https://doi.org/10.1109/TIP.2014.2323127Google ScholarGoogle ScholarCross RefCross Ref
  51. Jian Zhang, Debin Zhao, Ruiqin Xiong, Siwei Ma, and Wen Gao. 2014. Image restoration using joint statistical modeling in a space-transform domain. IEEE Transactions on Circuits and Systems for Video Technology 24, 6 (June 2014), 915--928. DOI:https://doi.org/10.1109/TCSVT.2014.2302380Google ScholarGoogle ScholarCross RefCross Ref
  52. Peixuan Zhang and Fang Li. 2014. A new adaptive weighted mean filter for removing salt-and-pepper noise. IEEE Signal Processing Letters 21, 10 (2014), 1280--1283.Google ScholarGoogle ScholarCross RefCross Ref
  53. Xiangjun Zhang and Xiaolin Wu. 2008. Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation. IEEE Transactions on Image Processing 17, 6 (June 2008), 887--896. DOI:https://doi.org/10.1109/TIP.2008.924279Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Z. Zhang, G. Ely, S. Aeron, N, Hao, and M. Kilmer. 2014. Novel methods for multilinear data completion and de-noising based on tensor-SVD. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). 3842--3849. DOI:https://doi.org/10.1109/CVPR.2014.485Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Dizhi Zhou, Xinyue Shen, and Wenjie Dong. 2012. Image zooming using directional cubic convolution interpolation. IET Image Processing 6, 6 (2012), 627--634.Google ScholarGoogle ScholarCross RefCross Ref
  56. M. Zhou, H. Chen, J. Paisley, L. Ren, L. Li, Z. Xing, D. Dunson, G. Sapiro, and L. Carin. 2012. Nonparametric Bayesian dictionary learning for analysis of noisy and incomplete images. IEEE Transactions on Image Processing 21, 1 (Jan. 2012), 130--144. DOI:https://doi.org/10.1109/TIP.2011.2160072Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Image/Video Restoration via Multiplanar Autoregressive Model and Low-Rank Optimization

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!