Abstract
In this article, we introduce an image/video restoration approach by utilizing the high-dimensional similarity in images/videos. After grouping similar patches from neighboring frames, we propose to build a multiplanar autoregressive (AR) model to exploit the correlation in cross-dimensional planes of the patch group, which has long been neglected by previous AR models. To further utilize the nonlocal self-similarity in images/videos, a joint multiplanar AR and low-rank based approach is proposed (MARLow) to reconstruct patch groups more effectively. Moreover, for video restoration, the temporal smoothness of the restored video is constrained by the Markov random field (MRF), where MRF encodes a priori knowledge about consistency of patches from neighboring frames. Specifically, we treat different restoration results (from different patch groups) of a certain patch as labels of an MRF, and temporal consistency among these restored patches is imposed. The proposed method is also suitable for other restoration applications such as interpolation and text removal. Extensive experimental results demonstrate that the proposed approach obtains encouraging performance comparing with state-of-the-art methods.
- Adobe Research. n.d. Content-Aware Fill. Retrieved October 23, 2019 from https://research.adobe.com/project/content-aware-fill/.Google Scholar
- Y. Boykov, O. Veksler, and R. Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11 (Nov. 2001), 1222--1239. DOI:https://doi.org/10.1109/34.969114Google Scholar
Digital Library
- Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2005. A review of image denoising algorithms, with a new one. Multiscale Modeling 8 Simulation 4, 2 (2005), 490--530.Google Scholar
- Jian-Feng Cai, Emmanuel J. Candès, and Zuowei Shen. 2010. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization 20, 4 (2010), 1956--1982.Google Scholar
Cross Ref
- Emmanuel J. Candès and Benjamin Recht. 2009. Exact matrix completion via convex optimization. Foundations of Computational Mathematics 9, 6 (2009), 717--772. DOI:https://doi.org/10.1007/s10208-009-9045-5Google Scholar
Cross Ref
- Y.-L. Chen, C.-T. Hsu, and H.-Y. M. Liao. 2014. Simultaneous tensor decomposition and completion using factor priors. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 3 (March 2014), 577--591. DOI:https://doi.org/10.1109/TPAMI.2013.164Google Scholar
Digital Library
- G. Chierchia, N. Pustelnik, B. Pesquet-Popescu, and J.-C. Pesquet. 2014. A nonlocal structure tensor-based approach for multicomponent image recovery problems. IEEE Transactions on Image Processing 23, 12 (Dec. 2014), 5531--5544. DOI:https://doi.org/10.1109/TIP.2014.2364141Google Scholar
Cross Ref
- K. Dabov, A. Foi, and K. Egiazarian. 2007. Video denoising by sparse 3D transform-domain collaborative filtering. In Proceedings of the European Signal Processing Conference. 145--149.Google Scholar
- K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing 16, 8 (Aug. 2007), 2080--2095. DOI:https://doi.org/10.1109/TIP.2007.901238Google Scholar
Cross Ref
- Weisheng Dong, Guangming Shi, and Xin Li. 2013. Nonlocal image restoration with bilateral variance estimation: A low-rank approach. IEEE Transactions on Image Processing 22, 2 (Feb. 2013), 700--711. DOI:https://doi.org/10.1109/TIP.2012.2221729Google Scholar
Digital Library
- W. Dong, L. Zhang, R. Lukac, and G. Shi. 2013. Sparse representation based image interpolation with nonlocal autoregressive modeling. IEEE Transactions on Image Processing 22, 4 (April 2013), 1382--1394. DOI:https://doi.org/10.1109/TIP.2012.2231086Google Scholar
Digital Library
- J.-J. Fadili, J.-L. Starck, and F. Murtagh. 2009. Inpainting and zooming using sparse representations. Computer Journal 52, 1 (2009), 64--79.Google Scholar
Digital Library
- W. B. Goh, M. N. Chong, S. Kalra, and D. Krishnan. 1996. Bi-directional 3D auto-regressive model approach to motion picture restoration. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP ’96), Vol. 4. 2275--2278. DOI:https://doi.org/10.1109/ICASSP.1996.545876Google Scholar
Digital Library
- Mohammad Golbabaee and Pierre Vandergheynst. 2012. Hyperspectral image compressed sensing via low-rank and joint-sparse matrix recovery. In Proceedings of the 2012 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’12). IEEE, Los Alamitos, CA, 2741--2744.Google Scholar
Cross Ref
- Liangtian He and Yilun Wang. 2014. Iterative support detection-based split Bregman method for wavelet frame-based image inpainting. IEEE Transactions on Image Processing 23, 12 (Dec. 2014), 5470--5485. DOI:https://doi.org/10.1109/TIP.2014.2362051Google Scholar
Cross Ref
- F. Heide, W. Heidrich, and G. Wetzstein. 2015. Fast and flexible convolutional sparse coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 5135--5143. DOI:https://doi.org/10.1109/CVPR.2015.7299149Google Scholar
- Wenrui Hu, Dacheng Tao, Wensheng Zhang, Yuan Xie, and Yehui Yang. 2015. A new low-rank tensor model for video completion. arXiv:1509.02027.Google Scholar
- Yao Hu, Debing Zhang, Jieping Ye, Xuelong Li, and Xiaofei He. 2013. Fast and accurate matrix completion via truncated nuclear norm regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 9 (2013), 2117--2130. DOI:https://doi.org/10.1109/TPAMI.2012.271Google Scholar
Digital Library
- Yao Hu, Chen Zhao, Deng Cai, Xiaofei He, and Xuelong Li. 2016. Atom decomposition with adaptive basis selection strategy for matrix completion. ACM Transactions on Multimedia Computing, Communications, and Applications 12, 3 (June 2016), Article 43, 25 pages. DOI:https://doi.org/10.1145/2903716Google Scholar
Digital Library
- J. B. Huang, A. Singh, and N. Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 5197--5206. DOI:https://doi.org/10.1109/CVPR.2015.7299156Google Scholar
- V. Jakhetiya, W. Lin, S. P. Jaiswal, S. C. Guntuku, and O. C. Au. 2017. Maximum a posterior and perceptually motivated reconstruction algorithm: A generic framework. IEEE Transactions on Multimedia 19, 1 (Jan. 2017), 93--106.Google Scholar
Digital Library
- Hui Ji, Sibin Huang, Zuowei Shen, and Yuhong Xu. 2011. Robust video restoration by joint sparse and low rank matrix approximation. SIAM Journal on Imaging Sciences 4, 4 (2011), 1122--1142.Google Scholar
Digital Library
- Hui Ji, Chaoqiang Liu, Zuowei Shen, and Yuhong Xu. 2010. Robust video denoising using low rank matrix completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’10). 1791--1798. DOI:https://doi.org/10.1109/CVPR.2010.5539849Google Scholar
Cross Ref
- A. Kokaram and P. Rayner. 1994. Detection and interpolation of replacement noise in motion picture sequences using 3D autoregressive modelling. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS’94), Vol. 3. 21--24. DOI:https://doi.org/10.1109/ISCAS.1994.409091Google Scholar
Cross Ref
- Chao Li, Lili Guo, and Andrzej Cichocki. 2014. Multi-tensor completion for estimating missing values in video data. In Proceedings of the 2014 Joint 7th International Conference on Soft Computing and Intelligent Systems (SCIS’14) and the 15th International Symposium on Advanced Intelligent Systems (ISIS’14). IEEE, Los Alamitos, CA, 1339--1342.Google Scholar
Cross Ref
- Chao Li, Qibin Zhao, Junhua Li, Andrzej Cichocki, and Lili Guo. 2015. Multi-Tensor Completion with Common Structures. Retrieved October 23, 2019 from http://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/9437.Google Scholar
Cross Ref
- Mading Li, Jiaying Liu, Zhiwei Xiong, Xiaoyan Sun, and Zongming Guo. 2016. MARLow: A joint multiplanar autoregressive and low-rank approach for image completion. In Proceedings of the European Conference on Computer Vision (ECCV’16). 819--834. DOI:https://doi.org/10.1007/978-3-319-46478-7_50Google Scholar
Cross Ref
- X. Li and M. T. Orchard. 2001. New edge-directed interpolation. IEEE Transactions on Image Processing 10, 10 (Oct. 2001), 1521--1527. DOI:https://doi.org/10.1109/83.951537Google Scholar
Digital Library
- R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia. 2015. Video super-resolution via deep draft-ensemble learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). 531--539. DOI:https://doi.org/10.1109/ICCV.2015.68Google Scholar
Digital Library
- Ji Liu, P. Musialski, P. Wonka, and J. Ye. 2009. Tensor completion for estimating missing values in visual data. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’09). 2114--2121. DOI:https://doi.org/10.1109/ICCV.2009.5459463Google Scholar
- J. Liu, P. Musialski, P. Wonka, and J. Ye. 2013. Tensor completion for estimating missing values in visual data. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1 (2013), 208--220. DOI:https://doi.org/10.1109/TPAMI.2012.39Google Scholar
Digital Library
- Luca Lorenzi, Farid Melgani, and Grégoire Mercier. 2013. Missing-area reconstruction in multispectral images under a compressive sensing perspective. IEEE Transactions on Geoscience and Remote Sensing 51, 7 (2013), 3998--4008.Google Scholar
Cross Ref
- M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian. 2012. Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms. IEEE Transactions on Image Processing 21, 9 (Sept. 2012), 3952--3966. DOI:https://doi.org/10.1109/TIP.2012.2199324Google Scholar
Digital Library
- M. Maggioni, V. Katkovnik, K. Egiazarian, and A. Foi. 2013. Nonlocal transform-domain filter for volumetric data denoising and reconstruction. IEEE Transactions on Image Processing 22, 1 (Jan. 2013), 119--133. DOI:https://doi.org/10.1109/TIP.2012.2210725Google Scholar
Digital Library
- J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. 2009. Non-local sparse models for image restoration. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’09). 2272--2279. DOI:https://doi.org/10.1109/ICCV.2009.5459452Google Scholar
- J. Mairal, M. Elad, and G. Sapiro. 2008. Sparse representation for color image restoration. IEEE Transactions on Image Processing 17, 1 (Jan. 2008), 53--69.Google Scholar
Digital Library
- D. Martin, C. Fowlkes, D. Tal, and J. Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the International Conference on Computer Vision (ICCV’01), Vol. 2. 416--423.Google Scholar
- S. Ono, T. Miyata, and I. Yamada. 2014. Cartoon-texture image decomposition using blockwise low-rank texture characterization. IEEE Transactions on Image Processing 23, 3 (March 2014), 1128--1142. DOI:https://doi.org/10.1109/TIP.2014.2299067Google Scholar
Digital Library
- Holger Rauhut, Karin Schnass, and Pierre Vandergheynst. 2008. Compressed sensing and redundant dictionaries. IEEE Transactions on Information Theory 54, 5 (2008), 2210--2219.Google Scholar
Digital Library
- Justin Romberg. 2009. Compressive sensing by random convolution. SIAM Journal on Imaging Sciences 2, 4 (2009), 1098--1128.Google Scholar
Digital Library
- Stefan Roth and Michael J. Black. 2009. Fields of experts. International Journal of Computer Vision 82, 2 (2009), 205--229.Google Scholar
Digital Library
- Tijana Ružić, Bruno Cornelis, Ljiljana Platiša, Aleksandra Pižurica, Ann Dooms, Wilfried Philips, Maximiliaan Martens, Marc De Mey, and Ingrid Daubechies. 2011. Virtual restoration of the Ghent Altarpiece using crack detection and inpainting. In Proceedings of the International Conference on Advanced Concepts for Intelligent Vision Systems. 417--428.Google Scholar
Cross Ref
- Huanfeng Shen, Xinghua Li, Liangpei Zhang, Dacheng Tao, and Chao Zeng. 2014. Compressed sensing-based inpainting of aqua moderate resolution imaging spectroradiometer band 6 using adaptive spectrum-weighted sparse Bayesian dictionary learning. IEEE Transactions on Geoscience and Remote Sensing 52, 2 (2014), 894--906.Google Scholar
Cross Ref
- L. Sun and J. Hays. 2012. Super-resolution from Internet-scale scene matching. In Proceedings of the 2012 IEEE International Conference on Computational Photography (ICCP’12). 1--12. DOI:https://doi.org/10.1109/ICCPhot.2012.6215221Google Scholar
- Kenny Kal Vin Toh and Nor Ashidi Mat Isa. 2010. Noise adaptive fuzzy switching median filter for salt-and-pepper noise reduction. IEEE Signal Processing Letters 17, 3 (2010), 281--284.Google Scholar
Cross Ref
- Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). 9446--9454.Google Scholar
- H. Wang, Y. Cen, Z. He, R. Zhao, Y. Cen, and F. Zhang. 2017. Robust generalized low-rank decomposition of multimatrices for image recovery. IEEE Transactions on Multimedia 19, 5 (May 2017), 969--983.Google Scholar
Digital Library
- Hua Wang, Feiping Nie, and Heng Huang. 2014. Low-Rank Tensor Completion with Spatio-Temporal Consistency. Retrieved October 23, 2019 from http://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8580.Google Scholar
Cross Ref
- Debing Zhang, Yao Hu, Jieping Ye, Xuelong Li, and Xiaofei He. 2012. Matrix completion by truncated nuclear norm regularization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). 2192--2199. DOI:https://doi.org/10.1109/CVPR.2012.6247927Google Scholar
- Jian Zhang, Debin Zhao, and Wen Gao. 2014. Group-based sparse representation for image restoration. IEEE Transactions on Image Processing 23, 8 (Aug. 2014), 3336--3351. DOI:https://doi.org/10.1109/TIP.2014.2323127Google Scholar
Cross Ref
- Jian Zhang, Debin Zhao, Ruiqin Xiong, Siwei Ma, and Wen Gao. 2014. Image restoration using joint statistical modeling in a space-transform domain. IEEE Transactions on Circuits and Systems for Video Technology 24, 6 (June 2014), 915--928. DOI:https://doi.org/10.1109/TCSVT.2014.2302380Google Scholar
Cross Ref
- Peixuan Zhang and Fang Li. 2014. A new adaptive weighted mean filter for removing salt-and-pepper noise. IEEE Signal Processing Letters 21, 10 (2014), 1280--1283.Google Scholar
Cross Ref
- Xiangjun Zhang and Xiaolin Wu. 2008. Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation. IEEE Transactions on Image Processing 17, 6 (June 2008), 887--896. DOI:https://doi.org/10.1109/TIP.2008.924279Google Scholar
Digital Library
- Z. Zhang, G. Ely, S. Aeron, N, Hao, and M. Kilmer. 2014. Novel methods for multilinear data completion and de-noising based on tensor-SVD. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). 3842--3849. DOI:https://doi.org/10.1109/CVPR.2014.485Google Scholar
Digital Library
- Dizhi Zhou, Xinyue Shen, and Wenjie Dong. 2012. Image zooming using directional cubic convolution interpolation. IET Image Processing 6, 6 (2012), 627--634.Google Scholar
Cross Ref
- M. Zhou, H. Chen, J. Paisley, L. Ren, L. Li, Z. Xing, D. Dunson, G. Sapiro, and L. Carin. 2012. Nonparametric Bayesian dictionary learning for analysis of noisy and incomplete images. IEEE Transactions on Image Processing 21, 1 (Jan. 2012), 130--144. DOI:https://doi.org/10.1109/TIP.2011.2160072Google Scholar
Digital Library
Index Terms
Image/Video Restoration via Multiplanar Autoregressive Model and Low-Rank Optimization
Recommendations
Image Restoration Based on 3-D Autoregressive Model via Low-Rank Minimization
DCC '15: Proceedings of the 2015 Data Compression ConferenceDue to all kinds of need of customers and the complicated transmitting environment of digital image and video resources, numerous practical applications emerge, e.g. Image in painting, interpolation, super-resolution and the removal of salt and pepper ...
Single image super-resolution via self-similarity and low-rank matrix recovery
We propose a novel single-image super resolution (SISR) approach using self-similarity of image and the low-rank matrix recovery (LRMR). The method performs multiple upsampling steps with relatively small magnification factors to recover a desired high ...
Robust Image Restoration via Reweighted Low-Rank Matrix Recovery
MMM 2014: Proceedings of the 20th Anniversary International Conference on MultiMedia Modeling - Volume 8325In this paper, we propose a robust image restoration method via reweighted low-rank matrix recovery. In the literature, Principal Component Pursuit (PCP) solves low-rank matrix recovery problem via a convex program of mixed nuclear norm and ℓ1 norm. ...






Comments