Abstract
Screen content images (SCIs) usually comprise various content types with sharp edges, in which artifacts or distortions can be effectively sensed by a vanilla structure similarity measurement in a full-reference manner. Nonetheless, almost all of the current state-of-the-art (SOTA) structure similarity metrics are “locally” formulated in a single-level manner, while the true human visual system (HVS) follows the multilevel manner; such mismatch could eventually prevent these metrics from achieving reliable quality assessment. To ameliorate this issue, this article advocates a novel solution to measure structure similarity “globally” from the perspective of sparse representation. To perform multilevel quality assessment in accordance with the real HVS, the abovementioned global metric will be integrated with the conventional local ones by resorting to the newly devised selective deep fusion network. To validate its efficacy and effectiveness, we have compared our method with 12 SOTA methods over two widely used large-scale public SCI datasets, and the quantitative results indicate that our method yields significantly higher consistency with subjective quality scores than the current leading works. Both the source code and data are also publicly available to gain widespread acceptance and facilitate new advancement and validation.
- Michal Aharon, Michael Elad, and Alfred Bruckstein. 2006. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54, 11 (2006), 4311–4322. Google Scholar
Digital Library
- Tsuishan Chang, Chunchi Chen, Ruling Liao, Chewei Kuo, and Wenhsiao Peng. 2014. Intra line copy for HEVC screen content coding. In Proceedings of the Asia-Pacific Signal & Information Processing Association, Summit & Conference. 1–8.Google Scholar
Cross Ref
- Chenglizhao Chen, Shuai Li, and Hong Qin. 2016. Robust salient motion detection in non-stationary videos via novel integrated strategies of spatio-temporal coherency clues and low-rank analysis. Pattern Recogn. 52 (2016), 410–432. Google Scholar
Digital Library
- Chenglizhao Chen, Shuai Li, Hong Qin, and Aimin Hao. 2015. Real-time and robust object tracking in video via low-rank coherency analysis in feature space. Pattern Recogn. 48, 9 (2015), 2885–2905. Google Scholar
Digital Library
- Chenglizhao Chen, Shuai Li, Hong Qin, and Aimin Hao. 2015. Structure-sensitive saliency detection via multilevel rank analysis in intrinsic feature space. IEEE Trans. Image Process. 24, 8 (2015), 2303–2316.Google Scholar
Digital Library
- Chenglizhao Chen, Shuai Li, Hong Qin, and Aimin Hao. 2016. Robust salient motion detection in non-stationary videos via novel integrated strategies of spatio-temporal coherency clues and low-rank analysis. Pattern Recogn. 52 (2016), 410–432. Google Scholar
Digital Library
- Chenglizhao Chen, Shuai Li, Hong Qin, Zhenkuan Pan, and Guowei Yang. 2018. Bi-level feature learning for video saliency detection. IEEE Trans. Multimedia. 20, 12 (2018), 3324–3336.Google Scholar
Digital Library
- Chenglizhao Chen, Shuai Li, Yongguang Wang, Hong Qin, and Aimin Hao. 2017. Video saliency detection via spatial-temporal fusion and low-rank coherency diffusion. IEEE Trans. Image Process. 26, 7 (2017), 3156–3170. Google Scholar
Digital Library
- Chenglizhao Chen, Yunxiao Li, Shuai Li, Hong Qin, and Aimin Hao. 2018. A novel bottom-up saliency detection method for video with dynamic background. IEEE Signal Process. Lett. 25, 2 (2018), 154–158.Google Scholar
Cross Ref
- Chenglizhao Chen, Guotao Wang, Chong Peng, Xiaowei Zhang, and Hong Qin. 2019. Improved robust video saliency detection based on long-term spatial-temporal information. IEEE Trans. Image Process. 29 (2019), 1090–1100.Google Scholar
Cross Ref
- Chenglizhao Chen, Jipeng Wei, Chong Peng, and Hong Qin. 2021. Depth quality-aware salient object detection. IEEE Trans. Image Process. 1, 1 (2021), 1–1.Google Scholar
- Chenglizhao Chen, Jipeng Wei, Chong Peng, Weizhong Zhang, and Hong Qin. 2020. Improved saliency detection in RGB-D images using two-phase depth estimation and selective deep fusion. IEEE Trans. Image Process. 29 (2020), 4296–4307.Google Scholar
Cross Ref
- Jianan Chen, Liquan Shen, Linru Zheng, and Xuhao Jiang. 2018. Naturalization module in neural networks for screen content image quality assessment. IEEE Signal Process. Lett. 25, 11 (2018), 1685–1689.Google Scholar
Cross Ref
- Zhengxue Cheng, Masaru Takeuchi, Kenji Kanai, and Jiro Katto. 2018. A fast no-reference screen content image quality prediction using convolutional neural networks. In Proceedings of the IEEE International Conference on Multimedia & Expo Workshops. 1–6.Google Scholar
Cross Ref
- Jia Deng, Wei Dong, Richard Socher, Lijia Li, Kai Li, and Feifei Li. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 248–255.Google Scholar
Cross Ref
- Yuming Fang, Jiebin Yan, Leida Li, Jinjian Wu, and Weisi Lin. 2018. No reference quality assessment for screen content images with both local and global feature representation. IEEE Trans. Image Process. 27, 4 (2018), 1600–1610. Google Scholar
Digital Library
- Yuming Fang, Jiebin Yan, Jiaying Liu, Shiqi Wang, Qiaohong Li, and Zongming Guo. 2017. Objective quality assessment of screen content images by uncertainty weighting. IEEE Trans. Image Process. 26, 4 (2017), 2016–2027. Google Scholar
Digital Library
- Pedro Garcia Freitas, Sana Alamgeer, Welington Y. L. Akamine, and Mylène C. Q. Farias. 2018. Blind image quality assessment based on multiscale salient local binary patterns. In Proceedings of the 9th ACM Multimedia Systems Conference. Association for Computing Machinery, 52–63. Google Scholar
Digital Library
- Ying Fu, Huanqiang Zeng, Lin Ma, Zhangkai Ni, Jianqing Zhu, and Kaikuang Ma. 2018. Screen content image quality assessment using multi-scale difference of gaussian. IEEE Trans. Circ. Syst. Video Technol. 28, 9 (2018), 2428–2432.Google Scholar
Cross Ref
- Ke Gu, Junfei Qiao, Xiongkuo Min, Guanghui Yue, Weisi Lin, and Daniel Thalmann. 2017. Evaluating quality of screen content images via structural variation analysis. IEEE Trans. Visual. Comput. Graph. 24, 10 (2017), 2689–2701. Google Scholar
Digital Library
- Ke Gu, Shiqi Wang, Huan Yang, Weisi Lin, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang. 2016. Saliency-guided quality assessment of screen content images. IEEE Trans. Multimedia 18, 6 (2016), 1098–1110.Google Scholar
Digital Library
- Ke Gu, Shie Wang, Gaohong Zhai, Siwei Ma, and Weisi Lin. 2015. Screen image quality assessment incorporating structural degradation measurement. In Proceedings of the IEEE International Symposium on Circuits and Systems. 125–128.Google Scholar
Cross Ref
- Ke Gu, Jun Zhou, Junfei Qiao, Guangtao Zhai, Weisi Lin, and Alan Bovik. 2017. No-reference quality assessment of screen content pictures. IEEE Trans. Image Process. 26, 8 (2017), 4005–4018.Google Scholar
Digital Library
- Han Hu, Yonggang Wen, Huanbo Luan, Tat-Seng Chua, and Xuelong Li. 2014. Toward multiscreen social TV with geolocation-aware social sense. IEEE Multimedia 21, 3 (2014), 10–19.Google Scholar
Cross Ref
- Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep features for text spotting. In Proceedings of the European Conference on Computer Vision. 512–528.Google Scholar
Cross Ref
- Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM International Conference on Multimedia. 675–678. Google Scholar
Digital Library
- Xuhao Jiang, Liquan Shen, Qing Ding, Linru Zheng, and Ping An. 2019. Screen content image quality assessment based on convolutional neural networks. J. Visual Commun. Image Represent. 67 (2019), 27–45.Google Scholar
- Xuhao Jiang, Liquan Shen, Guorui Feng, Liangwei Yu, and Ping An. 2019. Deep optimization model for screen content image quality assessment using neural networks, arXiv:1903.00705.Google Scholar
- Xuhao Jianga, Liquan Shen, Liangwei Yu, Mingxing Jiang, and Guorui Feng. 2019. No-reference screen content image quality assessment based on multi-region features. Neurocomputing 52 (2019), 410–432.Google Scholar
- Yunxiao Li, Shuai Li, Chenglizhao Chen, Aimin Hao, and Hong Qin. 2020. A plug-and-play scheme to adapt image saliency deep model for video data. IEEE Trans. Circ. Syst. Video Technol. 1, 1 (2020), 1–1.Google Scholar
Cross Ref
- Yunxiao Li, Shuai Li, Chenglizhao Chen, Hong Qin, and Aimin Hao. 2019. Accurate and robust video saliency detection via selfpaced diffusion. IEEE Trans. Multimedia. 22, 5 (2019), 1153–1167.Google Scholar
Cross Ref
- Anmin Liu, Weisi Lin, and Manish Narwaria. 2012. Image quality assessment based on gradient similarity. IEEE Trans. Image Process. 21, 4 (2012), 1500–1512. Google Scholar
Digital Library
- Guangxiao Ma, Chenglizhao Chen, Shuai Li, Chong Peng, Aimin Hao, and Hong Qin. 2019. Salient object detection via multiple instance joint re-learning. IEEE Trans. Multimedia 22, 2 (2019), 324--336.Google Scholar
Digital Library
- Guangxiao Ma, Shuai Li, Chenglizhao Chen, Aimin Hao, and Hong Qin. 2020. Stage-wise salient object detection in 360 omnidirectional image via object-level semantical saliency ranking. IEEE Trans. Visual. Comput. Graph. 22, 2 (2020), 324–336.Google Scholar
Digital Library
- Zhan Ma, Wei Wang, Meng Xu, and Haoping Yu. 2014. Advanced screen content coding using color table and index map. IEEE Trans. Image Process. 23, 10 (2014), 4399–4412.Google Scholar
Cross Ref
- Dan Miao, Jingjing Fu, Yan Lu, Shipeng Li, and ChangWen Chen. 2016. A high-fidelity and low-interaction-delay screen sharing system. ACM Trans. Multimedia Comput. Commun. Appl. 12, 3 (2016), 1–23. Google Scholar
Digital Library
- Xiongkuo Min, Kede Ma, Ke Gu, Guangtao Zhai, Zhou Wang, and Weisi Lin. 2017. Unified blind quality assessment of compressed natural, graphic and screen content images. IEEE Trans. Image Process. 26, 11 (2017), 5462–5474.Google Scholar
Digital Library
- Zhangkai Ni, Lin Ma, Huanqiang Zeng, Canhui Cai, and Kk. Ma. 2016. Gradient direction for screen content image quality assessment. IEEE Signal Process. Lett. 23, 10 (2016), 1394–1398.Google Scholar
Cross Ref
- Zhangkai Ni, Lin Ma, Huanqiang Zeng, Canhui Cai, and Kaikuang Ma. 2016. Screen content image quality assessment using edge model. In Proceedings of the IEEE International Conference on Image Processing. 81–85.Google Scholar
Cross Ref
- Zhangkai Ni, Lin Ma, Huanqiang Zeng, Jing Chen, Canhui Cai, and Kai-Kuang Ma. 2017. ESIM: Edge similarity for screen content image quality assessment. IEEE Trans. Image Process. 26, 10 (2017), 4818–4831.Google Scholar
Digital Library
- Zhangkai Ni, Huanqiang Zeng, Lin Ma, Junhui Hou, Jing Chen, and Kai-Kuang Ma. 2018. A gabor feature-based quality assessment model for the screen content images. IEEE Trans. Image Process. 27, 9 (2018), 4516–4528.Google Scholar
Digital Library
- Yagyenshchandra Pati, Ramin Rezaiifar, and Perinkulamsambamurthy Krishnaprasad. 1993. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of the 27th Asilomar Conference on Signals, Systems and Computers. 40–44.Google Scholar
Cross Ref
- Chong Peng, Chenglizhao Chen, Zhao Kang, and Qiang Cheng. 2020. Robust principal component analysis: A factorization based approach with linear complexity. Info. Sci. 513 (2020), 581–599.Google Scholar
- Chong Peng, Chenglizhao Chen, Zhao Kang, Jianbo Li, and Qiang Cheng. 2019. RES-PCA: A scalable approach to recovering low-rank matrices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).Google Scholar
Cross Ref
- Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. 2018. PieAPP: Perceptual image-error assessment through pairwise preference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1808–1817.Google Scholar
Cross Ref
- Jiansheng Qian, Lijuan Tang, Vinit Jakhetiya, Zhifang Xia, Ke Gu, and Hong Lu. 2017. Towards efficient blind quality evaluation of screen content images based on edge-preserving filter. Electron. Lett. 53, 9 (2017), 592–594.Google Scholar
Cross Ref
- Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations.Google Scholar
- Ruifeng Wang, Huan Yang, Zhenkuan Pan, Baoxiang Huang, and Guoji Hou. 2019. Screen content image quality assessment with edge features in gradient domain. IEEE Access 7 (2019), 5285–5295.Google Scholar
Cross Ref
- Shiqi Wang, Ke Gu, Kai Zeng, Zhou Wang, and Weisi Lin. 2015. Perceptual screen content image quality assessment and compression. In Proceedings of the IEEE International Conference on Image Processing. 1434–1438.Google Scholar
Digital Library
- Shiqi Wang, Ke Gu, Kai Zeng, Zhou Wang, and Weisi Lin. 2016. Objective quality assessment and perceptual compression of screen content images. IEEE Comput. Graph. Appl. 38, 1 (2016), 47–58.Google Scholar
Cross Ref
- Xuehao Wang, Shuai Li, Chenglizhao Chen, Yuming Fang, Aimin Hao, and Hong Qin. 2021. Data-level recombination and lightweight fusion scheme for RGB-D salient object detection. IEEE Trans. Image Process. 1, 1 (2021), 1–1.Google Scholar
- Zhou Wang, Alanconrad Bovik, Hamidrahim Sheikh, and Eerop Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600–612. Google Scholar
Digital Library
- Zhou Wang and Qiang Li. 2011. Information content weighting for perceptual image quality assessment. IEEE Trans. Image Process. 20, 5 (2011), 1185–1198. Google Scholar
Digital Library
- Zhou Wang and Xinli Shang. 2006. Spatial pooling strategies for perceptual image quality assessment. In Proceedings of the IEEE International Conference on Image Processing. 2945–2948.Google Scholar
Cross Ref
- Zhenyu Wu, Shuai Li, Chenglizhao Chen, Aimin Hao, and Hong Qin. 2021. A deeper look at salient object detection: Bi-stream network with a small training dataset. IEEE Trans. Multimedia 1, 1 (2021), 1–1.Google Scholar
- Lu Xing, Lei Cai, Huanqiang Zeng, Jing Chen, Jianqing Zhu, and Junhui Hou. 2017. A multi-scale contrast-based image quality assessment model for multi-exposure image fusion. Signal Process. 145 (2017), 233–240.Google Scholar
Cross Ref
- Wufeng Xue, Lei Zhang, Xuanqin Mou, and Alanc Bovik. 2014. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE Trans. Image Process. 23, 2 (2014), 684–695. Google Scholar
Digital Library
- Huan Yang, Yuming Fang, and Weisi Lin. 2015. Perceptual quality assessment of screen content images. IEEE Trans. Image Process. 24, 11 (2015), 4408–4421.Google Scholar
Digital Library
- Lin Zhang, Ying Shen, and Hongyu Li. 2014. VSI: A visual saliency-induced index for perceptual image quality assessment. IEEE Trans. Image Process. 23, 10 (2014), 4270–4281.Google Scholar
Cross Ref
- Xuande Zhang, Xiangchu Feng, Weiwei Wang, and Wufeng Xue. 2013. Edge strength similarity for image quality assessment. IEEE Signal Process. Lett. 20, 4 (2013), 319–322.Google Scholar
Cross Ref
- Yi Zhang, Damon Chandler, and Xuanqin Mou. 2018. Quality assessment of screen content images via convolutional-neural-network-based synthetic/natural segmentation. IEEE Trans. Image Process. 27, 10 (2018), 5113–5128.Google Scholar
Cross Ref
- Linru Zheng, Liquan Shen, Jianan Chen, Ping An, and Jun Luo. 2019. No-reference quality assessment for screen content images based on hybrid region features fusion. IEEE Trans. Multimedia 21, 8 (2019), 2057–2070.Google Scholar
Cross Ref
- Wujie Zhou, Lu Yu, Yang Zhou, Weiwei Qiu, Mingwei Wu, and Ting Luo. 2018. Local and global feature learning for blind quality evaluation of screen content and natural scene images. IEEE Trans. Image Process. 27, 5 (2018), 2086–2095.Google Scholar
Cross Ref
- Weijia Zhu, Wenpeng Ding, Jizheng Xu, Yunhui Shi, and Baocai Yin. 2015. Hash based block matching for screen content coding. IEEE Trans. Multimedia 17, 7 (2015), 935–944.Google Scholar
Digital Library
- Lingxuan Zuo, Hanli Wang, and Jie Fu. 2016. Screen content image quality assessment via convolutional neural network. In Proceedings of the IEEE International Conference on Image Processing. 2082–2086.Google Scholar
Cross Ref
Index Terms
Full-reference Screen Content Image Quality Assessment by Fusing Multilevel Structure Similarity
Recommendations
Full Reference Image Quality Assessment: Limitation
ICPR '14: Proceedings of the 2014 22nd International Conference on Pattern RecognitionIn this work, we propose to study the universality of the Full-Reference Image Quality metrics (FR-IQMs) and show the no-relevance to use this kind of metrics without considering the degradation type contained in the image. Different experimental tests ...
Full-reference image quality assessment by combining global and local distortion measures
Full-reference image quality assessment (FR-IQA) techniques compare a reference and a distorted/test image and predict the perceptual quality of the test image in terms of an objective score. Evaluation of FR-IQA techniques is carried out by comparing ...






Comments