skip to main content
research-article

Full-reference Screen Content Image Quality Assessment by Fusing Multilevel Structure Similarity

Authors Info & Claims
Published:22 July 2021Publication History
Skip Abstract Section

Abstract

Screen content images (SCIs) usually comprise various content types with sharp edges, in which artifacts or distortions can be effectively sensed by a vanilla structure similarity measurement in a full-reference manner. Nonetheless, almost all of the current state-of-the-art (SOTA) structure similarity metrics are “locally” formulated in a single-level manner, while the true human visual system (HVS) follows the multilevel manner; such mismatch could eventually prevent these metrics from achieving reliable quality assessment. To ameliorate this issue, this article advocates a novel solution to measure structure similarity “globally” from the perspective of sparse representation. To perform multilevel quality assessment in accordance with the real HVS, the abovementioned global metric will be integrated with the conventional local ones by resorting to the newly devised selective deep fusion network. To validate its efficacy and effectiveness, we have compared our method with 12 SOTA methods over two widely used large-scale public SCI datasets, and the quantitative results indicate that our method yields significantly higher consistency with subjective quality scores than the current leading works. Both the source code and data are also publicly available to gain widespread acceptance and facilitate new advancement and validation.

References

  1. Michal Aharon, Michael Elad, and Alfred Bruckstein. 2006. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54, 11 (2006), 4311–4322. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Tsuishan Chang, Chunchi Chen, Ruling Liao, Chewei Kuo, and Wenhsiao Peng. 2014. Intra line copy for HEVC screen content coding. In Proceedings of the Asia-Pacific Signal & Information Processing Association, Summit & Conference. 1–8.Google ScholarGoogle ScholarCross RefCross Ref
  3. Chenglizhao Chen, Shuai Li, and Hong Qin. 2016. Robust salient motion detection in non-stationary videos via novel integrated strategies of spatio-temporal coherency clues and low-rank analysis. Pattern Recogn. 52 (2016), 410–432. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Chenglizhao Chen, Shuai Li, Hong Qin, and Aimin Hao. 2015. Real-time and robust object tracking in video via low-rank coherency analysis in feature space. Pattern Recogn. 48, 9 (2015), 2885–2905. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Chenglizhao Chen, Shuai Li, Hong Qin, and Aimin Hao. 2015. Structure-sensitive saliency detection via multilevel rank analysis in intrinsic feature space. IEEE Trans. Image Process. 24, 8 (2015), 2303–2316.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Chenglizhao Chen, Shuai Li, Hong Qin, and Aimin Hao. 2016. Robust salient motion detection in non-stationary videos via novel integrated strategies of spatio-temporal coherency clues and low-rank analysis. Pattern Recogn. 52 (2016), 410–432. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Chenglizhao Chen, Shuai Li, Hong Qin, Zhenkuan Pan, and Guowei Yang. 2018. Bi-level feature learning for video saliency detection. IEEE Trans. Multimedia. 20, 12 (2018), 3324–3336.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Chenglizhao Chen, Shuai Li, Yongguang Wang, Hong Qin, and Aimin Hao. 2017. Video saliency detection via spatial-temporal fusion and low-rank coherency diffusion. IEEE Trans. Image Process. 26, 7 (2017), 3156–3170. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Chenglizhao Chen, Yunxiao Li, Shuai Li, Hong Qin, and Aimin Hao. 2018. A novel bottom-up saliency detection method for video with dynamic background. IEEE Signal Process. Lett. 25, 2 (2018), 154–158.Google ScholarGoogle ScholarCross RefCross Ref
  10. Chenglizhao Chen, Guotao Wang, Chong Peng, Xiaowei Zhang, and Hong Qin. 2019. Improved robust video saliency detection based on long-term spatial-temporal information. IEEE Trans. Image Process. 29 (2019), 1090–1100.Google ScholarGoogle ScholarCross RefCross Ref
  11. Chenglizhao Chen, Jipeng Wei, Chong Peng, and Hong Qin. 2021. Depth quality-aware salient object detection. IEEE Trans. Image Process. 1, 1 (2021), 1–1.Google ScholarGoogle Scholar
  12. Chenglizhao Chen, Jipeng Wei, Chong Peng, Weizhong Zhang, and Hong Qin. 2020. Improved saliency detection in RGB-D images using two-phase depth estimation and selective deep fusion. IEEE Trans. Image Process. 29 (2020), 4296–4307.Google ScholarGoogle ScholarCross RefCross Ref
  13. Jianan Chen, Liquan Shen, Linru Zheng, and Xuhao Jiang. 2018. Naturalization module in neural networks for screen content image quality assessment. IEEE Signal Process. Lett. 25, 11 (2018), 1685–1689.Google ScholarGoogle ScholarCross RefCross Ref
  14. Zhengxue Cheng, Masaru Takeuchi, Kenji Kanai, and Jiro Katto. 2018. A fast no-reference screen content image quality prediction using convolutional neural networks. In Proceedings of the IEEE International Conference on Multimedia & Expo Workshops. 1–6.Google ScholarGoogle ScholarCross RefCross Ref
  15. Jia Deng, Wei Dong, Richard Socher, Lijia Li, Kai Li, and Feifei Li. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 248–255.Google ScholarGoogle ScholarCross RefCross Ref
  16. Yuming Fang, Jiebin Yan, Leida Li, Jinjian Wu, and Weisi Lin. 2018. No reference quality assessment for screen content images with both local and global feature representation. IEEE Trans. Image Process. 27, 4 (2018), 1600–1610. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Yuming Fang, Jiebin Yan, Jiaying Liu, Shiqi Wang, Qiaohong Li, and Zongming Guo. 2017. Objective quality assessment of screen content images by uncertainty weighting. IEEE Trans. Image Process. 26, 4 (2017), 2016–2027. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Pedro Garcia Freitas, Sana Alamgeer, Welington Y. L. Akamine, and Mylène C. Q. Farias. 2018. Blind image quality assessment based on multiscale salient local binary patterns. In Proceedings of the 9th ACM Multimedia Systems Conference. Association for Computing Machinery, 52–63. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Ying Fu, Huanqiang Zeng, Lin Ma, Zhangkai Ni, Jianqing Zhu, and Kaikuang Ma. 2018. Screen content image quality assessment using multi-scale difference of gaussian. IEEE Trans. Circ. Syst. Video Technol. 28, 9 (2018), 2428–2432.Google ScholarGoogle ScholarCross RefCross Ref
  20. Ke Gu, Junfei Qiao, Xiongkuo Min, Guanghui Yue, Weisi Lin, and Daniel Thalmann. 2017. Evaluating quality of screen content images via structural variation analysis. IEEE Trans. Visual. Comput. Graph. 24, 10 (2017), 2689–2701. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Ke Gu, Shiqi Wang, Huan Yang, Weisi Lin, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang. 2016. Saliency-guided quality assessment of screen content images. IEEE Trans. Multimedia 18, 6 (2016), 1098–1110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Ke Gu, Shie Wang, Gaohong Zhai, Siwei Ma, and Weisi Lin. 2015. Screen image quality assessment incorporating structural degradation measurement. In Proceedings of the IEEE International Symposium on Circuits and Systems. 125–128.Google ScholarGoogle ScholarCross RefCross Ref
  23. Ke Gu, Jun Zhou, Junfei Qiao, Guangtao Zhai, Weisi Lin, and Alan Bovik. 2017. No-reference quality assessment of screen content pictures. IEEE Trans. Image Process. 26, 8 (2017), 4005–4018.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Han Hu, Yonggang Wen, Huanbo Luan, Tat-Seng Chua, and Xuelong Li. 2014. Toward multiscreen social TV with geolocation-aware social sense. IEEE Multimedia 21, 3 (2014), 10–19.Google ScholarGoogle ScholarCross RefCross Ref
  25. Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep features for text spotting. In Proceedings of the European Conference on Computer Vision. 512–528.Google ScholarGoogle ScholarCross RefCross Ref
  26. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM International Conference on Multimedia. 675–678. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Xuhao Jiang, Liquan Shen, Qing Ding, Linru Zheng, and Ping An. 2019. Screen content image quality assessment based on convolutional neural networks. J. Visual Commun. Image Represent. 67 (2019), 27–45.Google ScholarGoogle Scholar
  28. Xuhao Jiang, Liquan Shen, Guorui Feng, Liangwei Yu, and Ping An. 2019. Deep optimization model for screen content image quality assessment using neural networks, arXiv:1903.00705.Google ScholarGoogle Scholar
  29. Xuhao Jianga, Liquan Shen, Liangwei Yu, Mingxing Jiang, and Guorui Feng. 2019. No-reference screen content image quality assessment based on multi-region features. Neurocomputing 52 (2019), 410–432.Google ScholarGoogle Scholar
  30. Yunxiao Li, Shuai Li, Chenglizhao Chen, Aimin Hao, and Hong Qin. 2020. A plug-and-play scheme to adapt image saliency deep model for video data. IEEE Trans. Circ. Syst. Video Technol. 1, 1 (2020), 1–1.Google ScholarGoogle ScholarCross RefCross Ref
  31. Yunxiao Li, Shuai Li, Chenglizhao Chen, Hong Qin, and Aimin Hao. 2019. Accurate and robust video saliency detection via selfpaced diffusion. IEEE Trans. Multimedia. 22, 5 (2019), 1153–1167.Google ScholarGoogle ScholarCross RefCross Ref
  32. Anmin Liu, Weisi Lin, and Manish Narwaria. 2012. Image quality assessment based on gradient similarity. IEEE Trans. Image Process. 21, 4 (2012), 1500–1512. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Guangxiao Ma, Chenglizhao Chen, Shuai Li, Chong Peng, Aimin Hao, and Hong Qin. 2019. Salient object detection via multiple instance joint re-learning. IEEE Trans. Multimedia 22, 2 (2019), 324--336.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Guangxiao Ma, Shuai Li, Chenglizhao Chen, Aimin Hao, and Hong Qin. 2020. Stage-wise salient object detection in 360 omnidirectional image via object-level semantical saliency ranking. IEEE Trans. Visual. Comput. Graph. 22, 2 (2020), 324–336.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Zhan Ma, Wei Wang, Meng Xu, and Haoping Yu. 2014. Advanced screen content coding using color table and index map. IEEE Trans. Image Process. 23, 10 (2014), 4399–4412.Google ScholarGoogle ScholarCross RefCross Ref
  36. Dan Miao, Jingjing Fu, Yan Lu, Shipeng Li, and ChangWen Chen. 2016. A high-fidelity and low-interaction-delay screen sharing system. ACM Trans. Multimedia Comput. Commun. Appl. 12, 3 (2016), 1–23. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Xiongkuo Min, Kede Ma, Ke Gu, Guangtao Zhai, Zhou Wang, and Weisi Lin. 2017. Unified blind quality assessment of compressed natural, graphic and screen content images. IEEE Trans. Image Process. 26, 11 (2017), 5462–5474.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Zhangkai Ni, Lin Ma, Huanqiang Zeng, Canhui Cai, and Kk. Ma. 2016. Gradient direction for screen content image quality assessment. IEEE Signal Process. Lett. 23, 10 (2016), 1394–1398.Google ScholarGoogle ScholarCross RefCross Ref
  39. Zhangkai Ni, Lin Ma, Huanqiang Zeng, Canhui Cai, and Kaikuang Ma. 2016. Screen content image quality assessment using edge model. In Proceedings of the IEEE International Conference on Image Processing. 81–85.Google ScholarGoogle ScholarCross RefCross Ref
  40. Zhangkai Ni, Lin Ma, Huanqiang Zeng, Jing Chen, Canhui Cai, and Kai-Kuang Ma. 2017. ESIM: Edge similarity for screen content image quality assessment. IEEE Trans. Image Process. 26, 10 (2017), 4818–4831.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Zhangkai Ni, Huanqiang Zeng, Lin Ma, Junhui Hou, Jing Chen, and Kai-Kuang Ma. 2018. A gabor feature-based quality assessment model for the screen content images. IEEE Trans. Image Process. 27, 9 (2018), 4516–4528.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Yagyenshchandra Pati, Ramin Rezaiifar, and Perinkulamsambamurthy Krishnaprasad. 1993. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of the 27th Asilomar Conference on Signals, Systems and Computers. 40–44.Google ScholarGoogle ScholarCross RefCross Ref
  43. Chong Peng, Chenglizhao Chen, Zhao Kang, and Qiang Cheng. 2020. Robust principal component analysis: A factorization based approach with linear complexity. Info. Sci. 513 (2020), 581–599.Google ScholarGoogle Scholar
  44. Chong Peng, Chenglizhao Chen, Zhao Kang, Jianbo Li, and Qiang Cheng. 2019. RES-PCA: A scalable approach to recovering low-rank matrices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).Google ScholarGoogle ScholarCross RefCross Ref
  45. Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. 2018. PieAPP: Perceptual image-error assessment through pairwise preference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1808–1817.Google ScholarGoogle ScholarCross RefCross Ref
  46. Jiansheng Qian, Lijuan Tang, Vinit Jakhetiya, Zhifang Xia, Ke Gu, and Hong Lu. 2017. Towards efficient blind quality evaluation of screen content images based on edge-preserving filter. Electron. Lett. 53, 9 (2017), 592–594.Google ScholarGoogle ScholarCross RefCross Ref
  47. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations.Google ScholarGoogle Scholar
  48. Ruifeng Wang, Huan Yang, Zhenkuan Pan, Baoxiang Huang, and Guoji Hou. 2019. Screen content image quality assessment with edge features in gradient domain. IEEE Access 7 (2019), 5285–5295.Google ScholarGoogle ScholarCross RefCross Ref
  49. Shiqi Wang, Ke Gu, Kai Zeng, Zhou Wang, and Weisi Lin. 2015. Perceptual screen content image quality assessment and compression. In Proceedings of the IEEE International Conference on Image Processing. 1434–1438.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Shiqi Wang, Ke Gu, Kai Zeng, Zhou Wang, and Weisi Lin. 2016. Objective quality assessment and perceptual compression of screen content images. IEEE Comput. Graph. Appl. 38, 1 (2016), 47–58.Google ScholarGoogle ScholarCross RefCross Ref
  51. Xuehao Wang, Shuai Li, Chenglizhao Chen, Yuming Fang, Aimin Hao, and Hong Qin. 2021. Data-level recombination and lightweight fusion scheme for RGB-D salient object detection. IEEE Trans. Image Process. 1, 1 (2021), 1–1.Google ScholarGoogle Scholar
  52. Zhou Wang, Alanconrad Bovik, Hamidrahim Sheikh, and Eerop Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600–612. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Zhou Wang and Qiang Li. 2011. Information content weighting for perceptual image quality assessment. IEEE Trans. Image Process. 20, 5 (2011), 1185–1198. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Zhou Wang and Xinli Shang. 2006. Spatial pooling strategies for perceptual image quality assessment. In Proceedings of the IEEE International Conference on Image Processing. 2945–2948.Google ScholarGoogle ScholarCross RefCross Ref
  55. Zhenyu Wu, Shuai Li, Chenglizhao Chen, Aimin Hao, and Hong Qin. 2021. A deeper look at salient object detection: Bi-stream network with a small training dataset. IEEE Trans. Multimedia 1, 1 (2021), 1–1.Google ScholarGoogle Scholar
  56. Lu Xing, Lei Cai, Huanqiang Zeng, Jing Chen, Jianqing Zhu, and Junhui Hou. 2017. A multi-scale contrast-based image quality assessment model for multi-exposure image fusion. Signal Process. 145 (2017), 233–240.Google ScholarGoogle ScholarCross RefCross Ref
  57. Wufeng Xue, Lei Zhang, Xuanqin Mou, and Alanc Bovik. 2014. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE Trans. Image Process. 23, 2 (2014), 684–695. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Huan Yang, Yuming Fang, and Weisi Lin. 2015. Perceptual quality assessment of screen content images. IEEE Trans. Image Process. 24, 11 (2015), 4408–4421.Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Lin Zhang, Ying Shen, and Hongyu Li. 2014. VSI: A visual saliency-induced index for perceptual image quality assessment. IEEE Trans. Image Process. 23, 10 (2014), 4270–4281.Google ScholarGoogle ScholarCross RefCross Ref
  60. Xuande Zhang, Xiangchu Feng, Weiwei Wang, and Wufeng Xue. 2013. Edge strength similarity for image quality assessment. IEEE Signal Process. Lett. 20, 4 (2013), 319–322.Google ScholarGoogle ScholarCross RefCross Ref
  61. Yi Zhang, Damon Chandler, and Xuanqin Mou. 2018. Quality assessment of screen content images via convolutional-neural-network-based synthetic/natural segmentation. IEEE Trans. Image Process. 27, 10 (2018), 5113–5128.Google ScholarGoogle ScholarCross RefCross Ref
  62. Linru Zheng, Liquan Shen, Jianan Chen, Ping An, and Jun Luo. 2019. No-reference quality assessment for screen content images based on hybrid region features fusion. IEEE Trans. Multimedia 21, 8 (2019), 2057–2070.Google ScholarGoogle ScholarCross RefCross Ref
  63. Wujie Zhou, Lu Yu, Yang Zhou, Weiwei Qiu, Mingwei Wu, and Ting Luo. 2018. Local and global feature learning for blind quality evaluation of screen content and natural scene images. IEEE Trans. Image Process. 27, 5 (2018), 2086–2095.Google ScholarGoogle ScholarCross RefCross Ref
  64. Weijia Zhu, Wenpeng Ding, Jizheng Xu, Yunhui Shi, and Baocai Yin. 2015. Hash based block matching for screen content coding. IEEE Trans. Multimedia 17, 7 (2015), 935–944.Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Lingxuan Zuo, Hanli Wang, and Jie Fu. 2016. Screen content image quality assessment via convolutional neural network. In Proceedings of the IEEE International Conference on Image Processing. 2082–2086.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Full-reference Screen Content Image Quality Assessment by Fusing Multilevel Structure Similarity

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 17, Issue 3
      August 2021
      443 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3476118
      Issue’s Table of Contents

      Copyright © 2021 Association for Computing Machinery.

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 22 July 2021
      • Revised: 1 January 2021
      • Accepted: 1 January 2021
      • Received: 1 April 2020
      Published in tomm Volume 17, Issue 3

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!