skip to main content
research-article

Perceptual Hashing of Deep Convolutional Neural Networks for Model Copy Detection

Published:02 March 2023Publication History
Skip Abstract Section

Abstract

In recent years, many model intellectual property (IP) proof methods for IP protection have been proposed, such as model watermarking and model fingerprinting. However, with the increasing number of models transmitted and deployed on the Internet, quickly finding the suspect model among thousands of models on model-sharing platforms such as GitHub is in great demand, which concurrently triggers the new security problem of model copy detection for IP protection. As an important part of the model IP protection system, the model copy detection task has not received enough attention. Due to the high computational complexity, both model watermarking and model fingerprinting lack the capability to efficiently find suspected infringing models among tens of millions of models. In this article, inspired by the hash-based image retrieval methods, we introduce a novel model copy detection mechanism: perceptual hashing for convolutional neural networks (CNNs). The proposed perceptual hashing algorithm can convert the weights of CNNs to fixed-length binary hash codes so that the lightly modified version has the similar hash code as the original model. By comparing the similarity of a pair of hash codes between a query model and a test model in the model library, similar versions of a query model can be retrieved efficiently. To the best of our knowledge, this is the first perceptual hashing algorithm for deep neural network models. Specifically, we first select the important model weights based on the model compression theory, then calculate the normal test statistics (NTS) on the segments of important weights, and finally encode the NTS features into hash codes. The experiment performed on a model library containing 3,565 models indicates that our perceptual hashing scheme has a superior copy detection performance.

REFERENCES

  1. [1] Abdullahi Sani M., Wang Hongxia, and Li Tao. 2020. Fractal coding-based robust and alignment-free fingerprint image hashing. IEEE Trans. Inf. Forens. Secur. 15, 1 (2020), 25872601. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Adi Yossi, Baum Carsten, Cisse Moustapha, Pinkas Benny, and Keshet Joseph. 2018. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In Proceedings of the USENIX Security Symposium (USENIX Security’18). 16151631.Google ScholarGoogle Scholar
  3. [3] Agustsson Eirikur and Timofte Radu. 2017. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Bevilacqua Marco, Roumy Aline, Guillemot Christine, and Morel Marie line Alberi. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In British Machine Vision Conference. BMVA Press, 135.1–135.10. Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Blalock Davis, Ortiz Jose Javier Gonzalez, Frankle Jonathan, and Guttag John. 2020. What is the state of neural network pruning? Proceedings of Machine Learning and Systems 2 (2020), 129–146.Google ScholarGoogle Scholar
  6. [6] Blundell Charles, Cornebise Julien, Kavukcuoglu Koray, and Wierstra Daan. 2015. Weight uncertainty in neural networks. In International Conference on Machine Learning, Vol. 37. 16131622.Google ScholarGoogle Scholar
  7. [7] Cao Xiaoyu, Jia Jinyuan, and Gong Neil Zhenqiang. 2019. IPGuard: Protecting the intellectual property of deep neural networks via fingerprinting the classification boundary. arXiv:1910.12903 Retrieved from https://arxiv.org/abs/1910.129.03. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Carlini Nicholas and Wagner David. 2017. Towards evaluating the robustness of neural networks. In Proceedings of the IEEE Symposium on Security and Privacy. IEEE, 3957. Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Chatzikonstantinou Christos, Papadopoulos Georgios Th, Dimitropoulos Kosmas, and Daras Petros. 2020. Neural network compression using higher-order statistics and auxiliary reconstruction losses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 716717. Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Ciregan Dan, Meier Ueli, and Schmidhuber Jürgen. 2012. Multi-column deep neural networks for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 36423649.Google ScholarGoogle Scholar
  11. [11] Rouhani Bita Darvish, Chen Huili, and Koushanfar Farinaz. 2019. DeepSigns: An end-to-end watermarking framework for ownership protection of deep neural networks. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems. 485497.Google ScholarGoogle Scholar
  12. [12] Ding Kaimeng, Yang Zedong, Wang Yingying, and Liu Yueming. 2019. An improved perceptual hash algorithm based on U-Net for the authentication of high-resolution remote sensing image. Appl. Sci. 9, 15 (2019), 2972. Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Do Thanh-Toan, Hoang Tuan, Tan Dang-Khoa Le, Le Huu, Nguyen Tam V, and Cheung Ngai-Man. 2019. From selective deep convolutional features to compact binary representations for image retrieval. ACM Trans. Multimedia Comput. Commun. Appl. 15, 2 (2019), 122. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Fawcett Tom. 2006. An introduction to ROC analysis. Pattern Recogn. Lett. 27, 8 (June2006), 861874. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Guo Jinyang, Ouyang Wanli, and Xu Dong. 2020. Multi-dimensional pruning: A unified framework for model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15081517. Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Guo Jia and Potkonjak Miodrag. 2018. Watermarking deep neural networks for embedded systems. In Proceedings of the IEEE/ACM International Conference on Computer-Aided Design. IEEE, 18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Han Song, Pool Jeff, Tran John, and Dally William. 2015. Learning both weights and connections for efficient neural network. In Neural Information Processing Systems. 11351143.Google ScholarGoogle Scholar
  18. [18] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770778. Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Hong Richang, Yang Yang, Wang Meng, and Hua Xian-Sheng. 2015. Learning visual semantic relationships for efficient visual retrieval. IEEE Trans. Big Data. 1, 4 (2015), 152161. Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Huang Gao, Liu Zhuang, Maaten Laurens Van Der, and Weinberger Kilian Q. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 47004708. Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Khelifi Fouad and Jiang Jianmin. 2010. Perceptual image hashing based on virtual watermark detection. IEEE Trans. Image Process. 19, 4 (2010), 981994. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Kolouri Soheil, Saha Aniruddha, Pirsiavash Hamed, and Hoffmann Heiko. 2020. Universal litmus patterns: Revealing backdoor attacks in CNNs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 301310. Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Krizhevsky Alex. 2009. Learning Multiple Layers of Features from Tiny Images. Technical Report.Google ScholarGoogle Scholar
  24. [24] LeCun Yann, Bottou Léon, Bengio Yoshua, and Haffner Patrick. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 22782324. Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Ledig Christian, Theis Lucas, Huszar Ferenc, Caballero Jose, Cunningham Andrew, Acosta Alejandro, Aitken Andrew, Tejani Alykhan, Totz Johannes, Wang Zehan, and Shi Wenzhe. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Lee Suyoung, Song Wonho, Jana Suman, Cha Meeyoung, and Son Sooel. 2022. Evaluating the robustness of trigger set-based watermarks embedded in deep neural networks. IEEE Trans. Depend. Sec. Comput. (2022), 115. Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Li Yuanchun, Zhang Ziqi, Liu Bingyan, Yang Ziyue, and Liu Yunxin. 2021. ModelDiff: Testing-based DNN similarity comparison for model reuse detection. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021). Association for Computing Machinery, New York, NY, 139151. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Liang Xiaoping, Tang Zhenjun, Huang Ziqing, Zhang Xianquan, and Zhang Shichao. 2021. Efficient hashing method using 2D-2D PCA for image copy detection. IEEE Trans. Knowl. Data Eng. (2021), 11. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Liu Shiguang and Huang Ziqing. 2019. Efficient image hashing with geometric invariant vector distance for copy detection. ACM Trans. Multimedia Comput. Commun. Appl. 15, 4 (2019), 122. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Lukas Nils, Zhang Yuxuan, and Kerschbaum Florian. 2019. Deep neural network fingerprinting by conferrable adversarial examples. arXiv:1912.00888. Retrieved from https://arxiv.org/abs/1912.00888.Google ScholarGoogle Scholar
  31. [31] Lv Zhihan, Qiao Liang, Singh Amit Kumar, and Wang Qingjun. 2021. Fine-grained visual computing based on deep learning. ACM Trans. Multimidia Comput. Commun. Appl. 17, 1s (2021), 119. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Maimó Lorenzo Fernández, Gómez Ángel Luis Perales, Clemente Felix J. Garcia, Pérez Manuel Gil, and Pérez Gregorio Martínez. 2018. A self-adaptive deep learning-based system for anomaly detection in 5G networks. IEEE Access 6, 1 (2018), 77007712. Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Man Xin, Ouyang Deqiang, Li Xiangpeng, Song Jingkuan, and Shao Jie. 2022. Scenario-aware recurrent transformer for goal-directed video captioning. ACM Trans. Multim. Comput. Commun. Appl. 18, 4 (2022), 104:1–104:17.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Manning Christopher D., Surdeanu Mihai, Bauer John, Finkel Jenny Rose, Bethard Steven, and McClosky David. 2014. The stanford CoreNLP natural language processing toolkit. In Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 5560. Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Özyurt Fatih, Tuncer Türker, Avci Engin, Koç Mustafa, and Serhatlioğlu İhsan. 2019. A novel liver image classification method using perceptual hash-based convolutional neural network. Arab. J. Sci. Eng. 44, 4 (2019), 31733182.Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Razali Nornadiah Mohd, Wah Yap Bee, et al. 2011. Power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests. J. Stat. Model. Analyt. 2, 1 (2011), 2133.Google ScholarGoogle Scholar
  37. [37] Ribeiro Mauro, Grolinger Katarina, and Capretz Miriam AM. 2015. Mlaas: Machine learning as a service. In Proceedings of the IEEE International Conference on Machine Learning and Applications. IEEE, 896902. Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Rivest Ronald. 1992. RFC1321: The MD5 Message-Digest Algorithm.Google ScholarGoogle Scholar
  39. [39] Russakovsky Olga, Deng Jia, Su Hao, Krause Jonathan, Satheesh Sanjeev, Ma Sean, Huang Zhiheng, Karpathy Andrej, Khosla Aditya, Bernstein Michael, et al. 2015. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 3 (2015), 211252. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. [40] Sandler Mark, Howard Andrew, Zhu Menglong, Zhmoginov Andrey, and Chen Liang-Chieh. 2018. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 45104520. Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Shapiro Samuel Sanford and Wilk Martin B.. 1965. An analysis of variance test for normality (complete samples). Biometrika. 52, 3/4 (1965), 591611. Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Shi Qinghongya, Zhang Hong-Bo, Li Zhe, Du Ji-Xiang, Lei Qing, and Liu Jing-Hua. 2022. Shuffle-invariant network for action recognition in videos. ACM Trans. Multimedia Comput. Commun. Appl. 18, 3 (2022), 118. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. [43] Simonyan Karen and Zisserman Andrew. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Retrieved from https://arxiv.org/abs/1409.1556.Google ScholarGoogle Scholar
  44. [44] Song Jingkuan, Yang Yang, Yang Yi, Huang Zi, and Shen Heng Tao. 2013. Inter-media hashing for large-scale retrieval from heterogeneous data sources. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD ’13). Association for Computing Machinery, New York, NY, 785796. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. [45] Stallkamp Johannes, Schlipsing Marc, Salmen Jan, and Igel Christian. 2011. The german traffic sign recognition benchmark: A multi-class classification competition. In Proceedings of the International Joint Conference on Neural Networks. 14531460.Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Swaminathan Ashwin, Mao Yinian, and Wu Min. 2006. Robust and secure image hashing. IEEE Trans. Inf. Forens. Secur. 1, 2 (2006), 215230. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. [47] Tang Zhenjun, Chen Lv, Zhang Xianquan, and Zhang Shichao. 2019. Robust image hashing with tensor decomposition. IEEE Trans. Knowl. Data Eng. 31, 3 (2019), 549560. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. [48] Tian Xiaoyang, Shao Jie, Ouyang Deqiang, and Shen Heng Tao. 2022. UAV-satellite view synthesis for cross-view geo-localization. IEEE Trans. Circ. Syst. Video Technol. 32, 7 (2022), 48044815.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. [49] Tu Cheng-Hao, Yang Huei-Fang, Yang Shih-Min, Yeh Mei-Chen, and Chen Chu-Song. 2021. SemanticHash: Hash coding via semantics-guided label prototype learning. IEEE Trans. Artif. Intell. 2, 1 (2021), 42–57. Google ScholarGoogle ScholarCross RefCross Ref
  50. [50] Uchida Yusuke, Nagai Yuki, Sakazawa Shigeyuki, and Satoh Shin’ichi. 2017. Embedding watermarks into deep neural networks. In Proceedings of the ACM International Conference on Multimedia Retrieval. 269277. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. [51] Wang Chen, Liu Gaoyang, Huang Haojun, Feng Weijie, Peng Kai, and Wang Lizhe. 2019. MIASec: Enabling data indistinguishability against membership inference attacks in MLaaS. IEEE Trans. Sust. Comput. 5, 3 (2019), 365376. Google ScholarGoogle ScholarCross RefCross Ref
  52. [52] Wu Di, Zhou Xuebing, and Niu Xiamu. 2009. A novel image hash algorithm resistant to print-scan. Sign. Process. 89, 12 (2009), 24152424. Special Section: Visual Information Analysis for Security.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. [53] Zareapoor Masoumeh and Yang Jie. 2021. Equivariant adversarial network for image-to-image translation. ACM Trans. Multimedia Comput. Commun. Appl. 17, 2s (2021), 114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. [54] Zhang Jie, Chen Dongdong, Liao Jing, Fang Han, Ma Zehua, Zhang Weiming, Hua Gang, and Yu Nenghai. 2021. Exploring structure consistency for deep model watermarking. arXiv:2108.02360. Retrieved from https://arxiv.org/abs/2108.02360.Google ScholarGoogle Scholar
  55. [55] Zhang Jie, Chen Dongdong, Liao Jing, Fang Han, Zhang Weiming, Zhou Wenbo, Cui Hao, and Yu Nenghai. 2020. Model watermarking for image processing networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 1280512812. Google ScholarGoogle ScholarCross RefCross Ref
  56. [56] Zhang Jie, Chen Dongdong, Liao Jing, Zhang Weiming, Feng Huamin, Hua Gang, and Yu Nenghai. 2021. Deep model intellectual property protection via deep watermarking. IEEE Trans. Pattern Anal. Mach. Intell. 44, 8 (2021), 4005–4020. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. [57] Zhang Jie, Chen Dongdong, Liao Jing, Zhang Weiming, Hua Gang, and Yu Nenghai. 2020. Passport-aware normalization for deep model protection. Neural Inf. Process. Syst. 33 (2020).Google ScholarGoogle Scholar
  58. [58] Zhang Jialong, Gu Zhongshu, Jang Jiyong, Wu Hui, Stoecklin Marc Ph, Huang Heqing, and Molloy Ian. 2018. Protecting intellectual property of deep neural networks with watermarking. In Proceedings of the Asia Conference on Computer and Communications Security. 159172. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. [59] Zhang Ji, Song Jingkuan, Gao Lianli, Liu Ye, and Shen Heng Tao. 2022. Progressive meta-learning with curriculum. IEEE Trans. Circ. Syst. Video Technol. 32, 9 (2022), 59165930. Google ScholarGoogle ScholarCross RefCross Ref
  60. [60] Zhao Jingjing, Hu Qingyue, Liu Gaoyang, Ma Xiaoqiang, Chen Fei, and Hassan Mohammad Mehedi. 2020. AFA: Adversarial fingerprinting authentication for deep neural networks. Comput. Commun. 150, 1 (2020), 488497. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. [61] Zhou Zhili, Wang Yunlong, Wu QM Jonathan, Yang Ching-Nung, and Sun Xingming. 2016. Effective and efficient global context verification for image copy detection. IEEE Trans. Inf. Forens. Secur. 12, 1 (2016), 4863. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. [62] Zhu Anjie, Ouyang Deqiang, Liang Shuang, and Shao Jie. 2022. Step by step: A hierarchical framework for multi-hop knowledge graph reasoning with reinforcement learning. Knowl. Bas. Syst. 248, 1 (2022), 108843. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. [63] Zhu Han, Long Mingsheng, Wang Jianmin, and Cao Yue. 2016. Deep hashing network for efficient similarity retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30. Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Perceptual Hashing of Deep Convolutional Neural Networks for Model Copy Detection

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 3
        May 2023
        514 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3582886
        • Editor:
        • Abdulmotaleb El Saddik
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 2 March 2023
        • Online AM: 5 December 2022
        • Accepted: 6 November 2022
        • Revised: 1 October 2022
        • Received: 4 July 2022
        Published in tomm Volume 19, Issue 3

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!