skip to main content
research-article

Privacy-preserving Time-series Medical Images Analysis Using a Hybrid Deep Learning Framework

Published:16 June 2021Publication History
Skip Abstract Section

Abstract

Time-series medical images are an important type of medical data that contain rich temporal and spatial information. As a state-of-the-art, computer-aided diagnosis (CAD) algorithms are usually used on these image sequences to improve analysis accuracy. However, such CAD algorithms are often required to upload medical images to honest-but-curious servers, which introduces severe privacy concerns. To preserve privacy, the existing CAD algorithms support analysis on each encrypted image but not on the whole encrypted image sequences, which leads to the loss of important temporal information among frames. To meet this challenge, a convolutional-LSTM network, named HE-CLSTM, is proposed for analyzing time-series medical images encrypted by a fully homomorphic encryption mechanism. Specifically, several convolutional blocks are constructed to extract discriminative spatial features, and LSTM-based sequence analysis layers (HE-LSTM) are leveraged to encode temporal information from the encrypted image sequences. Moreover, a weighted unit and a sequence voting layer are designed to incorporate both spatial and temporal features with different weights to improve performance while reducing the missed diagnosis rate. The experimental results on two challenging benchmarks (a Cervigram dataset and the BreaKHis public dataset) provide strong evidence that our framework can encode visual representations and sequential dynamics from encrypted medical image sequences; our method achieved AUCs above 0.94 both on the Cervigram and BreaKHis datasets, constituting a significant margin of statistical improvement compared with several competing methods.

References

  1. Ahmed M. Alaa, Kyeong H. Moon, William Hsu, and Mihaela Van Der Schaar. 2016. ConfidentCare: A clinical decision support system for personalized breast cancer screening. IEEE Trans. Multimedia 18, 10 (2016), 1942–1955. DOI:https://doi.org/10.1109/TMM.2016.2589160Google ScholarGoogle ScholarCross RefCross Ref
  2. Shekoofeh Azizi, Sharareh Bayat, Pingkun Yan, Amir Tahmasebi, Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto, and Bradford Wood. 2018. Deep recurrent neural networks for prostate cancer detection: Analysis of temporal. IEEE Trans. Med. Imag. 37, 12 (2018), 2695–2703. DOI:https://doi.org/10.1109/TMI.2018.2849959Google ScholarGoogle ScholarCross RefCross Ref
  3. Ankan Kumar Bhunia, Aishik Konwer, Ayan Kumar Bhunia, Abir Bhowmick, Partha P. Roy, and Umapada Pal. 2019. Script identification in natural scene image and video frames using an attention based Convolutional-LSTM network. Pattern Recogn. 85 (2019), 172–184. DOI:https://doi.org/10.1016/j.patcog.2018.07.034Google ScholarGoogle ScholarCross RefCross Ref
  4. Sylvia F. Boj, Chang Il Hwang, Lindsey A. Baker, Iok In Christine Chio, Dannielle D. Engle, Vincenzo Corbo, Myrthe Jager, Mariano Ponz-Sarvise, Hervé Tiriac, Mona S. Spector, Ana Gracanin, Tobiloba Oni, Kenneth H. Yu, Ruben Van Boxtel, Meritxell Huch, Keith D. Rivera, John P. Wilson, Michael E. Feigin, Daniel Öhlund, Abram Handly-Santana, Christine M. Ardito-Abraham, Michael Ludwig, Ela Elyada, Brinda Alagesan, Giulia Biffi, Georgi N. Yordanov, Bethany Delcuze, Brianna Creighton, Kevin Wright, Youngkyu Park, Folkert H. M. Morsink, I. Quintus Molenaar, Inne H. Borel Rinkes, Edwin Cuppen, Yuan Hao, Ying Jin, Isaac J. Nijman, Christine Iacobuzio-Donahue, Steven D. Leach, Darryl J. Pappin, Molly Hammell, David S. Klimstra, Olca Basturk, Ralph H. Hruban, George Johan Offerhaus, Robert G. J. Vries, Hans Clevers and David A. Tuveson. 2015. Organoid models of human and mouse ductal pancreatic cancer. Cell 160, 1–2 (2015), 324–338. DOI:https://doi.org/10.1016/j.cell.2014.12.021Google ScholarGoogle ScholarCross RefCross Ref
  5. Joppe W. Bos, Kristin Lauter, Jake Loftus, and Michael Naehrig. 2013. Improved security for a ring-based fully homomorphic encryption scheme. In Proceedings of the IMA International Conference on Cryptography and Coding. 45–64. DOI:https://doi.org/10.1007/978-3-642-45239-0_4 Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Ümit Budak, Zafer Cömert, Zryan Najat Rashid, Abdulkadir Sengür, and Musa Çibuk. 2019. Computer-aided diagnosis system combining FCN and Bi-LSTM model for efficient breast cancer detection from histopathological images. Appl. Soft Comput. J. 85 (2019), 105765. DOI:https://doi.org/10.1016/j.asoc.2019.105765Google ScholarGoogle ScholarCross RefCross Ref
  7. Herve Chabanne, Amaury de Wargny, Jonathan Milgram, Constance Morel, and Emmanuel Prouff. 2017. Privacy-preserving classification on deep neural network. In Proceedings of the International Association for Cryptologic Research (IACR'17), 35.Google ScholarGoogle Scholar
  8. Christian Von Der Weth, Ashraf Abdul, Abhinav R. Kashyap, and Mohan S. Kankanhalli. 2019. CloseUp-community-driven live online search engine. ACM Trans. Internet Technol. 19, 3 (2019), 1–21. DOI:https://doi.org/10.1145/3301442 Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Shuai Ding, Shikang Hu, Jinxin Pan, Xiaojian Li, Gang Li, and Xiao Liu. 2019. A homogeneous ensemble method for predicting gastric cancer based on gastroscopy reports. Expert Syst. 37, 3 (2019), e12499. DOI:https://doi.org/10.1111/exsy.12499Google ScholarGoogle Scholar
  10. Shuai Ding, Ling Li, Zhenmin Li, Hao Wang, and Yanchun Zhang. 2019. Smart electronic gastroscope system using a cloudâedge collaborative framework. Fut. Gener. Comput. Syst. 100 (2019), 395–407. DOI:https://doi.org/10.1016/j.future.2019.04.031Google ScholarGoogle ScholarCross RefCross Ref
  11. Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Trevor Darrell. 2017. Long-term recurrent convolutional networks for visual recognition and description. IEEE Trans. Pattern Anal. Mach. Intell. 39, 4 (2017), 677–691. DOI:https://doi.org/10.1109/TPAMI.2016.2599174 Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Nathan Dowlin, Ran Gilad-Bachrach, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. 2016. CryptoNets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of the International Conference on Machine Learning. 201–210. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Li Duan, Dongxi Liu, Yang Zhang, Shiping Chen, Ren Ping Liu, Bo Cheng, and Junliang Chen. 2016. Secure data-centric access control for smart grid services based on publish/subscribe systems. ACM Trans. Internet Technol. 16, 4 (2016), 1–17. DOI:https://doi.org/10.1145/3007190 Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, and Sebastian Thrun. 2017. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 7639 (2017), 115–118. DOI:https://doi.org/10.1038/nature21056Google ScholarGoogle Scholar
  15. Junfeng Fan and Frederik Vercauteren. 2012. Somewhat practical fully homomorphic encryption. In Proceedings of the International Association for Cryptologic Research (IACR'12). 144.Google ScholarGoogle Scholar
  16. Jeremy Farrar. 2014. Medical data: Widen use in research. Nature 506, 7486 (2014), 34. DOI:https://doi.org/10.1038/506034bGoogle ScholarGoogle Scholar
  17. Craig Gentry. 2009. A Fully Homomorphic Encryption Scheme. Stanford University (2009). DOI:https://doi.org/10.1145/1536414.1536440 Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Francisco Javier González-Serrano, Ángel Navia-Vázquez, and Adrián Amor-Martín. 2017. Training support vector machines with privacy-protected data. Pattern Recogn. 72 (2017), 93–107. DOI:https://doi.org/10.1016/j.patcog.2017.06.016 Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Varun Gulshan, Lily Peng, Marc Coram, Martin C. Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica L. Mega, and Dale R. Webster. 2016. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J. Am. Med. Assoc. 316, 22 (2016), 2402–2410. DOI:https://doi.org/10.1001/jama.2016.17216Google ScholarGoogle ScholarCross RefCross Ref
  20. Zhongyun Hua, Shuang Yi, and Yicong Zhou. 2018. Medical image encryption using high-speed scrambling and pixel adaptive diffusion. Signal Process. 144 (2018), 134–144. DOI:https://doi.org/10.1016/j.sigpro.2017.10.004 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Multi-view Inference, Yi Zhou, Student Member, Li Liu, Ling Shao, and Senior Member. 2018. Vehicle re-identification by deep hidden multi-view inference. IEEE Trans. Image Process. 27, 7 (2018), 3275–3287.Google ScholarGoogle ScholarCross RefCross Ref
  22. Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2013. 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1 (2013), 221–231. DOI:https://doi.org/10.1109/TPAMI.2012.59 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’09). 248–255. DOI:https://doi.org/10.1109/CVPRW.2009.5206848Google ScholarGoogle ScholarCross RefCross Ref
  24. Yueming Jin, Qi Dou, Hao Chen, Lequan Yu, Jing Qin, Chi Wing Fu, and Pheng Ann Heng. 2017. SV-RCNet: Workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans. Med. Imag. 37, 5 (2017), 1114–1126. DOI:https://doi.org/10.1109/TMI.2017.2787657Google ScholarGoogle ScholarCross RefCross Ref
  25. Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C. S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei, Magdalena Ting, Jie Zhu, Christina Li, Sierra Hewett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang, Lianghong Zheng, Rui Hou, William Shi, Xin Fu, Yaou Duan, Viet A. N. Huu, Cindy Wen, Edward D. Zhang, Charlotte L. Zhang, Oulan Li, Xiaobo Wang, Michael A. Singer, Xiaodong Sun, Jie Xu, Ali Tafreshi, M. Anthony Lewis, Huimin Xia and Kang Zhang. 2018. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 5 (2018), 1122–1131. DOI:https://doi.org/10.1016/j.cell.2018.02.010Google ScholarGoogle ScholarCross RefCross Ref
  26. Neal Koblitz. 1987. Elliptic curve cryptosystems. Math. Comp. 48, 177 (1987), 203–203. DOI:https://doi.org/10.1090/S0025-5718-1987-0866109-5Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Sung-hyun Lee, Joon Hyun Song, and Il Kon Kim. 2016. CDA generation and integration for health information exchange based on cloud computing system. IEEE Trans. Serv. Comput. 9, 2 (2016), 241–249. DOI:https://doi.org/10.1109/TSC.2014.2363654 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. X.-B. Li and Jialun Qin. 2017. Anonymizing and sharing medical text records. Inf. Syst. Res. 28, 2 (2017), 332–352. DOI:https://doi.org/10.1287/isre.2016.0676 Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Yang Liu, Shanyu Tang, Ran Liu, Liping Zhang, and Zhao Ma. 2018. Secure and robust digital image watermarking scheme using logistic and RSA encryption. Expert Syst. Appl. 97 (2018), 95–105. DOI:https://doi.org/10.1016/j.eswa.2017.12.003Google ScholarGoogle ScholarCross RefCross Ref
  30. Chao Ma, Yulan Guo, Jungang Yang, and Wei An. 2019. Learning multi-view representation with LSTM for 3-D shape recognition and retrieval. IEEE Trans. Multimedia 21, 5 (2019), 1169–1182. DOI:https://doi.org/10.1109/TMM.2018.2875512Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Ahmet Can Mert, Erdinc Ozturk, and Erkay Savas. 2019. Design and implementation of encryption/decryption architectures for bfv homomorphic encryption scheme. IEEE Trans. VLSI Syst. 28, 2 (2019), 353--362. DOI:https://doi.org/10.1109/tvlsi.2019.2943127Google ScholarGoogle ScholarCross RefCross Ref
  32. Biotecnologia Microbiana, De Hidalgo, Marios Anthimopoilos, Srergios Christodoulidis, Lukas Ebner, Andreas Christe, and Stavroula Mougiakakou. 2016. Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans. Med. Imag. 35, 5 (2016), 1207–1216. DOI:https://doi.org/10.1109/TMI.2016.2535865Google ScholarGoogle ScholarCross RefCross Ref
  33. Declan O’Loughlin, Barbara L. Oliveira, Adam Santorelli, Emily Porter, Martin Glavin, Edward Jones, Milica Popovic, and Martin O’Halloran. 2019. Sensitivity and specificity estimation using patient-specific microwave imaging in diverse experimental breast phantoms. IEEE Trans. Med. Imag. 38, 1 (2019), 303–311. DOI:https://doi.org/10.1109/TMI.2018.2864150Google ScholarGoogle ScholarCross RefCross Ref
  34. Jane S. Paulsen, Jeff D. Long, Christopher A. Ross, Deborah L. Harrington, Cheryl J. Erwin, Janet K. Williams, Holly James Westervelt, and Hans J. Johnson. 2014. Prediction of manifest Huntington’s disease with clinical and imaging measures: A prospective observational study. Lancet Neurol. 13, 12 (2014), 1193–1201. DOI:https://doi.org/10.1016/S1474-4422(14)70238-8Google ScholarGoogle ScholarCross RefCross Ref
  35. W. Nicholson Price and Glenn Cohen. 2019. Privacy in the age of medical big data. Nat. Med. 25, 1 (2019), 37–43. DOI:https://doi.org/10.1038/s41591-018-0272-7Google ScholarGoogle ScholarCross RefCross Ref
  36. Ahmad Qaisar Ahmad Al Badawi, Yuriy Polyakov, Khin Mi Mi Aung, Bharadwaj Veeravalli, and Kurt Rohloff. 2019. Implementation and performance evaluation of RNS variants of the BFV homomorphic encryption scheme. IEEE Trans. Emerg. Topics Comput. (2019). DOI:https://doi.org/10.1109/tetc.2019.2902799Google ScholarGoogle Scholar
  37. Jean Louis Raisaro, Gwangbae Choi, Sylvain Pradervand, Raphael Colsenet, Nathalie Jacquemont, Nicolas Rosat, Vincent Mooser, and Jean Pierre Hubaux. 2018. Protecting privacy and security of genomic data in i2b2 with homomorphic encryption and differential privacy. IEEE/ACM Trans. Comput. Biol. Bioinf. 15, 5 (2018), 1413–1426. DOI:https://doi.org/10.1109/TCBB.2018.2854782 Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Monjoy Saha and Chandan Chakraborty. 2018. Her2Net: A deep framework for semantic segmentation and classification of cell membranes and nuclei in breast cancer evaluation. IEEE Trans. Image Process. 27, 5 (2018), 2189–2200. DOI:https://doi.org/10.1109/TIP.2018.2795742Google ScholarGoogle ScholarCross RefCross Ref
  39. V. M. Silva-García, R. Flores-Carapia, C. Rentería-Márquez, B. Luna-Benoso, and M. Aldape-Pérez. 2018. Substitution box generation using Chaos: An image encryption application. Appl. Math. Comput. 332 (2018), 123–135. DOI:https://doi.org/10.1016/j.amc.2018.03.019Google ScholarGoogle Scholar
  40. Wenjuan Tang, Ju Ren, and Yaoxue Zhang. 2019. Enabling trusted and privacy-preserving healthcare services in social media health networks. IEEE Trans. Multimedia 21, 3 (2019), 579–590. DOI:https://doi.org/10.1109/TMM.2018.2889934Google ScholarGoogle ScholarCross RefCross Ref
  41. Hao Wang, Shuai Ding, Desheng Wu, Youtao Zhang, and Shanlin Yang. 2019. Smart connected electronic gastroscope system for gastric cancer screening using multi-column convolutional neural networks. Int. J. Prod. Res. 57, 21 (2019), 6795--6806. DOI:https://doi.org/10.1080/00207543.2018.1464232Google ScholarGoogle ScholarCross RefCross Ref
  42. Hongyu Wang, Jun Feng, Zizhao Zhang, Hai Su, Lei Cui, Hua He, and Li Liu. 2018. Breast mass classification via deeply integrating the contextual information from multi-view data. Pattern Recogn. 80 (2018), 42–52. DOI:https://doi.org/10.1016/j.patcog.2018.02.026Google ScholarGoogle ScholarCross RefCross Ref
  43. Chenchu Xu, Lei Xu, Zhifan Gao, Shen Zhao, Heye Zhang, and Yanping Zhang. 2018. Direct delineation of myocardial infarction without contrast agents using a joint motion feature learning architecture. Med. Image Anal. 50 (2018), 82–94. DOI:https://doi.org/10.1016/j.media.2018.09.001Google ScholarGoogle ScholarCross RefCross Ref
  44. Tao Xu, Han Zhang, Cheng Xin, Edward Kim, L. Rodney Long, Zhiyun Xue, Sameer Antani, and Xiaolei Huang. 2017. Multi-feature based benchmark for cervical dysplasia classification evaluation. Pattern Recogn. 63 (2017), 468–475. DOI:https://doi.org/10.1016/j.patcog.2016.09.027 Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Hao Yang, Chunfeng Yuan, Bing Li, Yang Du, Junliang Xing, Weiming Hu, and Stephen J. Maybank. 2019. Asymmetric 3D convolutional neural networks for action recognition. Pattern Recogn. 85 (2019), 1–12. DOI:https://doi.org/10.1016/j.patcog.2018.07.028Google ScholarGoogle ScholarCross RefCross Ref
  46. Aexander Hauptmann Yi Zhu, Zhenzhong Lan, Shawn Newsam. 2018. Hidden two-stream convolutional networks for action recognition. In Proceedings of the Asian Conference on Computer Vision. Springer International Publishing, 363–378. DOI:https://doi.org/10.1007/978-3-030-20893-6Google ScholarGoogle Scholar
  47. Qingchen Zhang, Laurence T. Yang, and Zhikui Chen. 2016. Privacy preserving deep computation model on cloud for big data feature learning. IEEE Trans. Comput. 65, 5 (2016), 1351–1362. DOI:https://doi.org/10.1109/TC.2015.2470255 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Zongjian Zhang, Qiang Wu, Yang Wang, and Fang Chen. 2018. High-quality image captioning with fine-grained and semantic-guided visual attention. IEEE Trans. Multimedia 21, 7 (2018), 1681–1693. DOI:https://doi.org/10.1109/TMM.2018.2888822Google ScholarGoogle ScholarCross RefCross Ref
  49. Chong zhi Gao, Qiong Cheng, Pei He, Willy Susilo, and Jin Li. 2018. Privacy-preserving Naive Bayes classifiers secure against the substitution-then-comparison attack. Inf. Sci. 444 (2018), 72–88. DOI:https://doi.org/10.1016/j.ins.2018.02.058Google ScholarGoogle ScholarCross RefCross Ref
  50. Yi Zhuang, Nan Jiang, Qing Li, Lei Chen, and Chunhua Ju. 2015. Progressive batch medical image retrieval processing in mobile wireless networks. ACM Trans. Internet Technol. 15, 3 (2015), 1–27. DOI:https://doi.org/10.1145/2783437 Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Privacy-preserving Time-series Medical Images Analysis Using a Hybrid Deep Learning Framework

              Recommendations

              Comments

              Login options

              Check if you have access through your login credentials or your institution to get full access on this article.

              Sign in

              Full Access

              • Published in

                cover image ACM Transactions on Internet Technology
                ACM Transactions on Internet Technology  Volume 21, Issue 3
                August 2021
                522 pages
                ISSN:1533-5399
                EISSN:1557-6051
                DOI:10.1145/3468071
                • Editor:
                • Ling Liu
                Issue’s Table of Contents

                Copyright © 2021 Association for Computing Machinery.

                Publisher

                Association for Computing Machinery

                New York, NY, United States

                Publication History

                • Published: 16 June 2021
                • Online AM: 7 May 2020
                • Accepted: 1 February 2020
                • Revised: 1 January 2020
                • Received: 1 November 2019
                Published in toit Volume 21, Issue 3

                Permissions

                Request permissions about this article.

                Request Permissions

                Check for updates

                Qualifiers

                • research-article
                • Refereed

              PDF Format

              View or Download as a PDF file.

              PDF

              eReader

              View online with eReader.

              eReader

              HTML Format

              View this article in HTML Format .

              View HTML Format
              About Cookies On This Site

              We use cookies to ensure that we give you the best experience on our website.

              Learn more

              Got it!