Abstract
Time-series medical images are an important type of medical data that contain rich temporal and spatial information. As a state-of-the-art, computer-aided diagnosis (CAD) algorithms are usually used on these image sequences to improve analysis accuracy. However, such CAD algorithms are often required to upload medical images to honest-but-curious servers, which introduces severe privacy concerns. To preserve privacy, the existing CAD algorithms support analysis on each encrypted image but not on the whole encrypted image sequences, which leads to the loss of important temporal information among frames. To meet this challenge, a convolutional-LSTM network, named HE-CLSTM, is proposed for analyzing time-series medical images encrypted by a fully homomorphic encryption mechanism. Specifically, several convolutional blocks are constructed to extract discriminative spatial features, and LSTM-based sequence analysis layers (HE-LSTM) are leveraged to encode temporal information from the encrypted image sequences. Moreover, a weighted unit and a sequence voting layer are designed to incorporate both spatial and temporal features with different weights to improve performance while reducing the missed diagnosis rate. The experimental results on two challenging benchmarks (a Cervigram dataset and the BreaKHis public dataset) provide strong evidence that our framework can encode visual representations and sequential dynamics from encrypted medical image sequences; our method achieved AUCs above 0.94 both on the Cervigram and BreaKHis datasets, constituting a significant margin of statistical improvement compared with several competing methods.
- Ahmed M. Alaa, Kyeong H. Moon, William Hsu, and Mihaela Van Der Schaar. 2016. ConfidentCare: A clinical decision support system for personalized breast cancer screening. IEEE Trans. Multimedia 18, 10 (2016), 1942–1955. DOI:https://doi.org/10.1109/TMM.2016.2589160Google Scholar
Cross Ref
- Shekoofeh Azizi, Sharareh Bayat, Pingkun Yan, Amir Tahmasebi, Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto, and Bradford Wood. 2018. Deep recurrent neural networks for prostate cancer detection: Analysis of temporal. IEEE Trans. Med. Imag. 37, 12 (2018), 2695–2703. DOI:https://doi.org/10.1109/TMI.2018.2849959Google Scholar
Cross Ref
- Ankan Kumar Bhunia, Aishik Konwer, Ayan Kumar Bhunia, Abir Bhowmick, Partha P. Roy, and Umapada Pal. 2019. Script identification in natural scene image and video frames using an attention based Convolutional-LSTM network. Pattern Recogn. 85 (2019), 172–184. DOI:https://doi.org/10.1016/j.patcog.2018.07.034Google Scholar
Cross Ref
- Sylvia F. Boj, Chang Il Hwang, Lindsey A. Baker, Iok In Christine Chio, Dannielle D. Engle, Vincenzo Corbo, Myrthe Jager, Mariano Ponz-Sarvise, Hervé Tiriac, Mona S. Spector, Ana Gracanin, Tobiloba Oni, Kenneth H. Yu, Ruben Van Boxtel, Meritxell Huch, Keith D. Rivera, John P. Wilson, Michael E. Feigin, Daniel Öhlund, Abram Handly-Santana, Christine M. Ardito-Abraham, Michael Ludwig, Ela Elyada, Brinda Alagesan, Giulia Biffi, Georgi N. Yordanov, Bethany Delcuze, Brianna Creighton, Kevin Wright, Youngkyu Park, Folkert H. M. Morsink, I. Quintus Molenaar, Inne H. Borel Rinkes, Edwin Cuppen, Yuan Hao, Ying Jin, Isaac J. Nijman, Christine Iacobuzio-Donahue, Steven D. Leach, Darryl J. Pappin, Molly Hammell, David S. Klimstra, Olca Basturk, Ralph H. Hruban, George Johan Offerhaus, Robert G. J. Vries, Hans Clevers and David A. Tuveson. 2015. Organoid models of human and mouse ductal pancreatic cancer. Cell 160, 1–2 (2015), 324–338. DOI:https://doi.org/10.1016/j.cell.2014.12.021Google Scholar
Cross Ref
- Joppe W. Bos, Kristin Lauter, Jake Loftus, and Michael Naehrig. 2013. Improved security for a ring-based fully homomorphic encryption scheme. In Proceedings of the IMA International Conference on Cryptography and Coding. 45–64. DOI:https://doi.org/10.1007/978-3-642-45239-0_4 Google Scholar
Digital Library
- Ümit Budak, Zafer Cömert, Zryan Najat Rashid, Abdulkadir Sengür, and Musa Çibuk. 2019. Computer-aided diagnosis system combining FCN and Bi-LSTM model for efficient breast cancer detection from histopathological images. Appl. Soft Comput. J. 85 (2019), 105765. DOI:https://doi.org/10.1016/j.asoc.2019.105765Google Scholar
Cross Ref
- Herve Chabanne, Amaury de Wargny, Jonathan Milgram, Constance Morel, and Emmanuel Prouff. 2017. Privacy-preserving classification on deep neural network. In Proceedings of the International Association for Cryptologic Research (IACR'17), 35.Google Scholar
- Christian Von Der Weth, Ashraf Abdul, Abhinav R. Kashyap, and Mohan S. Kankanhalli. 2019. CloseUp-community-driven live online search engine. ACM Trans. Internet Technol. 19, 3 (2019), 1–21. DOI:https://doi.org/10.1145/3301442 Google Scholar
Digital Library
- Shuai Ding, Shikang Hu, Jinxin Pan, Xiaojian Li, Gang Li, and Xiao Liu. 2019. A homogeneous ensemble method for predicting gastric cancer based on gastroscopy reports. Expert Syst. 37, 3 (2019), e12499. DOI:https://doi.org/10.1111/exsy.12499Google Scholar
- Shuai Ding, Ling Li, Zhenmin Li, Hao Wang, and Yanchun Zhang. 2019. Smart electronic gastroscope system using a cloudâedge collaborative framework. Fut. Gener. Comput. Syst. 100 (2019), 395–407. DOI:https://doi.org/10.1016/j.future.2019.04.031Google Scholar
Cross Ref
- Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Trevor Darrell. 2017. Long-term recurrent convolutional networks for visual recognition and description. IEEE Trans. Pattern Anal. Mach. Intell. 39, 4 (2017), 677–691. DOI:https://doi.org/10.1109/TPAMI.2016.2599174 Google Scholar
Digital Library
- Nathan Dowlin, Ran Gilad-Bachrach, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. 2016. CryptoNets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of the International Conference on Machine Learning. 201–210. Google Scholar
Digital Library
- Li Duan, Dongxi Liu, Yang Zhang, Shiping Chen, Ren Ping Liu, Bo Cheng, and Junliang Chen. 2016. Secure data-centric access control for smart grid services based on publish/subscribe systems. ACM Trans. Internet Technol. 16, 4 (2016), 1–17. DOI:https://doi.org/10.1145/3007190 Google Scholar
Digital Library
- Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, and Sebastian Thrun. 2017. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 7639 (2017), 115–118. DOI:https://doi.org/10.1038/nature21056Google Scholar
- Junfeng Fan and Frederik Vercauteren. 2012. Somewhat practical fully homomorphic encryption. In Proceedings of the International Association for Cryptologic Research (IACR'12). 144.Google Scholar
- Jeremy Farrar. 2014. Medical data: Widen use in research. Nature 506, 7486 (2014), 34. DOI:https://doi.org/10.1038/506034bGoogle Scholar
- Craig Gentry. 2009. A Fully Homomorphic Encryption Scheme. Stanford University (2009). DOI:https://doi.org/10.1145/1536414.1536440 Google Scholar
Digital Library
- Francisco Javier González-Serrano, Ángel Navia-Vázquez, and Adrián Amor-Martín. 2017. Training support vector machines with privacy-protected data. Pattern Recogn. 72 (2017), 93–107. DOI:https://doi.org/10.1016/j.patcog.2017.06.016 Google Scholar
Digital Library
- Varun Gulshan, Lily Peng, Marc Coram, Martin C. Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica L. Mega, and Dale R. Webster. 2016. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J. Am. Med. Assoc. 316, 22 (2016), 2402–2410. DOI:https://doi.org/10.1001/jama.2016.17216Google Scholar
Cross Ref
- Zhongyun Hua, Shuang Yi, and Yicong Zhou. 2018. Medical image encryption using high-speed scrambling and pixel adaptive diffusion. Signal Process. 144 (2018), 134–144. DOI:https://doi.org/10.1016/j.sigpro.2017.10.004 Google Scholar
Digital Library
- Multi-view Inference, Yi Zhou, Student Member, Li Liu, Ling Shao, and Senior Member. 2018. Vehicle re-identification by deep hidden multi-view inference. IEEE Trans. Image Process. 27, 7 (2018), 3275–3287.Google Scholar
Cross Ref
- Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2013. 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1 (2013), 221–231. DOI:https://doi.org/10.1109/TPAMI.2012.59 Google Scholar
Digital Library
- Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’09). 248–255. DOI:https://doi.org/10.1109/CVPRW.2009.5206848Google Scholar
Cross Ref
- Yueming Jin, Qi Dou, Hao Chen, Lequan Yu, Jing Qin, Chi Wing Fu, and Pheng Ann Heng. 2017. SV-RCNet: Workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans. Med. Imag. 37, 5 (2017), 1114–1126. DOI:https://doi.org/10.1109/TMI.2017.2787657Google Scholar
Cross Ref
- Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C. S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei, Magdalena Ting, Jie Zhu, Christina Li, Sierra Hewett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang, Lianghong Zheng, Rui Hou, William Shi, Xin Fu, Yaou Duan, Viet A. N. Huu, Cindy Wen, Edward D. Zhang, Charlotte L. Zhang, Oulan Li, Xiaobo Wang, Michael A. Singer, Xiaodong Sun, Jie Xu, Ali Tafreshi, M. Anthony Lewis, Huimin Xia and Kang Zhang. 2018. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 5 (2018), 1122–1131. DOI:https://doi.org/10.1016/j.cell.2018.02.010Google Scholar
Cross Ref
- Neal Koblitz. 1987. Elliptic curve cryptosystems. Math. Comp. 48, 177 (1987), 203–203. DOI:https://doi.org/10.1090/S0025-5718-1987-0866109-5Google Scholar
Digital Library
- Sung-hyun Lee, Joon Hyun Song, and Il Kon Kim. 2016. CDA generation and integration for health information exchange based on cloud computing system. IEEE Trans. Serv. Comput. 9, 2 (2016), 241–249. DOI:https://doi.org/10.1109/TSC.2014.2363654 Google Scholar
Digital Library
- X.-B. Li and Jialun Qin. 2017. Anonymizing and sharing medical text records. Inf. Syst. Res. 28, 2 (2017), 332–352. DOI:https://doi.org/10.1287/isre.2016.0676 Google Scholar
Digital Library
- Yang Liu, Shanyu Tang, Ran Liu, Liping Zhang, and Zhao Ma. 2018. Secure and robust digital image watermarking scheme using logistic and RSA encryption. Expert Syst. Appl. 97 (2018), 95–105. DOI:https://doi.org/10.1016/j.eswa.2017.12.003Google Scholar
Cross Ref
- Chao Ma, Yulan Guo, Jungang Yang, and Wei An. 2019. Learning multi-view representation with LSTM for 3-D shape recognition and retrieval. IEEE Trans. Multimedia 21, 5 (2019), 1169–1182. DOI:https://doi.org/10.1109/TMM.2018.2875512Google Scholar
Digital Library
- Ahmet Can Mert, Erdinc Ozturk, and Erkay Savas. 2019. Design and implementation of encryption/decryption architectures for bfv homomorphic encryption scheme. IEEE Trans. VLSI Syst. 28, 2 (2019), 353--362. DOI:https://doi.org/10.1109/tvlsi.2019.2943127Google Scholar
Cross Ref
- Biotecnologia Microbiana, De Hidalgo, Marios Anthimopoilos, Srergios Christodoulidis, Lukas Ebner, Andreas Christe, and Stavroula Mougiakakou. 2016. Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans. Med. Imag. 35, 5 (2016), 1207–1216. DOI:https://doi.org/10.1109/TMI.2016.2535865Google Scholar
Cross Ref
- Declan O’Loughlin, Barbara L. Oliveira, Adam Santorelli, Emily Porter, Martin Glavin, Edward Jones, Milica Popovic, and Martin O’Halloran. 2019. Sensitivity and specificity estimation using patient-specific microwave imaging in diverse experimental breast phantoms. IEEE Trans. Med. Imag. 38, 1 (2019), 303–311. DOI:https://doi.org/10.1109/TMI.2018.2864150Google Scholar
Cross Ref
- Jane S. Paulsen, Jeff D. Long, Christopher A. Ross, Deborah L. Harrington, Cheryl J. Erwin, Janet K. Williams, Holly James Westervelt, and Hans J. Johnson. 2014. Prediction of manifest Huntington’s disease with clinical and imaging measures: A prospective observational study. Lancet Neurol. 13, 12 (2014), 1193–1201. DOI:https://doi.org/10.1016/S1474-4422(14)70238-8Google Scholar
Cross Ref
- W. Nicholson Price and Glenn Cohen. 2019. Privacy in the age of medical big data. Nat. Med. 25, 1 (2019), 37–43. DOI:https://doi.org/10.1038/s41591-018-0272-7Google Scholar
Cross Ref
- Ahmad Qaisar Ahmad Al Badawi, Yuriy Polyakov, Khin Mi Mi Aung, Bharadwaj Veeravalli, and Kurt Rohloff. 2019. Implementation and performance evaluation of RNS variants of the BFV homomorphic encryption scheme. IEEE Trans. Emerg. Topics Comput. (2019). DOI:https://doi.org/10.1109/tetc.2019.2902799Google Scholar
- Jean Louis Raisaro, Gwangbae Choi, Sylvain Pradervand, Raphael Colsenet, Nathalie Jacquemont, Nicolas Rosat, Vincent Mooser, and Jean Pierre Hubaux. 2018. Protecting privacy and security of genomic data in i2b2 with homomorphic encryption and differential privacy. IEEE/ACM Trans. Comput. Biol. Bioinf. 15, 5 (2018), 1413–1426. DOI:https://doi.org/10.1109/TCBB.2018.2854782 Google Scholar
Digital Library
- Monjoy Saha and Chandan Chakraborty. 2018. Her2Net: A deep framework for semantic segmentation and classification of cell membranes and nuclei in breast cancer evaluation. IEEE Trans. Image Process. 27, 5 (2018), 2189–2200. DOI:https://doi.org/10.1109/TIP.2018.2795742Google Scholar
Cross Ref
- V. M. Silva-García, R. Flores-Carapia, C. Rentería-Márquez, B. Luna-Benoso, and M. Aldape-Pérez. 2018. Substitution box generation using Chaos: An image encryption application. Appl. Math. Comput. 332 (2018), 123–135. DOI:https://doi.org/10.1016/j.amc.2018.03.019Google Scholar
- Wenjuan Tang, Ju Ren, and Yaoxue Zhang. 2019. Enabling trusted and privacy-preserving healthcare services in social media health networks. IEEE Trans. Multimedia 21, 3 (2019), 579–590. DOI:https://doi.org/10.1109/TMM.2018.2889934Google Scholar
Cross Ref
- Hao Wang, Shuai Ding, Desheng Wu, Youtao Zhang, and Shanlin Yang. 2019. Smart connected electronic gastroscope system for gastric cancer screening using multi-column convolutional neural networks. Int. J. Prod. Res. 57, 21 (2019), 6795--6806. DOI:https://doi.org/10.1080/00207543.2018.1464232Google Scholar
Cross Ref
- Hongyu Wang, Jun Feng, Zizhao Zhang, Hai Su, Lei Cui, Hua He, and Li Liu. 2018. Breast mass classification via deeply integrating the contextual information from multi-view data. Pattern Recogn. 80 (2018), 42–52. DOI:https://doi.org/10.1016/j.patcog.2018.02.026Google Scholar
Cross Ref
- Chenchu Xu, Lei Xu, Zhifan Gao, Shen Zhao, Heye Zhang, and Yanping Zhang. 2018. Direct delineation of myocardial infarction without contrast agents using a joint motion feature learning architecture. Med. Image Anal. 50 (2018), 82–94. DOI:https://doi.org/10.1016/j.media.2018.09.001Google Scholar
Cross Ref
- Tao Xu, Han Zhang, Cheng Xin, Edward Kim, L. Rodney Long, Zhiyun Xue, Sameer Antani, and Xiaolei Huang. 2017. Multi-feature based benchmark for cervical dysplasia classification evaluation. Pattern Recogn. 63 (2017), 468–475. DOI:https://doi.org/10.1016/j.patcog.2016.09.027 Google Scholar
Digital Library
- Hao Yang, Chunfeng Yuan, Bing Li, Yang Du, Junliang Xing, Weiming Hu, and Stephen J. Maybank. 2019. Asymmetric 3D convolutional neural networks for action recognition. Pattern Recogn. 85 (2019), 1–12. DOI:https://doi.org/10.1016/j.patcog.2018.07.028Google Scholar
Cross Ref
- Aexander Hauptmann Yi Zhu, Zhenzhong Lan, Shawn Newsam. 2018. Hidden two-stream convolutional networks for action recognition. In Proceedings of the Asian Conference on Computer Vision. Springer International Publishing, 363–378. DOI:https://doi.org/10.1007/978-3-030-20893-6Google Scholar
- Qingchen Zhang, Laurence T. Yang, and Zhikui Chen. 2016. Privacy preserving deep computation model on cloud for big data feature learning. IEEE Trans. Comput. 65, 5 (2016), 1351–1362. DOI:https://doi.org/10.1109/TC.2015.2470255 Google Scholar
Digital Library
- Zongjian Zhang, Qiang Wu, Yang Wang, and Fang Chen. 2018. High-quality image captioning with fine-grained and semantic-guided visual attention. IEEE Trans. Multimedia 21, 7 (2018), 1681–1693. DOI:https://doi.org/10.1109/TMM.2018.2888822Google Scholar
Cross Ref
- Chong zhi Gao, Qiong Cheng, Pei He, Willy Susilo, and Jin Li. 2018. Privacy-preserving Naive Bayes classifiers secure against the substitution-then-comparison attack. Inf. Sci. 444 (2018), 72–88. DOI:https://doi.org/10.1016/j.ins.2018.02.058Google Scholar
Cross Ref
- Yi Zhuang, Nan Jiang, Qing Li, Lei Chen, and Chunhua Ju. 2015. Progressive batch medical image retrieval processing in mobile wireless networks. ACM Trans. Internet Technol. 15, 3 (2015), 1–27. DOI:https://doi.org/10.1145/2783437 Google Scholar
Digital Library
Index Terms
Privacy-preserving Time-series Medical Images Analysis Using a Hybrid Deep Learning Framework
Recommendations
Privacy Preserving Signcryption Scheme
ICDCIT 2017: 13th International Conference on Distributed Computing and Internet Technology - Volume 10109Signcryption is a public-key cryptographic primitive that is a synthesis of encryption and digital signature schemes. We present a signcryption scheme using anonymous attribute based encryption. The scheme uses the notion of identity-based digital ...
Image inpainting based on deep learning: A review
AbstractImage inpainting is an important research direction in the study of computer vision, and is widely used in image editing and photo inpainting etc. Traditional image inpainting algorithms are often difficult to deal with large-scale ...
Highlights- Reviewing the image inpainting on deep learning of the past 15 years.
- ...
Publicly Verifiable Privacy-Preserving Group Decryption
Information Security and CryptologyAnonymity is one of the main concerns in group-oriented cryptography. However, most efforts, for instance, group signatures and ring signatures, are only made to provide anonymity on the sender's side. There are merely few works done to ensure anonymity ...






Comments