skip to main content
research-article

Smartphone-based Handwritten Signature Verification using Acoustic Signals

Authors Info & Claims
Published:05 November 2021Publication History
Skip Abstract Section

Abstract

Handwritten signature verification techniques, which can facilitate user authentication and enable secure information exchange, are still important in property safety. However, on-line automatic handwritten signature verification usually requires dynamic handwritten patterns captured by a special device, such as a sensor-instrumented pen, a tablet or a smartwatch on the dominant hand. This paper presents SonarSign, an on-line handwritten signature verification system based on inaudible acoustic signals. The key insight is to use acoustic signals to capture the dynamic handwritten signature patterns for verification. Particularly, SonarSign exploits the built-in speakers and microphones of smartphones to transmit a specially designed training sequence and record the corresponding echo for channel impulse response (CIR) estimation, respectively. Based on the sensitivity of CIR to the tiny surrounding environment changes including handwritten signature actions, SonarSign designs an attentional multi-modal Siamese network for end-to-end signatures verification. First, multi-modal CIR streams are fused to extract representative signature pattern features from spatio-temporal dimensions. Then an attentional Siamese network is elaborated to verify whether the given two signatures are from the same signatory. Extensive experiments in real-world scenarios show that SonarSign can achieve accurate and robust signatures verification with an AUC (Area Under ROC (Receiver Operating Characteristic) Curve) of 98.02% and an EER (Equal Error Rate) of 5.79% for unseen users.

References

  1. Kian Ahrabian and Bagher BabaAli. 2019. Usage of autoencoders and Siamese networks for online handwritten signature verification. Neural Computing and Applications 31, 12 (2019), 9321--9334.Google ScholarGoogle ScholarCross RefCross Ref
  2. Gabe Alvarez, Blue Sheffer, and Morgan Bryant. 2016. Offline signature verification with convolutional neural networks. Technical report, Stanford University (2016).Google ScholarGoogle Scholar
  3. Rummana Bari, Roy J Adams, Md Mahbubur Rahman, Megan Battles Parsons, Eugene H Buder, and Santosh Kumar. 2018. rconverse: Moment by moment conversation detection using a mobile respiration sensor. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 2, 1 (2018), 1--27.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Luana Batista, Eric Granger, and Robert Sabourin. 2012. Dynamic selection of generative--discriminative ensembles for off-line signature verification. Pattern Recognition 45, 4 (2012), 1326--1340.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Mengqi Chen, Jiawei Lin, Yongpan Zou, Rukhsana Ruby, and Kaishun Wu. 2020. SilentSign: Device-free Handwritten Signature Verification through Acoustic Sensing. In 2020 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 1--10.Google ScholarGoogle ScholarCross RefCross Ref
  6. Cao Dian, Dong Wang, Qian Zhang, Run Zhao, and Yinggang Yu. 2020. Towards Domain-independent Complex and Fine-grained Gesture Recognition with RFID. Proceedings of the ACM on Human-Computer Interaction 4, ISS (2020), 1--22.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Moises Diaz, Miguel A Ferrer, Donato Impedovo, Muhammad Imran Malik, Giuseppe Pirlo, and Réjean Plamondon. 2019. A perspective analysis of handwritten signature technology. Acm Computing Surveys 51, 6 (2019), 1--39.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Moises Diaz, Andreas Fischer, Miguel A Ferrer, and Réjean Plamondon. 2016. Dynamic signature verification system based on one real signature. IEEE transactions on cybernetics 48, 1 (2016), 228--239.Google ScholarGoogle Scholar
  9. Feng Ding, Dong Wang, Qian Zhang, and Run Zhao. 2019. ASSV: handwritten signature verification using acoustic signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1--22.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Tom Fawcett. 2006. An introduction to ROC analysis. Pattern recognition letters 27, 8 (2006), 861--874.Google ScholarGoogle Scholar
  11. Miguel A Ferrer, Jesus B Alonso, and Carlos M Travieso. 2005. Offline geometric parameters for automatic signature verification using fixed-point arithmetic. IEEE transactions on pattern analysis and machine intelligence 27, 6 (2005), 993--997.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Miguel A Ferrer, J Francisco Vargas, Aythami Morales, and Aaron Ordonez. 2012. Robustness of offline signature verification based on gray level features. IEEE Transactions on Information Forensics and Security 7, 3 (2012), 966--977.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Andreas Fischer and Réjean Plamondon. 2016. Signature verification based on the kinematic theory of rapid human movements. IEEE Transactions on Human-Machine Systems 47, 2 (2016), 169--180.Google ScholarGoogle ScholarCross RefCross Ref
  14. G. H. Givens, J. R. Beveridge, P. J. Phillips, B. Draper, Y. M. Lui, and D. Bolme. 2013. Introduction to face recognition and evaluation of algorithm performance. Computational Statistics & Data Analysis 67, 11 (2013), 236--247.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Marta Gomez-Barrero, Javier Galbally, Julian Fierrez, Javier Ortega-Garcia, and Réjean Plamondon. 2015. Enhanced on-line signature verification based on skilled forgery detection using sigma-lognormal features. In 2015 international conference on biometrics (ICB). IEEE, 501--506.Google ScholarGoogle ScholarCross RefCross Ref
  16. Donato Impedovo and Giuseppe Pirlo. 2008. Automatic signature verification: The state of the art. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38, 5 (2008), 609--635.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. PMLR, 448--456.Google ScholarGoogle Scholar
  18. Nidal S Kamel, Shohel Sayeed, and Grant A Ellis. 2008. Glove-based approach to online signature verification. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 6 (2008), 1109--1113.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, Vol. 2. Lille.Google ScholarGoogle Scholar
  20. Bence Kovari and Hassan Charaf. 2013. A study on the consistency and significance of local features in off-line signature verification. Pattern Recognition Letters 34, 3 (2013), 247--255.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Songxuan Lai, Lianwen Jin, and Weixin Yang. 2017. Online signature verification using recurrent neural network and length-normalized path signature descriptor. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), Vol. 1. IEEE, 400--405.Google ScholarGoogle ScholarCross RefCross Ref
  22. Neal S Latman and Emily Herb. 2013. A field study of the accuracy and reliability of a biometric iris recognition system. Science & Justice Journal of the Forensic Science Society 53, 2 (2013), 98--102.Google ScholarGoogle ScholarCross RefCross Ref
  23. Alona Levy, Ben Nassi, Yuval Elovici, and Erez Shmueli. 2018. Handwritten signature verification using wrist-worn devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1--26. Proc. ACM Hum.-Comput. Interact., Vol. 5, No. ISS, Article 499. Publication date: November 2021. Smartphone-based Handwritten Signature Verification using Acoustic Signals 499:25Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Dong Li, Jialin Liu, Sunghoon Ivan Lee, and Jie Xiong. 2020. FM-track: pushing the limits of contactless multi-target tracking using acoustic signals. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems. 150--163.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Kang Ling, Haipeng Dai, Yuntang Liu, Alex X Liu, Wei Wang, and Qing Gu. 2020. Ultragesture: Fine-grained gesture sensing and recognition. IEEE Transactions on Mobile Computing (2020).Google ScholarGoogle Scholar
  26. Jialin Liu, Dong Li, Lei Wang, and Jie Xiong. 2021. BlinkListener: " Listen" to Your Eye Blink Using Your Smartphone. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (2021), 1--27.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Li Lu, Jiadi Yu, Yingying Chen, Hongbo Liu, Yanmin Zhu, Yunfei Liu, and Minglu Li. 2018. LipPass: Lip Readingbased User Authentication on Smartphones Leveraging Acoustic Signals. In 2018 IEEE Conference on Computer Communications. IEEE, 1466--1474.Google ScholarGoogle Scholar
  28. Abhishek Malviya and Himanshu Yadav. 2020. An Assessment on Credit Card Fraud Detection: Survey. International Journal of Advanced Computer Technology 9, 5 (2020), 12--15.Google ScholarGoogle Scholar
  29. Wenguang Mao, Jian He, and Lili Qiu. 2016. Cat: high-precision acoustic motion tracking. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. 69--81.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Wenguang Mao, Wei Sun, Mei Wang, and Lili Qiu. 2020. DeepRange: Acoustic Ranging via Deep Learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--23.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Javier Ortega-Garcia, Julian Fierrez, Fernando Alonso-Fernandez, Javier Galbally, Manuel R Freire, Joaquin Gonzalez- Rodriguez, Carmen Garcia-Mateo, Jose-Luis Alba-Castro, Elisardo Gonzalez-Agulla, Enrique Otero-Muras, et al. 2009. The multiscenario multienvironment biosecure multimodal database (bmdb). IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 6 (2009), 1097--1111.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Dominique A Potvin, Kirsten M Parris, and Raoul A Mulder. 2011. Geographically pervasive effects of urban noise on frequency and syllable rate of songs and calls in silvereyes (Zosterops lateralis). Proceedings of the Royal Society B: Biological Sciences 278, 1717 (2011), 2464--2469.Google ScholarGoogle ScholarCross RefCross Ref
  33. Markku Pukkila. 2000. Channel estimation modeling. Nokia Research Center 17 (2000), 66.Google ScholarGoogle Scholar
  34. Yingyong Qi and Bobby R Hunt. 1994. Signature verification using global and grid features. Pattern Recognition 27, 12 (1994), 1621--1629.Google ScholarGoogle ScholarCross RefCross Ref
  35. A Rodríguez Valiente, A Trinidad, JR García Berrocal, C Górriz, and R Ramírez Camacho. 2014. Extended high-frequency (9--20 kHz) audiometry reference thresholds in 645 healthy subjects. International journal of audiology 53, 8 (2014), 531--545.Google ScholarGoogle Scholar
  36. Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. 2018. How does batch normalization help optimization?. In Advances in Neural Information Processing Systems. 2483--2493.Google ScholarGoogle Scholar
  37. Abhishek Sharma and Suresh Sundaram. 2016. An enhanced contextual DTW based system for online signature verification using vector quantization. Pattern Recognition Letters 84 (2016), 22--28.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Michael Stauffer, Paul Maergner, Andreas Fischer, Rolf Ingold, and Kaspar Riesen. 2019. Offline signature verification using structural dynamic time warping. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, 1117--1124.Google ScholarGoogle ScholarCross RefCross Ref
  39. Michael Stauffer, Paul Maergner, Andreas Fischer, and Kaspar Riesen. 2020. A survey of state of the art methods employed in the offline signature verification process. In New Trends in Business Information Systems and Technology. Springer, 17--30.Google ScholarGoogle Scholar
  40. Ke Sun and Xinyu Zhang. 2021. UltraSE: single-channel speech enhancement using ultrasound. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 160--173.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Ke Sun, Ting Zhao, Wei Wang, and Lei Xie. 2018. Vskin: Sensing touch gestures on surfaces of mobile devices using acoustic signals. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. 591--605.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Kalaivani Sundararajan and Damon LWoodard. 2018. Deep learning for biometrics: A survey. ACM Computing Surveys (CSUR) 51, 3 (2018), 1--34.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Jiayao Tan, Cam-Tu Nguyen, and Xiaoliang Wang. 2017. SilentTalk: Lip reading through ultrasonic sensing on mobile phones. In IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 1--9.Google ScholarGoogle ScholarCross RefCross Ref
  44. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008).Google ScholarGoogle Scholar
  45. Dangxiao Wang, Yuru Zhang, Chong Yao, Jun Wu, Huimin Jiao, and Muli Liu. 2009. Toward force-based signature verification: A pen-type sensor and preliminary validation. IEEE Transactions on Instrumentation and Measurement 59, 4 (2009), 752--762.Google ScholarGoogle ScholarCross RefCross Ref
  46. LeiWang, Xiang Zhang, Yuanshuang Jiang, Yong Zhang, Chenren Xu, Ruiyang Gao, and Daqing Zhang. 2021. Watching Your Phone's Back: Gesture Recognition by Sensing Acoustical Structure-borne Propagation. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (2021), 1--26.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Yanwen Wang, Jiaxing Shen, and Yuanqing Zheng. 2020. Push the Limit of Acoustic Gesture Recognition. IEEE Transactions on Mobile Computing (2020). Proc. ACM Hum.-Comput. Interact., Vol. 5, No. ISS, Article 499. Publication date: November 2021. 499:26 Run Zhao et al.Google ScholarGoogle Scholar
  48. Reid Kress Weisbord and David Horton. 2019. Inheritance Forgery. Duke LJ 69 (2019), 855.Google ScholarGoogle Scholar
  49. Wei Xu, ZhiWen Yu, ZhuWang, Bin Guo, and Qi Han. 2019. Acousticid: gait-based human identification using acoustic signal. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1--25.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Mohammad E Yahyatabar and Jamal Ghasemi. 2017. Online signature verification using double-stage feature extraction modelled by dynamic feature stability experiment. IET Biometrics 6, 6 (2017), 393--401.Google ScholarGoogle ScholarCross RefCross Ref
  51. E Yakhnich. 2001. Channel estimation for EGPRS modems. In IEEE VTS 53rd Vehicular Technology Conference, Spring 2001. Proceedings (Cat. No. 01CH37202), Vol. 1. IEEE, 419--422.Google ScholarGoogle ScholarCross RefCross Ref
  52. Jiadi Yu, Li Lu, Yingying Chen, Yanmin Zhu, and Linghe Kong. 2019. An indirect eavesdropping attack of keystrokes on touch screen through acoustic sensing. IEEE Transactions on Mobile Computing (2019).Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-grained acoustic-based device-free tracking. In Proceedings of the 15th annual international conference on mobile systems, applications, and services. 15--28.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Cheng Zhang, AbdelKareem Bedri, Gabriel Reyes, Bailey Bercik, Omer T Inan, Thad E Starner, and Gregory D Abowd. 2016. TapSkin: Recognizing on-skin input for smartwatches. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces. 13--22.Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. David Zhang. 2002. Biometric Solutions. 263--288 pages.Google ScholarGoogle Scholar
  56. Fusang Zhang, Zhi Wang, Beihong Jin, Jie Xiong, and Daqing Zhang. 2020. Your Smart Speaker Can" Hear" Your Heartbeat! Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--24.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Maotian Zhang, Qian Dai, Panlong Yang, Jie Xiong, Chang Tian, and Chaocan Xiang. 2018. idial: Enabling a virtual dial plate on the hand back for around-device interaction. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1--20.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Qian Zhang, Dong Wang, Run Zhao, and Yinggang Yu. 2021. SoundLip: Enabling Word and Sentence-level Lip Interaction for Smart Devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (2021), 1--28.Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Qian Zhang, Dong Wang, Run Zhao, Yinggang Yu, and Junjie Shen. 2021. Sensing to Hear: Speech Enhancement for Mobile Devices Using Acoustic Signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 3 (2021), 1--30.Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Yongzhao Zhang, Wei-Hsiang Huang, Chih-Yun Yang, Wen-Ping Wang, Yi-Chao Chen, Chuang-Wen You, Da-Yuan Huang, Guangtao Xue, and Jiadi Yu. 2020. Endophasia: Utilizing Acoustic-Based Imaging for Issuing Contact-Free Silent Speech Commands. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--26.Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Yichi Zhang, Meng Yu, Na Li, Chengzhu Yu, Jia Cui, and Dong Yu. 2019. Seq2seq attentional siamese neural networks for text-dependent speaker verification. In ICASSP 2019--2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 6131--6135.Google ScholarGoogle ScholarCross RefCross Ref
  62. Bing Zhou, Jay Lohokare, Ruipeng Gao, and Fan Ye. 2018. EchoPrint: Two-factor authentication using acoustics and vision on smartphones. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. 321--336.Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Yongpan Zou, Meng Zhao, Zimu Zhou, Jiawei Lin, Mo Li, and Kaishun Wu. 2018. BiLock: User authentication via dental occlusion biometrics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1--20.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Smartphone-based Handwritten Signature Verification using Acoustic Signals

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!