skip to main content
research-article

A Surgical Scene Replay System for Learning Gastroenterological Endoscopic Surgery Skill by Multiple Synchronized-Video and Gaze Representation

Published:29 May 2021Publication History
Skip Abstract Section

Abstract

Gastroenterological endoscopic surgery needs complex surgical skills such as a sensation of body movement and manipulation of the endoscope that is hard to be explained verbally. Prior research reported that endoscopic surgery is one of the most challenging surgery to teach. Thus, surgeons need long-term practice to master the skills. To support learning such skills, we developed a surgical scene replay system. First, we surveyed 12 surgeons to reveal the reason for the difficulty and elicited three requirements for our system: (1) provide multiple videos that include an endoscope, a fluoroscopy, and hand manipulation for the endoscope to observe the surgery from multiple aspects; (2) visualize an experts' gaze position to understand experts' intention of hand manipulation, and (3) enlarge the size of the gazed video to inform learners where they should pay attention to. Our user study with the system indicates that participants could understand the experts' intentions and tacit knowledge easier than the existing video materials.

References

  1. Deepak Akkil and Poika Isokoski. 2016. Gaze Augmentation in Egocentric Video Improves Awareness of Intention. Association for Computing Machinery, New York, NY, USA, 1573--1584. https://doi.org/10.1145/2858036.2858127 Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Knut Augestad, Khayam Butt, Dejan Ignjatovic, Deborah Keller, and Ravi Kiran. 2020. Video-based coaching in surgical education: a systematic review and meta-analysis. Surgical Endoscopy 34 (02 2020). https://doi.org/10.1007/s00464- 019-07265-0Google ScholarGoogle Scholar
  3. Timothy J. Babineau, James Becker, Gary Gibbons, Stephen Sentovich, Donald Hess, Sharon Robertson, and Michael Stone. 2004. The "Cost" of Operative Training for Surgical Residents. Archives of Surgery 139, 4 (04 2004), 366--370. https://doi.org/10.1001/archsurg.139.4.366 arXiv:https://jamanetwork.com/journals/jamasurgery/articlepdf/396495/sne3006.pdfGoogle ScholarGoogle Scholar
  4. Ido Badash, Karen Burtt, Carlos A Solorzano, and Joseph N Carey. 2016. Innovations in surgery simulation: a review of past, current and future techniques. Annals of translational medicine 4, 23 (dec 2016), 453. https://doi.org/10.21037/ atm.2016.12.24Google ScholarGoogle ScholarCross RefCross Ref
  5. Guillermo Bernal, Dishaan Ahuja, and Federico Casalegno. 2015. EMG-Based Biofeedback Tool for Augmenting Manual Fabrication and Improved Exchange of Empirical Knowledge. In Proceedings of the XVI International Conference on Human Computer Interaction (Vilanova i la Geltru, Spain) (Interacción '15). Association for Computing Machinery, New York, NY, USA, Article 61, 8 pages. https://doi.org/10.1145/2829875.2829932 Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Matthew Bridges and Daniel L Diamond. 1999. The financial impact of teaching surgical residents in the operating room. The American Journal of Surgery 177, 1 (1999), 28 -- 32. https://doi.org/10.1016/S0002--9610(98)00289-XGoogle ScholarGoogle ScholarCross RefCross Ref
  7. Diogo Cabral, João G. Valente, Urândia Aragão, Carla Fernandes, and Nuno Correia. 2012. Evaluation of a Multimodal Video Annotator for Contemporary Dance. In Proceedings of the International Working Conference on Advanced Visual Interfaces (Capri Island, Italy) (AVI '12). Association for Computing Machinery, New York, NY, USA, 572--579. https://doi.org/10.1145/2254556.2254663 Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Lin Chen, Peng Zhang, and Baoxin Li. 2014. Instructive Video Retrieval Based on Hybrid Ranking and Attribute Learning: A Case Study on Surgical Skill Training. In Proceedings of the 22nd ACM International Conference on Multimedia (Orlando, Florida, USA) (MM '14). Association for Computing Machinery, New York, NY, USA, 1045--1048. https://doi.org/10.1145/2647868.2655050 Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Pei-Yu Chi, Joyce Liu, Jason Linder, Mira Dontcheva, Wilmot Li, and Bjoern Hartmann. 2013. DemoCut: Generating Concise Instructional Videos for Physical Demonstrations. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST '13). Association for Computing Machinery, New York, NY, USA, 141--150. https://doi.org/10.1145/2501988.2502052 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Chun-Hong Huang, T. L. Won, Chun-Yeh Liu, Yen-Da Chen, and Yung-Hui Chen. 2009. Multiple-video-based Elearning Platform for Physical Education. In 2009 Joint Conferences on Pervasive Computing (JCPC). 21--26. https: //doi.org/10.1109/JCPC.2009.5420220Google ScholarGoogle Scholar
  11. Anender Dhariwal, Ram Prabhu, Abhay Dalvi, and Avinash Supe. 2007. Effectiveness of box trainers in laparoscopic training. Journal of minimal access surgery 3 (04 2007), 57--63. https://doi.org/10.4103/0972--9941.33274Google ScholarGoogle Scholar
  12. Diana L. Diesen, Loretta Erhunmwunsee, Kyla M. Bennett, Kfir Ben-David, Basil Yurcisin, Eugene P. Ceppa, Philip A. Omotosho, Alexander Perez, and Aurora Pryor. 2011. Effectiveness of Laparoscopic Computer Simulator Versus Usage of Box Trainer for Endoscopic Surgery Training of Novices. Journal of Surgical Education 68, 4 (2011), 282 -- 289. https://doi.org/10.1016/j.jsurg.2011.02.007Google ScholarGoogle ScholarCross RefCross Ref
  13. Adam J. Goodman, Joshua Melson, Harry R. Aslanian, Manoop S. Bhutani, Kumar Krishnan, David R. Lichtenstein, Udayakumar Navaneethan, Rahul Pannala, Mansour A. Parsi, Allison R. Schulman, Amrita Sethi, Shelby A. Sullivan, Nirav Thosani, Guru Trikudanathan, Arvind J. Trindade, Rabindra R. Watson, and John T. Maple. 2019. Endoscopic simulators. Gastrointestinal Endoscopy 90, 1 (2019), 1 -- 12. https://doi.org/10.1016/j.gie.2018.10.037Google ScholarGoogle ScholarCross RefCross Ref
  14. Kristine Hagelsteen, Richard Johansson, Mikael Ekelund, Anders Bergenfelz, and Magnus Anderberg. 2019. Performance and perception of haptic feedback in a laparoscopic 3D virtual reality simulator. Minimally Invasive Therapy & Allied Technologies 28, 5 (2019), 309--316. https://doi.org/10.1080/13645706.2018.1539012 arXiv:https://doi.org/10.1080/13645706.2018.1539012 PMID: 30663462.Google ScholarGoogle ScholarCross RefCross Ref
  15. Randy S. Haluck, Renee L. Marshall, Thomas M. Krummel, and Michael G. Melkonian. 2001. Are surgery training programs ready for virtual reality? a survey of program directors in general surgery1 1Work performed at the Penn State Milton S Hershey Medical Center. Journal of the American College of Surgeons 193, 6 (2001), 660 -- 665. https://doi.org/10.1016/S1072--7515(01)01066--3Google ScholarGoogle ScholarCross RefCross Ref
  16. Wenjing He, Simon Bryns, Karen Kroeker, Anup Basu, Daniel Birch, and Bin Zheng. 2020. Eye gaze of endoscopists during simulated colonoscopy. Journal of Robotic Surgery 14, 1 (2020), 137--143. https://doi.org/10.1007/s11701-019- 00950--1Google ScholarGoogle ScholarCross RefCross Ref
  17. Frouke Hermens, Rhona Flin, and Irfan Ahmed. 2013. Eye movements in surgery: A literature review. Journal of Eye Movement Research 6, 4 (2013), 1--11. https://doi.org/10.16910/jemr.6.4.4Google ScholarGoogle ScholarCross RefCross Ref
  18. Keita Higuch, Ryo Yonetani, and Yoichi Sato. 2016. Can Eye Help You? Effects of Visualizing Eye Fixations on Remote Collaboration Scenarios for Physical Tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 5180--5190. https://doi.org/10.1145/2858036.2858438 Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Minsik Hong and JerzyW. Rozenblit. 2015. Haptic Guidance with Fuzzy Control in Simulation-Based Surgical Training. In Proceedings of the Symposium on Modeling and Simulation in Medicine (Alexandria, Virginia) (MSM '15). Society for Computer Simulation International, San Diego, CA, USA, 25--30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Yue Yung Hu, Laura M. Mazer, Steven J. Yule, Alexander F. Arriaga, Caprice C. Greenberg, Stuart R. Lipsitz, Atul A. Gawande, and Douglas S. Smink. 2017. Complementing operating room teaching with video-based coaching. JAMA Surgery 152, 4 (1 April 2017), 318--325. https://doi.org/10.1001/jamasurg.2016.4619Google ScholarGoogle Scholar
  21. Ellen A. Isaacs and John C. Tang. 1993. What Video Can and Can't Do for Collaboration: A Case Study. In Proceedings of the First ACM International Conference on Multimedia (Anaheim, California, USA) (MULTIMEDIA '93). Association for Computing Machinery, New York, NY, USA, 199--206. https://doi.org/10.1145/166266.166289 Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Gazi Islam, Kanav Kahol, Baoxin Li, Marshall Smith, and Vimla L. Patel. 2016. Affordable, web-based surgical skill training and evaluation tool. Journal of Biomedical Informatics 59 (2016), 102 -- 114. https://doi.org/10.1016/j.jbi.2015. 11.002 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Pichamol Jirapinyo, Nitin Kumar, and Christopher C Thompson. 2015. Validation of an endoscopic part-task training box as a skill assessment tool. Gastrointestinal Endoscopy 81, 4 (apr 2015), 967--973. https://doi.org/10.1016/j.gie.2014.08.007Google ScholarGoogle ScholarCross RefCross Ref
  24. Shunichi Kasahara, Mitsuhito Ando, Kiyoshi Suganuma, and Jun Rekimoto. 2016. Parallel Eyes: Exploring Human Capability and Behaviors with Paralleled First Person View Sharing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 1561--1572. https://doi.org/10.1145/2858036.2858495 Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Moritz Kassner, William Patera, and Andreas Bulling. 2014. Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction. In Adjunct Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Seattle, Washington) (UbiComp '14 Adjunct). ACM, New York, NY, USA, 1151--1160. https://doi.org/10.1145/2638728.2641695 Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Rana S.A. Khan, Geoffrey Tien, M. Stella Atkins, Bin Zheng, Ormond N.M. Panton, and Adam T. Meneghetti. 2012. Analysis of eye gaze: Do novice surgeons look at the same location as expert surgeons during a laparoscopic operation? Surgical Endoscopy 26, 12 (2012), 3536--3540. https://doi.org/10.1007/s00464-012--2400--7Google ScholarGoogle ScholarCross RefCross Ref
  27. Navaneeth Kamballur Kottayil, Rositsa Bogdanova, Irene Cheng, Bin Zheng, and Anup Basu. 2016. Investigation of gaze patterns in multi view laparoscopic surgery. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS 2016-Octob (2016), 4031--4034. https://doi.org/10.1109/EMBC.2016.7591611Google ScholarGoogle ScholarCross RefCross Ref
  28. Andrey Iskrenov Kotzev, Dennis Yang, and Peter V. Draganov. 2019. How to master endoscopic submucosal dissection in the USA. Digestive Endoscopy 31, 1 (2019), 94--100. https://doi.org/10.1111/den.13240 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/den.13240Google ScholarGoogle ScholarCross RefCross Ref
  29. Uwe Kühnapfel, Hüseyin Çakmak, and Heiko Maaß. 2000. Endoscopic surgery training using virtual reality and deformable tissue simulation. Computers Graphics 24, 5 (2000), 671 -- 682. https://doi.org/10.1016/S0097--8493(00)00070- 4 Dynamic Medical Visualization (special topic).Google ScholarGoogle ScholarCross RefCross Ref
  30. Benjamin Law, Alan J. Lomax, M. Stella Atkins, Christine L. Mackenzie, and A. E. Kirkpatrick. 2004. Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment. Eye Tracking Research and Applications Symposium (ETRA) 1, 212 (2004), 41--47. https://doi.org/10.1145/968363.968370 Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Kai Lehmann, Joerg Ritz, Heiko Maass, Hüseyin Çakmak, Uwe Kühnapfel, Christoph Germer, Georg Bretthauer, and Heinz Buhr. 2005. A Prospective Randomized Study to Test the Transfer of Basic Psychomotor Skills From Virtual Reality to Physical Reality in a Comparable Training Setting. Annals of surgery 241 (04 2005), 442--9. https://doi.org/10.1097/01.sla.0000154552.89886.91Google ScholarGoogle ScholarCross RefCross Ref
  32. Yilin Elaine Liu, Michael Nitsche, and Benjamin N. Sugar. 2019. Reflection on Tacit Knowledge - Effect of Providing EMG Visualization on Reflections on Throwing Clay. In Proceedings of the 2019 on Creativity and Cognition (San Diego, CA, USA) (Camp;C '19). Association for Computing Machinery, New York, NY, USA, 619--625. https://doi.org/10.11453325480.3326576 Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Ganapathy Mani and Wei Li. 2013. 3D Web Based Surgical Training through Comparative Analysis. In Proceedings of the 18th International Conference on 3D Web Technology (San Sebastian, Spain) (Web3D '13). Association for Computing Machinery, New York, NY, USA, 83--86. https://doi.org/10.1145/2466533.2466553 Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Helena M. Mentis, Yuanyuan Feng, Azin Semsar, and Todd A. Ponsky. 2020. Remotely Shaping the View in Surgical Telementoring. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3313831.3376622 Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Jennifer A. Moon. 2004. A Handbook of Reflective and Experiential Learning: Theory and Practice (1st ed.). Routledge.Google ScholarGoogle Scholar
  36. Bernd Münzer, Klaus Schoeffmann, and Laszlo Böszörmenyi. 2018. Content-Based Processing and Analysis of Endoscopic Images and Videos: A Survey. Multimedia Tools Appl. 77, 1 (Jan. 2018), 1323--1362. https://doi.org/10.1007/ s11042-016--4219-z Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Lee J. Nelson, Paul Potrac, and Ryan Groom. 2014. Receiving video-based feedback in elite ice-hockey: a player's perspective. Sport, Education and Society 19, 1 (2014), 19--40. https://doi.org/10.1080/13573322.2011.613925 arXiv:https://doi.org/10.1080/13573322.2011.613925Google ScholarGoogle ScholarCross RefCross Ref
  38. Lucian Panait, Ehab Akkary, Robert Bell, Kurt Roberts, and Andrew Duffy. 2009. The Role of Haptic Feedback in Laparoscopic Simulation Training. The Journal of surgical research 156 (05 2009), 312--6. https://doi.org/10.1016/j.jss. 2009.04.018Google ScholarGoogle Scholar
  39. Rachel Prentice. 2007. Drilling Surgeons: The Social Lessons of Embodied Surgical Learning. Science Technology Human Values - SCI TECHNOL HUM VAL 32 (09 2007), 534--553. https://doi.org/10.1177/0895904805303201Google ScholarGoogle ScholarCross RefCross Ref
  40. Jessica Raque, Adam Goble, Veronica Jones, Lindsey Waldman, and Erica Sutton. 2015. The Relationship of Endoscopic Proficiency to Educational Expense for Virtual Reality Simulator Training Amongst Surgical Trainees. The American surgeon 81 (07 2015), 747--52. https://doi.org/10.1177/000313481508100727Google ScholarGoogle Scholar
  41. Jean-Philippe Rivière, Sarah Fdili Alaoui, Baptiste Caramiaux, and Wendy E. Mackay. 2019. Capturing Movement Decomposition to Support Learning and Teaching in Contemporary Dance. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 86 (Nov. 2019), 22 pages. https://doi.org/10.1145/3359188 Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. David A Rogers, Glenn Regehr, Thomas R Howdieshell, Karen A Yeh, and Ellen Palm. 2000. The impact of external feedback on computer-assisted learning for surgical technical skill training. The American Journal of Surgery 179, 4 (2000), 341 -- 343. https://doi.org/10.1016/S0002--9610(00)00341-XGoogle ScholarGoogle ScholarCross RefCross Ref
  43. David A Rogers, Glenn Regehr, Karen A Yeh, and Thomas R Howdieshell. 1998. Computer-assisted Learning versus a Lecture and Feedback Seminar for Teaching a Basic Surgical Technical Skill 11Presented at the 17th Annual Meeting of the Association for Surgical Education, Philadelphia, Pennsylvania, April 9--12, 1997. The American Journal of Surgery 175, 6 (1998), 508 -- 510. https://doi.org/10.1016/S0002--9610(98)00087--7Google ScholarGoogle ScholarCross RefCross Ref
  44. Daniela K. Rosner and Kimiko Ryokai. 2009. Reflections on Craft: Probing the Creative Process of Everyday Knitters. In Proceedings of the Seventh ACM Conference on Creativity and Cognition (Berkeley, California, USA) (Camp;C '09). Association for Computing Machinery, New York, NY, USA, 195--204. https://doi.org/10.1145/1640233.1640264 Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. M Schijven and J Jakimowicz. 2003. Virtual reality surgical laparoscopic simulators. Surgical Endoscopy And Other Interventional Techniques 17, 12 (2003), 1943--1950. https://doi.org/10.1007/s00464-003--9052--6Google ScholarGoogle ScholarCross RefCross Ref
  46. Marlies Schijven and Jack Jakimowicz. 2004. Virtual reality surgical laparoscopic simulators - How to choose. Surgical endoscopy 17 (01 2004), 1943--50. https://doi.org/10.1007/s00464-003--9052--6Google ScholarGoogle Scholar
  47. Yasuyuki Sumi, Masaki Suwa, and Koichi Hanaue. 2018. Effects of Viewing Multiple Viewpoint Videos on Metacognition of Collaborative Experiences. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/ 10.1145/3173574.3174222 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. John C Tang and Ellen Isaacs. 1992. Why do users like video? Computer Supported Cooperative Work (CSCW) 1, 3 (1992), 163--196. https://doi.org/10.1007/BF00752437Google ScholarGoogle ScholarCross RefCross Ref
  49. Balasaravanan Thoravi Kumaravel, Cuong Nguyen, Stephen DiVerdi, and Björn Hartmann. 2019. TutoriVR: A Video- Based Tutorial System for Design Applications in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300514 Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Barry D. Wilson. 2008. Development in video technology for coaching. Sports Technology 1, 1 (2008), 34--40. https://doi.org/10.1002/jst.9 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/jst.9Google ScholarGoogle ScholarCross RefCross Ref
  51. Bin Zheng, Xianta Jiang, Roman Bednarik, and M. Stell Atkins. 2017. Gaze characteristics of video watching in a surgical setting. Proceedings of the 2nd Workshop on Eye Tracking and Visualization, ETVIS 2016 October 2016 (2017), 11--15. https://doi.org/10.1109/ETVIS.2016.7851158Google ScholarGoogle ScholarCross RefCross Ref
  52. M. Zhou, S. Tse, A. DereviankoA, D.B. Jones, Steven Schwaitzberg, and Caroline Cao. 2013. Effect of Haptic Feedback in Laparoscopic Surgery Skill Acquisition. Virtual Technologies in Medicine (01 2013), 33--38. https://doi.org/10.46594/2687-0037_2013_1_33Google ScholarGoogle Scholar

Index Terms

  1. A Surgical Scene Replay System for Learning Gastroenterological Endoscopic Surgery Skill by Multiple Synchronized-Video and Gaze Representation

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!