Abstract
Gastroenterological endoscopic surgery needs complex surgical skills such as a sensation of body movement and manipulation of the endoscope that is hard to be explained verbally. Prior research reported that endoscopic surgery is one of the most challenging surgery to teach. Thus, surgeons need long-term practice to master the skills. To support learning such skills, we developed a surgical scene replay system. First, we surveyed 12 surgeons to reveal the reason for the difficulty and elicited three requirements for our system: (1) provide multiple videos that include an endoscope, a fluoroscopy, and hand manipulation for the endoscope to observe the surgery from multiple aspects; (2) visualize an experts' gaze position to understand experts' intention of hand manipulation, and (3) enlarge the size of the gazed video to inform learners where they should pay attention to. Our user study with the system indicates that participants could understand the experts' intentions and tacit knowledge easier than the existing video materials.
- Deepak Akkil and Poika Isokoski. 2016. Gaze Augmentation in Egocentric Video Improves Awareness of Intention. Association for Computing Machinery, New York, NY, USA, 1573--1584. https://doi.org/10.1145/2858036.2858127 Google Scholar
Digital Library
- Knut Augestad, Khayam Butt, Dejan Ignjatovic, Deborah Keller, and Ravi Kiran. 2020. Video-based coaching in surgical education: a systematic review and meta-analysis. Surgical Endoscopy 34 (02 2020). https://doi.org/10.1007/s00464- 019-07265-0Google Scholar
- Timothy J. Babineau, James Becker, Gary Gibbons, Stephen Sentovich, Donald Hess, Sharon Robertson, and Michael Stone. 2004. The "Cost" of Operative Training for Surgical Residents. Archives of Surgery 139, 4 (04 2004), 366--370. https://doi.org/10.1001/archsurg.139.4.366 arXiv:https://jamanetwork.com/journals/jamasurgery/articlepdf/396495/sne3006.pdfGoogle Scholar
- Ido Badash, Karen Burtt, Carlos A Solorzano, and Joseph N Carey. 2016. Innovations in surgery simulation: a review of past, current and future techniques. Annals of translational medicine 4, 23 (dec 2016), 453. https://doi.org/10.21037/ atm.2016.12.24Google Scholar
Cross Ref
- Guillermo Bernal, Dishaan Ahuja, and Federico Casalegno. 2015. EMG-Based Biofeedback Tool for Augmenting Manual Fabrication and Improved Exchange of Empirical Knowledge. In Proceedings of the XVI International Conference on Human Computer Interaction (Vilanova i la Geltru, Spain) (Interacción '15). Association for Computing Machinery, New York, NY, USA, Article 61, 8 pages. https://doi.org/10.1145/2829875.2829932 Google Scholar
Digital Library
- Matthew Bridges and Daniel L Diamond. 1999. The financial impact of teaching surgical residents in the operating room. The American Journal of Surgery 177, 1 (1999), 28 -- 32. https://doi.org/10.1016/S0002--9610(98)00289-XGoogle Scholar
Cross Ref
- Diogo Cabral, João G. Valente, Urândia Aragão, Carla Fernandes, and Nuno Correia. 2012. Evaluation of a Multimodal Video Annotator for Contemporary Dance. In Proceedings of the International Working Conference on Advanced Visual Interfaces (Capri Island, Italy) (AVI '12). Association for Computing Machinery, New York, NY, USA, 572--579. https://doi.org/10.1145/2254556.2254663 Google Scholar
Digital Library
- Lin Chen, Peng Zhang, and Baoxin Li. 2014. Instructive Video Retrieval Based on Hybrid Ranking and Attribute Learning: A Case Study on Surgical Skill Training. In Proceedings of the 22nd ACM International Conference on Multimedia (Orlando, Florida, USA) (MM '14). Association for Computing Machinery, New York, NY, USA, 1045--1048. https://doi.org/10.1145/2647868.2655050 Google Scholar
Digital Library
- Pei-Yu Chi, Joyce Liu, Jason Linder, Mira Dontcheva, Wilmot Li, and Bjoern Hartmann. 2013. DemoCut: Generating Concise Instructional Videos for Physical Demonstrations. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST '13). Association for Computing Machinery, New York, NY, USA, 141--150. https://doi.org/10.1145/2501988.2502052 Google Scholar
Digital Library
- Chun-Hong Huang, T. L. Won, Chun-Yeh Liu, Yen-Da Chen, and Yung-Hui Chen. 2009. Multiple-video-based Elearning Platform for Physical Education. In 2009 Joint Conferences on Pervasive Computing (JCPC). 21--26. https: //doi.org/10.1109/JCPC.2009.5420220Google Scholar
- Anender Dhariwal, Ram Prabhu, Abhay Dalvi, and Avinash Supe. 2007. Effectiveness of box trainers in laparoscopic training. Journal of minimal access surgery 3 (04 2007), 57--63. https://doi.org/10.4103/0972--9941.33274Google Scholar
- Diana L. Diesen, Loretta Erhunmwunsee, Kyla M. Bennett, Kfir Ben-David, Basil Yurcisin, Eugene P. Ceppa, Philip A. Omotosho, Alexander Perez, and Aurora Pryor. 2011. Effectiveness of Laparoscopic Computer Simulator Versus Usage of Box Trainer for Endoscopic Surgery Training of Novices. Journal of Surgical Education 68, 4 (2011), 282 -- 289. https://doi.org/10.1016/j.jsurg.2011.02.007Google Scholar
Cross Ref
- Adam J. Goodman, Joshua Melson, Harry R. Aslanian, Manoop S. Bhutani, Kumar Krishnan, David R. Lichtenstein, Udayakumar Navaneethan, Rahul Pannala, Mansour A. Parsi, Allison R. Schulman, Amrita Sethi, Shelby A. Sullivan, Nirav Thosani, Guru Trikudanathan, Arvind J. Trindade, Rabindra R. Watson, and John T. Maple. 2019. Endoscopic simulators. Gastrointestinal Endoscopy 90, 1 (2019), 1 -- 12. https://doi.org/10.1016/j.gie.2018.10.037Google Scholar
Cross Ref
- Kristine Hagelsteen, Richard Johansson, Mikael Ekelund, Anders Bergenfelz, and Magnus Anderberg. 2019. Performance and perception of haptic feedback in a laparoscopic 3D virtual reality simulator. Minimally Invasive Therapy & Allied Technologies 28, 5 (2019), 309--316. https://doi.org/10.1080/13645706.2018.1539012 arXiv:https://doi.org/10.1080/13645706.2018.1539012 PMID: 30663462.Google Scholar
Cross Ref
- Randy S. Haluck, Renee L. Marshall, Thomas M. Krummel, and Michael G. Melkonian. 2001. Are surgery training programs ready for virtual reality? a survey of program directors in general surgery1 1Work performed at the Penn State Milton S Hershey Medical Center. Journal of the American College of Surgeons 193, 6 (2001), 660 -- 665. https://doi.org/10.1016/S1072--7515(01)01066--3Google Scholar
Cross Ref
- Wenjing He, Simon Bryns, Karen Kroeker, Anup Basu, Daniel Birch, and Bin Zheng. 2020. Eye gaze of endoscopists during simulated colonoscopy. Journal of Robotic Surgery 14, 1 (2020), 137--143. https://doi.org/10.1007/s11701-019- 00950--1Google Scholar
Cross Ref
- Frouke Hermens, Rhona Flin, and Irfan Ahmed. 2013. Eye movements in surgery: A literature review. Journal of Eye Movement Research 6, 4 (2013), 1--11. https://doi.org/10.16910/jemr.6.4.4Google Scholar
Cross Ref
- Keita Higuch, Ryo Yonetani, and Yoichi Sato. 2016. Can Eye Help You? Effects of Visualizing Eye Fixations on Remote Collaboration Scenarios for Physical Tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 5180--5190. https://doi.org/10.1145/2858036.2858438 Google Scholar
Digital Library
- Minsik Hong and JerzyW. Rozenblit. 2015. Haptic Guidance with Fuzzy Control in Simulation-Based Surgical Training. In Proceedings of the Symposium on Modeling and Simulation in Medicine (Alexandria, Virginia) (MSM '15). Society for Computer Simulation International, San Diego, CA, USA, 25--30. Google Scholar
Digital Library
- Yue Yung Hu, Laura M. Mazer, Steven J. Yule, Alexander F. Arriaga, Caprice C. Greenberg, Stuart R. Lipsitz, Atul A. Gawande, and Douglas S. Smink. 2017. Complementing operating room teaching with video-based coaching. JAMA Surgery 152, 4 (1 April 2017), 318--325. https://doi.org/10.1001/jamasurg.2016.4619Google Scholar
- Ellen A. Isaacs and John C. Tang. 1993. What Video Can and Can't Do for Collaboration: A Case Study. In Proceedings of the First ACM International Conference on Multimedia (Anaheim, California, USA) (MULTIMEDIA '93). Association for Computing Machinery, New York, NY, USA, 199--206. https://doi.org/10.1145/166266.166289 Google Scholar
Digital Library
- Gazi Islam, Kanav Kahol, Baoxin Li, Marshall Smith, and Vimla L. Patel. 2016. Affordable, web-based surgical skill training and evaluation tool. Journal of Biomedical Informatics 59 (2016), 102 -- 114. https://doi.org/10.1016/j.jbi.2015. 11.002 Google Scholar
Digital Library
- Pichamol Jirapinyo, Nitin Kumar, and Christopher C Thompson. 2015. Validation of an endoscopic part-task training box as a skill assessment tool. Gastrointestinal Endoscopy 81, 4 (apr 2015), 967--973. https://doi.org/10.1016/j.gie.2014.08.007Google Scholar
Cross Ref
- Shunichi Kasahara, Mitsuhito Ando, Kiyoshi Suganuma, and Jun Rekimoto. 2016. Parallel Eyes: Exploring Human Capability and Behaviors with Paralleled First Person View Sharing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 1561--1572. https://doi.org/10.1145/2858036.2858495 Google Scholar
Digital Library
- Moritz Kassner, William Patera, and Andreas Bulling. 2014. Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction. In Adjunct Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Seattle, Washington) (UbiComp '14 Adjunct). ACM, New York, NY, USA, 1151--1160. https://doi.org/10.1145/2638728.2641695 Google Scholar
Digital Library
- Rana S.A. Khan, Geoffrey Tien, M. Stella Atkins, Bin Zheng, Ormond N.M. Panton, and Adam T. Meneghetti. 2012. Analysis of eye gaze: Do novice surgeons look at the same location as expert surgeons during a laparoscopic operation? Surgical Endoscopy 26, 12 (2012), 3536--3540. https://doi.org/10.1007/s00464-012--2400--7Google Scholar
Cross Ref
- Navaneeth Kamballur Kottayil, Rositsa Bogdanova, Irene Cheng, Bin Zheng, and Anup Basu. 2016. Investigation of gaze patterns in multi view laparoscopic surgery. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS 2016-Octob (2016), 4031--4034. https://doi.org/10.1109/EMBC.2016.7591611Google Scholar
Cross Ref
- Andrey Iskrenov Kotzev, Dennis Yang, and Peter V. Draganov. 2019. How to master endoscopic submucosal dissection in the USA. Digestive Endoscopy 31, 1 (2019), 94--100. https://doi.org/10.1111/den.13240 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/den.13240Google Scholar
Cross Ref
- Uwe Kühnapfel, Hüseyin Çakmak, and Heiko Maaß. 2000. Endoscopic surgery training using virtual reality and deformable tissue simulation. Computers Graphics 24, 5 (2000), 671 -- 682. https://doi.org/10.1016/S0097--8493(00)00070- 4 Dynamic Medical Visualization (special topic).Google Scholar
Cross Ref
- Benjamin Law, Alan J. Lomax, M. Stella Atkins, Christine L. Mackenzie, and A. E. Kirkpatrick. 2004. Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment. Eye Tracking Research and Applications Symposium (ETRA) 1, 212 (2004), 41--47. https://doi.org/10.1145/968363.968370 Google Scholar
Digital Library
- Kai Lehmann, Joerg Ritz, Heiko Maass, Hüseyin Çakmak, Uwe Kühnapfel, Christoph Germer, Georg Bretthauer, and Heinz Buhr. 2005. A Prospective Randomized Study to Test the Transfer of Basic Psychomotor Skills From Virtual Reality to Physical Reality in a Comparable Training Setting. Annals of surgery 241 (04 2005), 442--9. https://doi.org/10.1097/01.sla.0000154552.89886.91Google Scholar
Cross Ref
- Yilin Elaine Liu, Michael Nitsche, and Benjamin N. Sugar. 2019. Reflection on Tacit Knowledge - Effect of Providing EMG Visualization on Reflections on Throwing Clay. In Proceedings of the 2019 on Creativity and Cognition (San Diego, CA, USA) (Camp;C '19). Association for Computing Machinery, New York, NY, USA, 619--625. https://doi.org/10.11453325480.3326576 Google Scholar
Digital Library
- Ganapathy Mani and Wei Li. 2013. 3D Web Based Surgical Training through Comparative Analysis. In Proceedings of the 18th International Conference on 3D Web Technology (San Sebastian, Spain) (Web3D '13). Association for Computing Machinery, New York, NY, USA, 83--86. https://doi.org/10.1145/2466533.2466553 Google Scholar
Digital Library
- Helena M. Mentis, Yuanyuan Feng, Azin Semsar, and Todd A. Ponsky. 2020. Remotely Shaping the View in Surgical Telementoring. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3313831.3376622 Google Scholar
Digital Library
- Jennifer A. Moon. 2004. A Handbook of Reflective and Experiential Learning: Theory and Practice (1st ed.). Routledge.Google Scholar
- Bernd Münzer, Klaus Schoeffmann, and Laszlo Böszörmenyi. 2018. Content-Based Processing and Analysis of Endoscopic Images and Videos: A Survey. Multimedia Tools Appl. 77, 1 (Jan. 2018), 1323--1362. https://doi.org/10.1007/ s11042-016--4219-z Google Scholar
Digital Library
- Lee J. Nelson, Paul Potrac, and Ryan Groom. 2014. Receiving video-based feedback in elite ice-hockey: a player's perspective. Sport, Education and Society 19, 1 (2014), 19--40. https://doi.org/10.1080/13573322.2011.613925 arXiv:https://doi.org/10.1080/13573322.2011.613925Google Scholar
Cross Ref
- Lucian Panait, Ehab Akkary, Robert Bell, Kurt Roberts, and Andrew Duffy. 2009. The Role of Haptic Feedback in Laparoscopic Simulation Training. The Journal of surgical research 156 (05 2009), 312--6. https://doi.org/10.1016/j.jss. 2009.04.018Google Scholar
- Rachel Prentice. 2007. Drilling Surgeons: The Social Lessons of Embodied Surgical Learning. Science Technology Human Values - SCI TECHNOL HUM VAL 32 (09 2007), 534--553. https://doi.org/10.1177/0895904805303201Google Scholar
Cross Ref
- Jessica Raque, Adam Goble, Veronica Jones, Lindsey Waldman, and Erica Sutton. 2015. The Relationship of Endoscopic Proficiency to Educational Expense for Virtual Reality Simulator Training Amongst Surgical Trainees. The American surgeon 81 (07 2015), 747--52. https://doi.org/10.1177/000313481508100727Google Scholar
- Jean-Philippe Rivière, Sarah Fdili Alaoui, Baptiste Caramiaux, and Wendy E. Mackay. 2019. Capturing Movement Decomposition to Support Learning and Teaching in Contemporary Dance. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 86 (Nov. 2019), 22 pages. https://doi.org/10.1145/3359188 Google Scholar
Digital Library
- David A Rogers, Glenn Regehr, Thomas R Howdieshell, Karen A Yeh, and Ellen Palm. 2000. The impact of external feedback on computer-assisted learning for surgical technical skill training. The American Journal of Surgery 179, 4 (2000), 341 -- 343. https://doi.org/10.1016/S0002--9610(00)00341-XGoogle Scholar
Cross Ref
- David A Rogers, Glenn Regehr, Karen A Yeh, and Thomas R Howdieshell. 1998. Computer-assisted Learning versus a Lecture and Feedback Seminar for Teaching a Basic Surgical Technical Skill 11Presented at the 17th Annual Meeting of the Association for Surgical Education, Philadelphia, Pennsylvania, April 9--12, 1997. The American Journal of Surgery 175, 6 (1998), 508 -- 510. https://doi.org/10.1016/S0002--9610(98)00087--7Google Scholar
Cross Ref
- Daniela K. Rosner and Kimiko Ryokai. 2009. Reflections on Craft: Probing the Creative Process of Everyday Knitters. In Proceedings of the Seventh ACM Conference on Creativity and Cognition (Berkeley, California, USA) (Camp;C '09). Association for Computing Machinery, New York, NY, USA, 195--204. https://doi.org/10.1145/1640233.1640264 Google Scholar
Digital Library
- M Schijven and J Jakimowicz. 2003. Virtual reality surgical laparoscopic simulators. Surgical Endoscopy And Other Interventional Techniques 17, 12 (2003), 1943--1950. https://doi.org/10.1007/s00464-003--9052--6Google Scholar
Cross Ref
- Marlies Schijven and Jack Jakimowicz. 2004. Virtual reality surgical laparoscopic simulators - How to choose. Surgical endoscopy 17 (01 2004), 1943--50. https://doi.org/10.1007/s00464-003--9052--6Google Scholar
- Yasuyuki Sumi, Masaki Suwa, and Koichi Hanaue. 2018. Effects of Viewing Multiple Viewpoint Videos on Metacognition of Collaborative Experiences. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/ 10.1145/3173574.3174222 Google Scholar
Digital Library
- John C Tang and Ellen Isaacs. 1992. Why do users like video? Computer Supported Cooperative Work (CSCW) 1, 3 (1992), 163--196. https://doi.org/10.1007/BF00752437Google Scholar
Cross Ref
- Balasaravanan Thoravi Kumaravel, Cuong Nguyen, Stephen DiVerdi, and Björn Hartmann. 2019. TutoriVR: A Video- Based Tutorial System for Design Applications in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300514 Google Scholar
Digital Library
- Barry D. Wilson. 2008. Development in video technology for coaching. Sports Technology 1, 1 (2008), 34--40. https://doi.org/10.1002/jst.9 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/jst.9Google Scholar
Cross Ref
- Bin Zheng, Xianta Jiang, Roman Bednarik, and M. Stell Atkins. 2017. Gaze characteristics of video watching in a surgical setting. Proceedings of the 2nd Workshop on Eye Tracking and Visualization, ETVIS 2016 October 2016 (2017), 11--15. https://doi.org/10.1109/ETVIS.2016.7851158Google Scholar
Cross Ref
- M. Zhou, S. Tse, A. DereviankoA, D.B. Jones, Steven Schwaitzberg, and Caroline Cao. 2013. Effect of Haptic Feedback in Laparoscopic Surgery Skill Acquisition. Virtual Technologies in Medicine (01 2013), 33--38. https://doi.org/10.46594/2687-0037_2013_1_33Google Scholar
Index Terms
A Surgical Scene Replay System for Learning Gastroenterological Endoscopic Surgery Skill by Multiple Synchronized-Video and Gaze Representation
Recommendations
Advanced Virtual Endoscopic Pituitary Surgery
Endoscopy has recently been introduced to endonasal transsphenoidal pituitary surgery as a minimally invasive procedure for the removal of various kinds of pituitary tumors. To reduce morbidity and mortality with this new technique, the surgeon must be ...
Tracking of instruments in minimally invasive surgery for surgical skill analysis
Miar'06: Proceedings of the Third international conference on Medical Imaging and Augmented RealityIntraoperative assistance systems aim to improve the quality of the surgery and enhance the surgeon’s capabilities. Preferable would be a system which provides support depending on the surgery context and surgical skills accomplished. Therefore, the ...
Sensing and manipulation problems in endoscopic surgery: Experiment, analysis, and observation
The minimally invasive nature of endoscopic surgery allows operations to be performed through small incisions, producing significantly less damage to good tissue than in conventional open surgery. Patient recovery times are thus greatly reduced. This ...






Comments