skip to main content
research-article

Creating and Evaluating Chatbots as Eligibility Assistants for Clinical Trials: An Active Deep Learning Approach towards User-centered Classification

Authors Info & Claims
Published:30 December 2020Publication History
Skip Abstract Section

Abstract

Clinical trials are important tools to improve knowledge about the effectiveness of new treatments for all diseases, including cancers. However, studies show that fewer than 5% of cancer patients are enrolled in any type of research study or clinical trial. Although there is a wide variety of reasons for the low participation rate, we address this issue by designing a chatbot to help users determine their eligibility via interactive, two-way communication. The chatbot is supported by a user-centered classifier that uses an active deep learning approach to separate complex eligibility criteria into questions that can be easily answered by users and information that requires verification by their doctors. We collected all the available clinical trial eligibility criteria from the National Cancer Institute's website to evaluate the chatbot and the classifier. Experimental results show that the active deep learning classifier outperforms the baseline k-nearest neighbor method. In addition, an in-person experiment was conducted to evaluate the effectiveness of the chatbot. The results indicate that the participants who used the chatbot achieved better understanding about eligibility than those who used only the website. Furthermore, interfaces with chatbots were rated significantly better in terms of perceived usability, interactivity, and dialogue.

References

  1. M. Amith, Z. H. U. Anna, R. Cunningham, L. I. N. Rebecca, L. Savas, S. H. A. Y. Laura, and T. A. O. Cui. 2019. Early usability assessment of a conversational agent for HPV vaccination. Stud. Health Technol. Inform. 257 (2019), 17--23.Google ScholarGoogle Scholar
  2. R. F. Azevedo, D. Morrow, J. Graumlich, A. Willemsen-Dunlap, M. Hasegawa-Johnson, T. S. Huang, and D. J. Halpin. 2018. Using conversational agents to explain medication instructions to older adults. In Proceedings of the AMIA Annual Symposium. 185. American Medical Informatics Association.Google ScholarGoogle Scholar
  3. C. S. Bennette, S. D. Ramsey, C. L. McDermott, J. J. Carlson, A. Basu, and D. L. Veenstra. 2016. Predicting low accrual in the national cancer institute's cooperative group clinical trials. J. Nat. Cancer Inst. 108, 2 (2016).Google ScholarGoogle Scholar
  4. T. Bickmore, H. Trinh, R. Asadi, and S. Olafsson. 2018. Safety first: Conversational agents for health care. In Studies in Conversational UX Design, 33--57. Springer, Cham.Google ScholarGoogle Scholar
  5. P. P. Breitfeld, M. Weisburd, J. M. Overhage, G. Sledge Jr, and W. M. Tierney. 1999. Pilot study of a point-of-use decision support tool for cancer clinical trials eligibility. J. Amer. Med. Inf. Assoc. 6, 6 (1999), 466--477.Google ScholarGoogle Scholar
  6. R. W. Carlson, S. W. Tu, N. M. Lane, T. L. Lai, C. A. Kemper, M. A. Musen, and E. H. Shortliffe. 1995. Computer-based screening of patients with HIV/AIDS for clinical-trial eligibility. Online J. Curr. Clin. Trials 4, 179 (1995).Google ScholarGoogle Scholar
  7. C. H. Chuan. 2018. Classifying eligibility criteria in clinical trials using active deep learning. In Proceedings of the 17th IEEE International Conference on Machine Learning and Applications. 305--310.Google ScholarGoogle Scholar
  8. M. Cuggia, P. Besana, and D. Glasspool. 2011. Comparing semi-automatic systems for recruitment of patients to clinical trials. Int. J. Med. Inf. 80, 6 (2011), 371--388.Google ScholarGoogle ScholarCross RefCross Ref
  9. E. Fink, L. O. Hall, D. B. Goldgof, B. D. Goswami, M. Boonstra, and J. P. Krischer. 2003. Experiments on the automated selection of patients for clinical trials. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. 4541--4545.Google ScholarGoogle Scholar
  10. E. Fink, P. K. Kokku, S. Nikiforou, L. O. Hall, D. B. Goldgof, and J. P. Krischer. 2004. Selection of patients for clinical trials: An interactive web-based system. Artif. Intell. Med. 31, 241--254.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. A. Ho, J. Hancock, and A. S. Miner. 2018. Psychological, relational, and emotional effects of self-disclosure after conversations with a chatbot. J. Commun. 68, 4 (2018), 712--733.Google ScholarGoogle ScholarCross RefCross Ref
  12. F. Köpcke, D. Lubgan, R. Fietkau, A. Scholler, C. Nau, M. Stürzl, and D. Toddenroth. 2013. Evaluating predictive modeling algorithms to assess patient eligibility for clinical trials from routine data. BMC Med. Inf. Dec. Making 13, 1 (2013), 134.Google ScholarGoogle Scholar
  13. M. Kusner, Y. Sun, N. Kolkin, and K. Weinberger. 2015. From word embeddings to document distances. In Proceedings of the International Conference on Machine Learning. 957--966.Google ScholarGoogle Scholar
  14. L. Laranjo, A. G. Dunn, H. L. Tong, A. B. Kocaballi, J. Chen, R. Bashir, D. Surian, B. Gallego, F. Magrabi, A. Y. Lau, and E. Coiera. 2018. Conversational agents in healthcare: A systematic review. J. Amer. Med. Inf. Assoc. 25, 9 (2018), 1248--1258.Google ScholarGoogle ScholarCross RefCross Ref
  15. J. Lilleberg, Y. Zhu, and Y. Zhang. 2015. Support vector machines and word2vec for text classification with semantic features. In Proceedings of the IEEE 14th International Conference on Cognitive Informatics and Cognitive Computing. 136--140.Google ScholarGoogle Scholar
  16. Z. Luo, R. Duffy, S. Johnson, and C. Weng. 2010. Corpus-based approach to creating a semantic lexicon for clinical research eligibility criteria from UMLS. In Proceedings of the AMIA Summit on Clinical Research Informatics. 26--31.Google ScholarGoogle Scholar
  17. Z. Luo, M. Yetisgen-Yildiz, and C. Weng. 2011. Dynamic categorization of clinical research eligibility criteria by hierarchical clustering. J. Biomed. Inf. 44, 6 (2011), 927--935.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. L. V. D. Maaten and G. Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9 (2008), 2579--2605.Google ScholarGoogle Scholar
  19. J. M. Metz, C. Coyle, C. Hudson, and M. Hampshire. 2005. An internet-based cancer clinical trials matching resource. J. Med. Internet Res. 7, 3 (2005), e24.Google ScholarGoogle Scholar
  20. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. 2013. Distributed representations of words and phrases and their compositionality. Adv. Neural Inf. Proc. Syst. 3111--3119.Google ScholarGoogle Scholar
  21. K. Milian, A. Bucur, and A. Ten Teije. 2012. Formalization of clinical trial eligibility criteria: Evaluation of a pattern-based approach. In Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine. 1--4.Google ScholarGoogle Scholar
  22. K. Milian, A. Ten Teije, A. Bucur, and F. Van Harmelen. 2011. Patterns of clinical trial eligibility criteria. In Proceedings of the International Workshop on Knowledge Representation for Health Care. Springer, Berlin, 145--157.Google ScholarGoogle Scholar
  23. R. Miotto and C. Weng. 2015. Case-based reasoning using electronic health records efficiently identifies eligible patients for clinical trials. J. Amer. Med. Inf. Assoc. 22, e1 (2015), 141--150.Google ScholarGoogle ScholarCross RefCross Ref
  24. J. L. Z. Montenegro, C. A. da Costa, and R. da Rosa Righi. 2019. Survey of conversational agents in health. Exp. Syst. Applic. 129 (2019), 56--67.Google ScholarGoogle ScholarCross RefCross Ref
  25. V. H. Murthy, H. M. Krumbolz, and C. P. Gross. 2004. Participation in cancer clinical trials: Race-, sex-, and age-based disparities. J. Amer. Med. Assoc. 291 (2004), 2720--2726.Google ScholarGoogle Scholar
  26. J. Niland, D. Dorr, G. El Saadawi, P. Embi, R. L. Richesson et al. 2007. Knowledge representation of eligibility criteria in clinical trials. In Proceedings of the American Medical Informatics Association Annual Symposium.Google ScholarGoogle Scholar
  27. M. Peleg, S. Tu, J. Bury, P. Ciccarese, J. Fox et al. 2003. Comparing computer-interpretable guideline models: A case-study approach. J. Amer. Med. Inf. Assoc. 10, (2003) 52--68.Google ScholarGoogle Scholar
  28. D. Rubin, J. Gennari, S. Srinivas, A. Yuen, H. Kaizer, M. Musen, et al. 1999. Tool support for authoring eligibility criteria for cancer trials. In Proceedings of the AMIA Symposium. 369--373.Google ScholarGoogle Scholar
  29. I. Sim, B. Olasov, and S. Carini. 2004. An ontology of randomized controlled trials for evidence-based practice: Content specification and evaluation using the competency decomposition method. J. Biomed. Inf. 37 (2004), 108--119.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. D. Tang, F. Wei, N. Yang, M. Zhou, T. Liu, and B. Qin. 2014. Learning sentiment-specific word embedding for Twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics 1 (2014), 1555--1565.Google ScholarGoogle Scholar
  31. S. W. Tu, M. Peleg, S. Carini, M. Bobak, J. Ross, D. Rubin, and I. Sim. 2011. A practical method for transforming free-text eligibility criteria into computable criteria. J. Biomed. Inf. 44, 2 (2011), 239--250.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. D. Utami, B. Barry, T. Bickmore, and M. Paasche-Orlow. 2013. A conversational agent-based clinical trial search engine. In Proceedings of the Annual Symposium on Human-Computer Interaction and Information Retrieval (HCIR’13).Google ScholarGoogle Scholar
  33. R. Wallace. 2003. The Elements of AIML Style. Alice AI Foundation.Google ScholarGoogle Scholar
  34. Z. Zhang, T. W. Bickmore, and M. K. Paasche-Orlow. 2017. Perceived organizational affiliation and its effects on patient trust: Role modeling with embodied conversational agents. Patient Educ. Counsel. 100, 9 (2017), 1730--1737.Google ScholarGoogle ScholarCross RefCross Ref
  35. S. Zhang, F. Liang, W. Li, and I. Tannock. 2016. Comparison of eligibility criteria between protocols, registries, and publications of cancer clinical trials. JNCI: J. Nat. Cancer Ins. 108 (2016), 11.Google ScholarGoogle Scholar

Index Terms

  1. Creating and Evaluating Chatbots as Eligibility Assistants for Clinical Trials: An Active Deep Learning Approach towards User-centered Classification

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Computing for Healthcare
          ACM Transactions on Computing for Healthcare  Volume 2, Issue 1
          Special Issue on Wearable Technologies for Smart Health: Part 2 and Regular Papers
          January 2021
          204 pages
          ISSN:2691-1957
          EISSN:2637-8051
          DOI:10.1145/3446563
          Issue’s Table of Contents

          Copyright © 2020 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 30 December 2020
          • Accepted: 1 May 2020
          • Revised: 1 March 2020
          • Received: 1 November 2019
          Published in health Volume 2, Issue 1

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!