Measuring the Performance of a Location-Aware Text Prediction System

Abstract

In recent years, some works have discussed the conception of location-aware Augmentative and Alternative Communication (AAC) systems with very positive feedback from participants. However, in most cases, complementary quantitative evaluations have not been carried out to confirm those results. To contribute to clarifying the validity of these approaches, our study quantitatively evaluated the effect of using language models with location knowledge on the efficiency of a word and sentence prediction system. Using corpora collected for three different locations (classroom, school cafeteria, home), location-specific language models were trained with sentences from each location and compared with a traditional all-purpose language model, trained on all corpora. User tests showed a modest mean improvement of 2.4% and 1.3% for Words Per Minute (WPM) and Keystroke Saving Rate (KSR), respectively, but the differences were not statistically significant. Since our text prediction system relies on the concept of sentence reuse, we ran a set of simulations with language models having different sentence knowledge levels (0%, 25%, 50%, 75%, 100%). We also introduced in the comparison a second location-aware strategy that combines the location-specific approach with the all-purpose approach (mixed approach). The mixed language models performed better under low sentence-reuse conditions (0%, 25%, 50%) with 1.0%, 1.3%, and 1.2% KSR improvements, respectively. The location-specific language models performed better under high sentence-reuse conditions (75%, 100%) with 1.7% and 1.5% KSR improvements, respectively.

References

1. L. Adams, L. Hunt, and M. Moore. 2003. The aware system: Prototyping an augmentative communication interface. In Proceedings of the Rehabilitation Engineering Society of North America (RESNA’03).
2. A. Dix, J. Finlay, G. Abowd, and R. Beale. 2004. Human-Computer Interaction. Pearson Education Limited, Harlow, England.
3. N. Alm and J. L. Arnott. 1998. Computer-assisted conversation for nonvocal people using prestored texts. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 28, 3, 318--328.
4. N. Alm, J. Todman, L. Elder, and A. F. Newell. 1993. Computer aided conversation for severely physically impaired non-speaking people. In Proceedings of the INTERACT‘93 and CHI’93 Conference on Human Factors in Computing Systems. ACM, 236--241.
5. J. L. Arnott, A. F. Newell, and N. Alm. 1992. Prediction and conversational momentum in an augmentative communication system. Communications of the ACM 35, 5, 46--57.
6. A. Cook and S. Hussey. 1995. Assistive Technologies: Principles and Practice. Mosby Year Book, St. Louis, MO.
7. J. J. Darragh, I. H. Witten, and M. L. James. 1990. The reactive keyboard: A predictive typing aid. Computer 23, 11, 41--49.
8. A. Davis, M. Moore, and V. Storey. 2003. Context-aware communication for severely disabled users. In Proceedings of the 2003 Conference on Universal Usability. ACM, New York, NY, 106--111.
9. P. W. Demasco and K. F. McCoy. 1992. Generating text from compressed input: An intelligent interface for people with severe motor impairments. Communications of the ACM 35, 5, 68--78.
10. C. Demmans Epp, R. Campigotto, A. Levy, and R. Baecker. 2011. MarcoPolo: Context-sensitive mobile communication support. In Proceedings of the Rehabilitation Engineering and Assistive Technology Society of North America Annual Conference and the 3rd International Conference on Technology and Aging.
11. E. Dominowska. 2002. A Communication Aid with Context-Aware Vocabulary Prediction. Master Thesis, Massachusetts Institute of Technology, Cambridge, MA.
12. L. Garcia, L. de Oliveira, and D. de Matos. 2013. Word and sentence prediction: Using the best of the two worlds to assist AAC users. Assistive Technology: From Research to Practice: AAATE 2013, 33, 326.
13. L. Garcia and L. de Oliveira. 2005. Description and evaluation of a Portuguese AAC system. In Proceedings of the 11th International Conference on Human-Computer Interaction.
14. M. Gea, M. J. Rodríguez, M. L. Rodríguez, N. Medina, R. López-Cózar, P. Paderewski, and F. L. Gutiérrez. 2006. Adaptive and context-aware hypermedia model for users with communication disabilities. In HCI related papers of Interacción 2004. Springer, 19--28.
15. E. P. Giachin. 1995. Phrase bigrams for continuous speech recognition. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP’95). IEEE, 1, 225--228.
16. K. Grabski and T. Scheffer. 2004. Sentence completion. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 433--439.
17. D. J. Higginbotham. 1992. Evaluation of keystroke savings across five assistive communication technologies. Augmentative and Alternative Communication 8, 4, 258--272.
18. D. J. Higginbotham, A. M. Bisantz, M. Sunm, K. Adams, and F. Yik. 2009. The effect of context priming and task type on augmentative communication performance. Augmentative and Alternative Communication 25, 1, 19--31.
19. D. Higginbotham, G. Lesher, B. Moulton, and B. Roark. 2012. The application of natural language processing to augmentative and alternative communication. Assistive Technology 24, 1, 14--24.
20. D. Jurafsky and J. H. Martin. 2009. Speech and Language Processing, An Introduction to Natural Language Processing, Computational Linguistics, and Speech (2nd ed.). Prentice-Hall, Upper Saddle River, NJ.
21. S. K. Kane, B. Linam-Church, K. Althoff, and D. McCall. 2012. What we talk about: Designing a context-aware communication tool for people with aphasia. In Proceedings of the 14th International Conference on Computers and Accessibility. ACM, New York, NY, 49--56.
22. G. Kim, J. Park, M. Han, S. Park, and S. Ha. 2009. Context-aware communication support system with pictographic cards. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, New York, NY, 86.
23. H. H. Koester and S. Levine. 1996. Effect of a word prediction feature on user performance. Augmentative and Alternative Communication 12, 3, 155--168.
24. H. H. Koester and S. Levine. 1998. Model simulations of user performance with word prediction. Augmentative and Alternative Communication 14, 1, 25--36.
25. G. W. Lesher and G. J. Rinkus. 2002. Domain-specific word prediction for augmentative communication. In Proceedings of the RESNA 2002 Annual Conference.
26. G. W. Lesher, B. J. Moulton, and D. J. Higginbotham. 1999. Effects of ngram order and training text size on word prediction. In Proceedings of the RESNA’99 Annual Conference. 52--54.
27. G. W. Lesher, B. J. Moulton, D. J. Higginbotham, and B. Alsofrom. 2002. Limits of human word prediction performance. In Proceedings of the CSUN 2002.
28. G. W. Lesher, B. J. Moulton, G. Rinkus, and D. J. Higginbotham. 2000. A Universal Logging Format for Augmentative Communication. US Department of Education, Office of Educational Research and Improvement, Educational Resources Information Center, Washington, DC.
29. C. Marvin, D. Beukelman, and D. Bilyeu. 1994. Vocabulary-use patterns in preschool children: Effects of context and time sampling. Augmentative and Alternative Communication 10, 4, 224--236.
30. M. Mitchell and R. Sproat. 2012. Discourse-based modeling for AAC. In Proceedings of the 3rd Workshop on Speech and Language Processing for Assistive Technologies. Association for Computational Linguistics. 9--18.
31. E. Reiter, R. Turner, N. Alm, R. Black, M. Dempster, and A. Waller. 2009. Using NLG to help language-impaired users tell stories and participate in social dialogues. In Proceedings of the 12th European Workshop on Natural Language Generation. Association for Computational Linguistics. 1--8.
32. S. M. Shieber and E. Baker. 2003. Abbreviated text input. In Proceedings of the 8th International Conference on Intelligent User Interfaces. ACM, New York, NY, 293--296.
33. J. Todman, N. Alm, J. Higginbotham, and P. File. 2008. Whole utterance approaches in AAC. Augmentative and Alternative Communication 24, 3, 235--254.
34. K. Trnka and K. F. McCoy. 2008. Evaluating word prediction: framing keystroke savings. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers. Association for Computational Linguistics, 261--264.
35. K. Trnka, J. McCaw, D. Yarrington, K. F. McCoy, and C. Pennington. 2009. User interaction with word prediction: The effects of prediction quality. ACM Transactions on Accessible Computing (TACCESS) 1, 3, 17.
36. K. Trnka, D. Yarrington, K. McCoy, and C. Pennington. 2006. Topic modeling in fringe word prediction for AAC. In Proceedings of the 11th International Conference on Intelligent User Interfaces. ACM, New York, NY, 276--278.
37. M. A. Turk and A. P. Pentland. 1991. Face recognition using eigenfaces. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 1991 (CVPR’91). IEEE. 586--591.
38. H. Venkatagiri. 1993. Efficiency of lexical prediction as a communication acceleration technique. Augmentative and Alternative Communication 9, 3, 161--167.
39. H. Venkatagiri. 1994. Effect of window size on rate of communication in a lexical prediction AAC system. Augmentative and Alternative Communication 10, 2, 105--112.
40. P. Viola and M. J. Jones. 2004. Robust real-time face detection. International Journal of Computer Vision 57, 2, 137--154.
41. B. Wisenburn and D. J. Higginbotham. 2008. An AAC application using speaking partner speech recognition to automatically produce contextually relevant utterances: Objective results. Augmentative and Alternative Communication 24, 2, 100--109.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Natural language interfaces