Staging transformations for multimodal web interaction management

ABSTRACT

Multimodal interfaces are becoming increasingly ubiquitous with the advent of mobile devices, accessibility considerations, and novel software technologies that combine diverse interaction media. In addition to improving access and delivery capabilities, such interfaces enable flexible and personalized dialogs with websites, much like a conversation between humans. In this paper, we present a software framework for multimodal web interaction management that supports mixed-initiative dialogs between users and websites. A mixed-initiative dialog is one where the user and the website take turns changing the flow of interaction. The framework supports the functional specification and realization of such dialogs using staging transformations -- a theory for representing and reasoning about dialogs based on partial input. It supports multiple interaction interfaces, and offers sessioning, caching, and co-ordination functions through the use of an interaction manager. Two case studies are presented to illustrate the promise of this approach.

References

1. Speech Application Language Tags (SALT) Specification. Technical report, SALT Forum, July 2002. Version 1.0.
2. J. F. Allen, C. I. Guinn, and E. Horvitz. Mixed-Initiative Interaction. IEEE Intelligent Systems, Vol. 14(5):pages 14--23, September--October 1999.
3. E. André and T. Rist. From Adaptive Hypertext to Personalized Web Companions. CACM, Vol. 45(5):pages 43--46, May 2002.
4. Y. Aridor, D. Carmel, Y. S. Maarek, A. Soffer, and R. Lempel. Knowledge Encapsulation for Focused Search from Pervasive Devices. In Proc. WWW10, pages 754--764, 2001.
5. D. L. Atkins, T. Ball, G. Bruns, and K. Cox. Mawl: A Domain-Specific Language for Form-Based Services. IEEE Transactions on Software Engineering, Vol. 25(3):pages 334--346, May--June 1999.
6. J. Axelsson, C. Cross, H. Lie, G. McCobb, T. Raman, and L. Wilson (eds.). Xhtml+voice profile 1.0. W3C Note, December 2001.
7. N. J. Belkin, C. Cool, A Stein, and U. Thiel. Cases, Scripts, and Information Seeking Strategies: On the Design of Interactive Information Retrieval Systems. Expert Systems with Applications, Vol. 9(3):pages 379--395, 1995.
8. D. W. Binkley and K. B. Gallagher. Program Slicing. In Advances in Computers, volume 43, pages 1--50. 1996.
9. P. De Bra, P. Brusilovsky, and G.-J. Houben. Adaptive Hypermedia: From Systems to Framework. ACM Computing Surveys, Vol. 31(4es), December 1999. Article No. 12.
10. C. Brabrand, A. Møller, and M. I. Schwartzbach. The <bigwig> Project. ACM Transactions on Internet Technology, Vol. 2(2):pages 79--14, May 2002.
11. P. Brusilovsky. Adaptive Hypermedia. User Modeling and User-Adapted Interaction, Vol. 11(1--2):pages 87--110, 2001.
12. O. Buyukkokten, H. Garcia-Molina, and A. Paepcke. Seeing the Whole in Parts: Text Summarization for Web Browsing on Handheld Devices. In Proc. WWW10, pages 652--662, 2001.
13. R. Capra, M. Narayan, S. Perugini, N. Ramakrishnan, and M. A. P 233;rez-Qui 241;ones. The Staging Transformation Approach to Mixing Initiative. In G. Tecuci, editor, Working Notes of the IJCAI 2003 Workshop on Mixed-Initiative Intelligent Systems, pages 23--29. AAAI/MIT Press, August 2003.
14. R. Capra, M. A. Pérez-Quiñones, and N. Ramakrishnan. WebContext: Remote Access to Shared Context. In Proc. PUI, November 2001.
15. S. K. Card, T. P. Moran, and A. Newell. Computer Text-Editing: An Information-Processing Analysis of a Routine Cognitive Skill. Cognitive Psychology, Vol. 12:pages 32--74, 1980.
16. J. Chen, B. Zhou, J. Shi, H. Zhang, and Q. Fengwu. Function-Based Object Model Towards Website Adaptation. In Proc. WWW10, pages 587--596, 2001.
17. Y. Chen, W.-Y. Ma, and H.-J. Zhang. Detecting Web Page Structure for Adaptive Viewing on Small Form Factor Devices. In Proc. WWW12, pages 225--233, 2003.
18. D. Cohen, M. Herscovici, Y. Petruschka, Y. S. Maarek, and A. Soffer. Personalized Pocket Directories for Mobile Devices. In Proc. WWW11, pages 627--638, 2002.
19. A. Coles, E. Deliot, T. Melamed, and K. Lansard. A Framework for Coordinated Multi-Modal Browsing with Multiple Clients. In Proc. WWW12, pages 718--726, 2003.
20. S. Dumais. Tightly Coupling Structure and Search. In Proc. SIGIR Workshop on Information Reduction, July 1997.
21. J. Freire, B. Kumar, and D. Lieuwen. WebViews: Accessing Personalized Web Content and Services. In Proc. WWW10, pages 576--586, 2001.
22. P. Graunke, R. Findler, S. Krishnamurthi, and M. Felleisen. Automatically Restructuring Programs for the Web. In Proc. ASE, November 2001.
23. B. J. Grosz and C. L. Sidner. Attention, Intentions, and the Structure of Discourse. Computational Linguistics, Vol. 12:pages 175--204, 1986.
24. M. A. Hearst, A. Elliott, J. English, R. Sinha, K. Swearingen, and K.-P. Yee. Finding the Flow in Web Site Search. CACM, Vol. 45(9):pages 42--49, September 2002.
25. S. Horwitz, T. Reps, and D. W. Binkley. Interprocedural Slicing Using Dependency Graphs. ACM Transactions on Programming Languages and Systems, Vol. 12(1):pages 26--60, January 1990.
26. N. D. Jones. An Introduction to Partial Evaluation. ACM Computing Surveys, Vol. 28(3):pages 480--503, September 1996.
27. I.-Y. Ko, K.-T. Yao, and R. Neches. Dynamic Coordination of Information Management Services for Processing Dynamic Web Content. In Proc. WWW11, pages 355--365, 2002.
28. J. Lai. Conversation Interfaces. CACM, Vol. 43(9):pages 24--27, September 2000.
29. G. Marchionini. Information Seeking in Electronic Environments. Cambridge Series on Human-Computer Interaction. Cambridge University Press, 1997.
30. S. McGlashan, D. Burnett, P. Danielsen, J. Ferrans, A. Hunt, G. Karam, D. Ladd, B. Lucas, B. Porter, K. Rehor, and S. Tryphonas. Voice eXtensible Markup Language: VoiceXML. Technical report, VoiceXML Forum, October 2001. Version 2.0.
31. B. Mobashier, R. Cooley, and J. Srivastava. Automatic Personalization Based on Web Usage Mining. CACM, Vol. 43(8):pages 142--151, August 2000.
32. B. A. Myers and M. Beigl. Guest Editors' Introduction: Handheld Computing. IEEE Computer, Vol. 36(9):pages 27--29, September 2003.
33. S. Perugini, M. E. Pinney, N. Ramakrishnan, M. A. Pérez-Quiñones, and M. B. Rosson. Taking the Initiative with Extempore: Exploring Out-of-Turn Interactions with Websites. Technical Report cs.HC/0312016, Computing Research Repository (CoRR), December 2003.
34. S. Perugini and N. Ramakrishnan. Personalizing Interactions with Information Systems. In Advances in Computers, volume 57: Information Repositories, pages 323--382. September 2003.
35. S. Perugini and N. Ramakrishnan. Personalizing Web Sites with Mixed-Initiative Interaction. IEEE IT Professional, Vol. 5(2):pages 9--15, March--April 2003.
36. J. Pokorny. Static Pages are Dead: How a Modular Approach is Changing Interaction Design. ACM Interactions, Vol. 8(5):pages 19--24, September--October 2001.
37. D. Quan, D. Huynh, D. R. Karger, and R. Miller. User Interface Continuations. In Sixteenth ACM Symposium on User Interface Software and Technology (UIST), November 2003.
38. C. Queinnec. The Influence of Browsers on Evaluators or, Continuations to ProgramWeb Servers. In Proc. ICFP, pages 23--33, September 2000.
39. S. Rollins and N. Sundaresan. AVoN Calling: AXL for Voice-Enabled Web Navigation. In Proc. WWW9, 2000.
40. G. M. Sacco. Dynamic Taxonomies: A Model for Large Information Bases. IEEE Transactions on Knowledge and Data Engineering, Vol. 12(3):pages 468--479, May--June 2000.
41. S. Srinivasan and E. Brown. Is Speech Recognition Becoming Mainstream? IEEE Computer, Vol. 35(4):pages 38--41, April 2002.
42. J. Steinberg and J. Pasquale. A Web Middleware Architecture for Dynamic Customization of Content for Wireless Clients. In Proc. WWW11, pages 639--650, 2002.
43. J. Veitch. A Conversation with Paul Graham. CACM, Vol. 41(5):pages 52--54, May 1998.
44. K. Wang. A Study of Semantic Synchronous Understanding on Speech Interface Design. In Proc. UIST'03, November 2003.
45. T. Winograd and F. Flores, editors. Understanding Computers and Cognition -- A New Foundation for Design. Addison-Wesley, Reading, PA, 1987.
46. S. A. Wolfman, T. Lau, P. Domingos, and D. S. Weld. Mixed Initiative Interfaces for Learning Tasks: SMARTedit Talks Back. In Proc. IUI, pages 167--174, 2001.
47. N. Yankelovich. How Do Users Know What To Say? ACM Interactions, 3(6):pages 32--43, November--December 1996.

Index Terms

• Applied computing

  • Document management and text processing

    • Document preparation

      • Multi / mixed media creation

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Graphics input devices

    • Interaction paradigms

      • Hypertext / hypermedia

• Software and its engineering

  • Software creation and management

    • Designing software

      • Software implementation planning

        • Software design techniques

    • Software development process management

• Theory of computation

  • Semantics and reasoning

    • Program semantics