skip to main content
research-article

NLUBroker: A QoE-driven Broker System for Natural Language Understanding Services

Published:01 February 2022Publication History
Skip Abstract Section

Abstract

Cloud-based Natural Language Understanding (NLU) services are becoming more popular with the development of artificial intelligence. More applications are integrated with cloud-based NLU services to enhance the way people communicate with machines. However, with NLU services provided by different companies powered by unrevealed AI technology, how to choose the best one is a problem for developers. Existing tools that can provide guidance to developers and make recommendations based on their needs are severely limited. This article comprehensively evaluates multiple state-of-the-art NLU services, and the results indicate that there is no absolute winner for different usage requirements. Motivated by this observation, we provide several insights and propose NLUBroker, a Quality of Experience-driven (QoE-driven) broker system, to select the proper service according to the environment. NLUBroker senses the client and service status and leverages a solution to the multi-armed bandit problem to conduct online learning, aiming to achieve maximum expected QoE. The performance of NLUBroker is evaluated in both simulation and real-world environments, and the evaluation results demonstrate that NLUBroker is an efficient solution for selecting NLU services. It is adaptive to changes in the environment, outperforms three baseline methods we evaluated and improves overall QoE up to 1.5× for the evaluated state-of-the-art NLU services.

REFERENCES

  1. [1] Amazon. 2020. Amazon Lex. Retrieved from https://aws.amazon.com/lex/.Google ScholarGoogle Scholar
  2. [2] Aske Austin and Zhao Xinghui. 2018. Supporting multi-provider serverless computing on the edge. In Proceedings of the 47th International Conference on Parallel Processing Companion. 16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Auer Peter. 2002. Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn. Res. 3, (November2002), 397422. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. [4] Baraković Sabina and Skorin-Kapov Lea. 2017. Modelling the relationship between design/performance factors and perceptual features contributing to Quality of Experience for mobile Web browsing. Comput. Hum. Behav. 74 (2017), 311329.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Braun Daniel, Hernandez-Mendez Adrian, Matthes Florian, and Langen Manfred. 2017. Evaluating natural language understanding services for conversational question answering systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue. 174185.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Cooper Brian F., Silberstein Adam, Tam Erwin, Ramakrishnan Raghu, and Sears Russell. 2010. Benchmarking cloud serving systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing. ACM, 143154. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Coucke Alice, Saade Alaa, Ball Adrien, Bluche Théodore, Caulier Alexandre, Leroy David, Doumouro Clément, Gisselbrecht Thibault, Caltagirone Francesco, Lavril Thibaut, et al. 2018. Snips voice platform: An embedded spoken language understanding system for private-by-design voice interfaces. arXiv:1805.10190, Retrieved from https://arxiv.org/abs/1805.10190.Google ScholarGoogle Scholar
  8. [8] Dastjerdi Amir Vahid and Buyya Rajkumar. 2017. A taxonomy of QoS management and service selection methodologies for cloud computing. In Cloud Computing. CRC Press, 109–131.Google ScholarGoogle Scholar
  9. [9] Dilmegani Cem. 2021. Guide to Choose Your Chatbot Platform: Top 5 Systems Reviewed. Retrieved January 6th, 2022 from https://research.aimultiple.com/dialogflow/.Google ScholarGoogle Scholar
  10. [10] Eisa Mona, Younas Muhammad, Basu Kashinath, and Zhu Hong. 2016. Trends and directions in cloud service selection. In Proceedings of the IEEE Symposium on Service-Oriented System Engineering (SOSE’16). IEEE, 423432.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Elger Peter and Shanaghy Eóin. 2020. AI as a Service: Serverless Machine Learning with AWS. Manning Publications.Google ScholarGoogle Scholar
  12. [12] Elhabbash Abdessalam, Samreen Faiza, Hadley James, and Elkhatib Yehia. 2019. Cloud brokerage: A systematic survey. ACM Comput. Surv. 51, 6 (2019), 128. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Facebook. 2020. Wit.ai. Retrieved from https://wit.ai/.Google ScholarGoogle Scholar
  14. [14] Gan Yu, Zhang Yanqi, Cheng Dailun, Shetty Ankitha, Rathi Priyal, Katarki Nayan, Bruno Ariana, Hu Justin, Ritchken Brian, Jackson Brendon, et al. 2019. An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems. In Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems. ACM, 318. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Google. 2019. Dialogflow. Retrieved from https://cloud.google.com/dialogflow.Google ScholarGoogle Scholar
  16. [16] Gray Jim. 1992. Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Hentschel Raoul and Strahringer Susanne. 2020. A broker-based framework for the recommendation of cloud services: A research proposal. Respons. Des. Implement. Use Inf Commun. Technol. 12066 (2020), 409.Google ScholarGoogle Scholar
  18. [18] Hintjens Pieter. 2013. ZeroMQ: Messaging for Many Applications. O’Reilly Media, Inc.Google ScholarGoogle Scholar
  19. [19] IBM. [n.d.]. IBM Watson Products and Solutions. Retrieved from https://www.ibm.com/watson/services/conversation/.Google ScholarGoogle Scholar
  20. [20] Jacobs Ian, Powers Stephen, Seguin Bill, and Lynch Diane. 2017. The top 10 chatbots for enterprise customer service. Forrest. Rep. (2017).Google ScholarGoogle Scholar
  21. [21] Jyoti Amrita and Shrimali Manish. 2020. Dynamic provisioning of resources based on load balancing and service broker policy in cloud computing. Clust. Comput. 23, 1 (2020), 377395.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Jyoti Amrita, Shrimali Manish, Tiwari Shailesh, and Singh Harivans Pratap. 2020. Cloud computing using load balancing and service broker policy for IT service: A taxonomy and survey. J. Ambient Intell. Humaniz. Comput. 11, 11 (2020), 47854814.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Kantarci Atakan. 2021. Natural Language Platforms: Top NLP APIs & Comparison. Retrieved January 6th, 2022 from https://research.aimultiple.com/natural-language-platforms/.Google ScholarGoogle Scholar
  24. [24] learn Scikit. 2020. Scikit-Learn Machine Learning in Python. Retrieved from https://scikit-learn.org/.Google ScholarGoogle Scholar
  25. [25] Li Ang, Yang Xiaowei, Kandula Srikanth, and Zhang Ming. 2010. CloudCmp: Comparing public cloud providers. In Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement. ACM, 114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Li Lihong, Chu Wei, Langford John, and Schapire Robert E.. 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web. 661670. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] Liu Xingkun, Eshghi Arash, Swietojanski Pawel, and Rieser Verena. 2021. Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction. Springer, 165–183.Google ScholarGoogle Scholar
  28. [28] Microsoft. 2019. Language Understanding (LUIS). Retrieved from https://www.luis.ai/.Google ScholarGoogle Scholar
  29. [29] Möller Sebastian and Raake Alexander. 2014. Quality of Experience: Advanced Concepts, Applications and Methods. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Nawaz Falak, Asadabadi Mehdi Rajabi, Janjua Naeem Khalid, Hussain Omar Khadeer, Chang Elizabeth, and Saberi Morteza. 2018. An MCDM method for cloud service selection using a markov chain and the best-worst method. Knowl.-Bas. Syst. 159 (2018), 120131.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Ouyang Tao, Li Rui, Chen Xu, Zhou Zhi, and Tang Xin. 2019. Adaptive user-managed service placement for mobile edge computing: An online learning approach. In Proceedings of the IEEE Conference on Computer Communications (INFOCOM’19). IEEE, 14681476.Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Patiniotakis Ioannis, Verginadis Yiannis, and Mentzas Gregoris. 2015. PuLSaR: Preference-based cloud service selection for cloud service brokers. J. Internet Serv. Appl. 6, 1 (2015), 114.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Resnik Philip and Lin Jimmy. 2010. Evaluation of NLP systems. Handbk. Comput. Ling. Nat. Lang. Process. 57 (2010), 271295.Google ScholarGoogle Scholar
  34. [34] Shridhar Kumar, Sahu Amit, Dash Ayushman, Alonso Pedro, Pihlgren Gustav, Pondeknath Vinay, Simistira Fotini, and Liwicki Marcus. 2019. Subword semantic hashing for intent classification on small datasets. In International Joint Conference on Neural Networks (IJCNN’19). IEEE, 1–6.Google ScholarGoogle Scholar
  35. [35] SimPy Team. 2020. Simpy: Discrete Event Simulation for Python. Retrieved from https://simpy.readthedocs.io/en/latest/.Google ScholarGoogle Scholar
  36. [36] Sun Le, Dong Hai, Hussain Farookh Khadeer, Hussain Omar Khadeer, and Chang Elizabeth. 2014. Cloud service selection: State-of-the-art and future research directions. J. Netw. Comput. Appl. 45 (2014), 134150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Tendrl. 2017. Best Practices for Response Times and Latency. Retrieved from https://github.com/Tendrl/documentation/wiki/Best-Practices-for-Response-Times-and-Latency.Google ScholarGoogle Scholar
  38. [38] Wang Lei, Zhan Jianfeng, Luo Chunjie, Zhu Yuqing, Yang Qiang, He Yongqiang, Gao Wanling, Jia Zhen, Shi Yingjie, Zhang Shujie, et al. 2014. BigDataBench: A big data benchmark suite from internet services. In Proceedings of the IEEE 20th International Symposium on High Performance Computer Architecture (HPCA’14). IEEE, 488499.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Wanigasekara Nirandika, Schmalfuss Jenny, Carlson Darren, and Rosenblum David S. 2016. A bandit approach for intelligent IoT service composition across heterogeneous smart spaces. In Proceedings of the 6th International Conference on the Internet of Things. 121129. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. [40] Wu Weiqiang, Yang Jing, and Shen Cong. 2020. Stochastic linear contextual bandits with diverse contexts. In Proceedings of the International Conference on Artificial Intelligence and Statistics. PMLR, 23922401.Google ScholarGoogle Scholar
  41. [41] Xu Lanyu, Iyengar Arun, and Shi Weisong. 2019. NLUBroker: A flexible and responsive broker for cloud-based natural language understanding services. In Proceedings of the 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud’19).Google ScholarGoogle Scholar

Index Terms

  1. NLUBroker: A QoE-driven Broker System for Natural Language Understanding Services

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Internet Technology
          ACM Transactions on Internet Technology  Volume 22, Issue 3
          August 2022
          631 pages
          ISSN:1533-5399
          EISSN:1557-6051
          DOI:10.1145/3498359
          • Editor:
          • Ling Liu
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 1 February 2022
          • Accepted: 1 November 2021
          • Revised: 1 September 2021
          • Received: 1 February 2021
          Published in toit Volume 22, Issue 3

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!