Abstract
Cloud-based Natural Language Understanding (NLU) services are becoming more popular with the development of artificial intelligence. More applications are integrated with cloud-based NLU services to enhance the way people communicate with machines. However, with NLU services provided by different companies powered by unrevealed AI technology, how to choose the best one is a problem for developers. Existing tools that can provide guidance to developers and make recommendations based on their needs are severely limited. This article comprehensively evaluates multiple state-of-the-art NLU services, and the results indicate that there is no absolute winner for different usage requirements. Motivated by this observation, we provide several insights and propose NLUBroker, a Quality of Experience-driven (QoE-driven) broker system, to select the proper service according to the environment. NLUBroker senses the client and service status and leverages a solution to the multi-armed bandit problem to conduct online learning, aiming to achieve maximum expected QoE. The performance of NLUBroker is evaluated in both simulation and real-world environments, and the evaluation results demonstrate that NLUBroker is an efficient solution for selecting NLU services. It is adaptive to changes in the environment, outperforms three baseline methods we evaluated and improves overall QoE up to 1.5× for the evaluated state-of-the-art NLU services.
- [1] . 2020. Amazon Lex. Retrieved from https://aws.amazon.com/lex/.Google Scholar
- [2] . 2018. Supporting multi-provider serverless computing on the edge. In Proceedings of the 47th International Conference on Parallel Processing Companion. 1–6. Google Scholar
Digital Library
- [3] . 2002. Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn. Res. 3, (November2002), 397–422. Google Scholar
Digital Library
- [4] . 2017. Modelling the relationship between design/performance factors and perceptual features contributing to Quality of Experience for mobile Web browsing. Comput. Hum. Behav. 74 (2017), 311–329.Google Scholar
Cross Ref
- [5] . 2017. Evaluating natural language understanding services for conversational question answering systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue. 174–185.Google Scholar
Cross Ref
- [6] . 2010. Benchmarking cloud serving systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing. ACM, 143–154. Google Scholar
Digital Library
- [7] . 2018. Snips voice platform: An embedded spoken language understanding system for private-by-design voice interfaces. arXiv:1805.10190, Retrieved from https://arxiv.org/abs/1805.10190.Google Scholar
- [8] . 2017. A taxonomy of QoS management and service selection methodologies for cloud computing. In Cloud Computing. CRC Press, 109–131.Google Scholar
- [9] . 2021. Guide to Choose Your Chatbot Platform: Top 5 Systems Reviewed. Retrieved January 6th, 2022 from https://research.aimultiple.com/dialogflow/.Google Scholar
- [10] . 2016. Trends and directions in cloud service selection. In Proceedings of the IEEE Symposium on Service-Oriented System Engineering (SOSE’16). IEEE, 423–432.Google Scholar
Cross Ref
- [11] . 2020. AI as a Service: Serverless Machine Learning with AWS. Manning Publications.Google Scholar
- [12] . 2019. Cloud brokerage: A systematic survey. ACM Comput. Surv. 51, 6 (2019), 1–28. Google Scholar
Digital Library
- [13] . 2020. Wit.ai. Retrieved from https://wit.ai/.Google Scholar
- [14] . 2019. An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems. In Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems. ACM, 3–18. Google Scholar
Digital Library
- [15] . 2019. Dialogflow. Retrieved from https://cloud.google.com/dialogflow.Google Scholar
- [16] . 1992. Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc. Google Scholar
Digital Library
- [17] . 2020. A broker-based framework for the recommendation of cloud services: A research proposal. Respons. Des. Implement. Use Inf Commun. Technol. 12066 (2020), 409.Google Scholar
- [18] . 2013. ZeroMQ: Messaging for Many Applications. O’Reilly Media, Inc.Google Scholar
- [19] . [n.d.]. IBM Watson Products and Solutions. Retrieved from https://www.ibm.com/watson/services/conversation/.Google Scholar
- [20] . 2017. The top 10 chatbots for enterprise customer service. Forrest. Rep. (2017).Google Scholar
- [21] . 2020. Dynamic provisioning of resources based on load balancing and service broker policy in cloud computing. Clust. Comput. 23, 1 (2020), 377–395.Google Scholar
Digital Library
- [22] . 2020. Cloud computing using load balancing and service broker policy for IT service: A taxonomy and survey. J. Ambient Intell. Humaniz. Comput. 11, 11 (2020), 4785–4814.Google Scholar
Cross Ref
- [23] . 2021. Natural Language Platforms: Top NLP APIs & Comparison. Retrieved January 6th, 2022 from https://research.aimultiple.com/natural-language-platforms/.Google Scholar
- [24] . 2020. Scikit-Learn Machine Learning in Python. Retrieved from https://scikit-learn.org/.Google Scholar
- [25] . 2010. CloudCmp: Comparing public cloud providers. In Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement. ACM, 1–14. Google Scholar
Digital Library
- [26] . 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web. 661–670. Google Scholar
Digital Library
- [27] . 2021. Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction. Springer, 165–183.Google Scholar
- [28] . 2019. Language Understanding (LUIS). Retrieved from https://www.luis.ai/.Google Scholar
- [29] . 2014. Quality of Experience: Advanced Concepts, Applications and Methods. Springer. Google Scholar
Digital Library
- [30] . 2018. An MCDM method for cloud service selection using a markov chain and the best-worst method. Knowl.-Bas. Syst. 159 (2018), 120–131.Google Scholar
Cross Ref
- [31] . 2019. Adaptive user-managed service placement for mobile edge computing: An online learning approach. In Proceedings of the IEEE Conference on Computer Communications (INFOCOM’19). IEEE, 1468–1476.Google Scholar
Cross Ref
- [32] . 2015. PuLSaR: Preference-based cloud service selection for cloud service brokers. J. Internet Serv. Appl. 6, 1 (2015), 1–14.Google Scholar
Cross Ref
- [33] . 2010. Evaluation of NLP systems. Handbk. Comput. Ling. Nat. Lang. Process. 57 (2010), 271–295.Google Scholar
- [34] . 2019. Subword semantic hashing for intent classification on small datasets. In International Joint Conference on Neural Networks (IJCNN’19). IEEE, 1–6.Google Scholar
- [35] . 2020. Simpy: Discrete Event Simulation for Python. Retrieved from https://simpy.readthedocs.io/en/latest/.Google Scholar
- [36] . 2014. Cloud service selection: State-of-the-art and future research directions. J. Netw. Comput. Appl. 45 (2014), 134–150. Google Scholar
Digital Library
- [37] . 2017. Best Practices for Response Times and Latency. Retrieved from https://github.com/Tendrl/documentation/wiki/Best-Practices-for-Response-Times-and-Latency.Google Scholar
- [38] . 2014. BigDataBench: A big data benchmark suite from internet services. In Proceedings of the IEEE 20th International Symposium on High Performance Computer Architecture (HPCA’14). IEEE, 488–499.Google Scholar
Cross Ref
- [39] . 2016. A bandit approach for intelligent IoT service composition across heterogeneous smart spaces. In Proceedings of the 6th International Conference on the Internet of Things. 121–129. Google Scholar
Digital Library
- [40] . 2020. Stochastic linear contextual bandits with diverse contexts. In Proceedings of the International Conference on Artificial Intelligence and Statistics. PMLR, 2392–2401.Google Scholar
- [41] . 2019. NLUBroker: A flexible and responsive broker for cloud-based natural language understanding services. In Proceedings of the 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud’19).Google Scholar
Index Terms
NLUBroker: A QoE-driven Broker System for Natural Language Understanding Services
Recommendations
Management of SOA based NGN service exposure, service discovery and service composition
IM'09: Proceedings of the 11th IFIP/IEEE international conference on Symposium on Integrated Network ManagementNext generation telecommunication network operators securely opening up their network capabilities and services to third party service providers require flexible service delivery platforms. Policy based service exposure, service discovery and service ...
Towards Quality of Experience Driven Service Composition
SERVICES '14: Proceedings of the 2014 IEEE World Congress on ServicesWeb service composition enables seamless and dynamic integration of applications on the web. Generally a user has to find services, select proper services and form a flow to create a service composition. The performance of the composed application is ...
Intelligent Broker
ENASE 2015: Proceedings of the 10th International Conference on Evaluation of Novel Approaches to Software EngineeringEver since the introduction of the service oriented model of computing (SOA), service discovery has been the major research challenge in SOA. Service consumers usually prefer to express their requirements informally. Expressing requirements in such a ...






Comments