short-paper

Automated generation of test oracles for RESTful APIs

Author:

Juan C. AlonsoAuthors Info & Claims

ESEC/FSE 2022: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 1808 - 1810

https://doi.org/10.1145/3540250.3559080

Published: 09 November 2022 Publication History

Abstract

Test case generation tools for RESTful APIs have proliferated in recent years. However, despite their promising results, they all share the same limitation: they can only detect crashes (i.e., server errors) and disconformities with the API specification. In this paper, we present a technique for the automated generation of test oracles for RESTful APIs through the detection of invariants. In practice, our approach aims to learn the expected properties of the output by analysing previous API requests and their corresponding responses. For this, we extended the popular tool Daikon for dynamic detection of likely invariants. A preliminary evaluation conducted on a set of 8 operations from 6 industrial APIs reveals a total precision of 66.5% (reaching 100% in 2 operations). Moreover, our approach revealed 6 reproducible bugs in APIs with millions of users: Amadeus, GitHub and OMDb.

References

[1]

2022. DAIKON instrumenters. https://plse.cs.washington.edu/daikon/download/doc/daikon.html#Front-ends-_0028instrumentation_0029 Accessed July 2022

[2]

2022. OpenAPI Specification. https://www.openapis.org accessed July 2022

[3]

Afsoon Afzal, Claire Le Goues, and Christopher Steven Timperley. 2021. Mithra: Anomaly Detection as an Oracle for Cyberphysical Systems. IEEE Transactions on Software Engineering, 1–1. https://doi.org/10.1109/TSE.2021.3120680

[4]

Juan C. Alonso, Alberto Martin-Lopez, Sergio Segura, Jose Maria Garcia, and Antonio Ruiz-Cortes. 2022. ARTE: Automated Generation of Realistic Test Inputs for Web APIs. IEEE Transactions on Software Engineering, https://doi.org/10.1109/TSE.2022.3150618

[5]

V. Atlidakis, P. Godefroid, and M. Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 748–758. https://doi.org/10.1109/ICSE.2019.00083

Digital Library

[6]

Jake Cobb, James A. Jones, Gregory M. Kapfhammer, and Mary Jean Harrold. 2011. Dynamic Invariant Detection for Relational Databases. In Proceedings of the Ninth International Workshop on Dynamic Analysis (WODA ’11). Association for Computing Machinery, New York, NY, USA. 12–17. isbn:9781450308113 https://doi.org/10.1145/2002951.2002955

Digital Library

[7]

Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, and Chen Xiao. 2007. The Daikon system for dynamic detection of likely invariants. Science of Computer Programming, 69, 1 (2007), 35–45. issn:0167-6423 https://doi.org/10.1016/j.scico.2007.01.015 Special issue on Experimental Software and Toolkits

Digital Library

[8]

Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-based Software Architectures. Ph. D. Dissertation. isbn:0-599-87118-0

[9]

Zac Hatfield-Dodds and Dmitry Dygalo. 2021. Deriving Semantics-Aware Fuzzers from Web API Schemas. arXiv preprint arXiv:2112.10328.

[10]

Daniel Jacobson, Greg Brail, and Dan Woods. 2011. APIs: A Strategy Guide. O’Reilly Media, Inc. isbn:1449308929, 9781449308926

[11]

Deborah S Katz, Christopher S Timperley, and Claire Le Goues. 2022. Using Dynamic Binary Instrumentation to Detect Failures in Robotics Software. arXiv preprint arXiv:2201.12464.

[12]

Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2020. RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs. In International Conference on Service-Oriented Computing. 459–475.

[13]

Ali Mesbah, Arie van Deursen, and Danny Roest. 2012. Invariant-Based Automatic Testing of Modern Web Applications. IEEE Transactions on Software Engineering, 38, 1 (2012), 35–53. https://doi.org/10.1109/TSE.2011.28

Digital Library

[14]

Facundo Molina, Pablo Ponzio, Nazareno Aguirre, and Marcelo Frias. 2021. EvoSpex: An Evolutionary Algorithm for Learning Postconditions. In 2021 IEEE/ACM 43st International Conference on Software Engineering (ICSE). 1223–1235. https://doi.org/10.1109/ICSE43902.2021.00112

Digital Library

[15]

Saurabh Sinha Myeongsoo Kim, Qi Xin and Alessandro Orso. 2022. Automated Test Generation for REST APIs: No Time to Rest Yet. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis.

[16]

Leonard Richardson, Mike Amundsen, and Sam Ruby. 2013. RESTful Web APIs. O’Reilly Media, Inc. isbn:1449358063, 9781449358068

[17]

Sergio Segura, José A. Parejo, Javier Troya, and Antonio Ruiz-Cortés. 2018. Metamorphic Testing of RESTful Web APIs. IEEE Transactions on Software Engineering, 44, 11 (2018), 1083–1099. https://doi.org/10.1109/TSE.2017.2764464

[18]

Dimitri Stallenberg, Mitchell Olsthoorn, and Annibale Panichella. 2021. Improving Test Case Generation for REST APIs Through Hierarchical Clustering. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 117–128. https://doi.org/10.1109/ASE51524.2021.9678586

Digital Library

[19]

E. Viglianisi, M. Dallago, and M. Ceccato. 2020. RESTTESTGEN: Automated Black-Box Testing of RESTful APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). 142–152. https://doi.org/10.1109/ICST46399.2020.00024

[20]

Xiaodong Yang, Omar Ali Beg, Matthew Kenigsberg, and Taylor T. Johnson. 2022. A Framework for Identification and Validation of Affine Hybrid Automata from Input-Output Traces. ACM Trans. Cyber-Phys. Syst., 6, 2 (2022), Article 13, apr, 24 pages. issn:2378-962X https://doi.org/10.1145/3470455

Digital Library

[21]

Yuntong Zhang, Xiang Gao, Gregory J Duck, and Abhik Roychoudhury. 2022. Program Vulnerability Repair via Inductive Inference.

Cited By

Alonso JSegura SRuiz-Cortés AJust RFraser G(2023)AGORA: Automated Generation of Test Oracles for REST APIsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598114(1018-1030)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598114
Hamberger PKlammer CLuger TMoser MPfeiffer MPiereder C(2023)Specification-based Test Case Generation for C++ Engineering Software2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00066(519-529)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00066

Recommendations

AGORA: Automated Generation of Test Oracles for REST APIs
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Test case generation tools for REST APIs have grown in number and complexity in recent years. However, their advanced capabilities for automated input generation contrast with the simplicity of their test oracles, which limit the types of failures ...
Automated test generation for REST APIs: no time to rest yet
ISSTA 2022: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis

Modern web services routinely provide REST APIs for clients to access their functionality. These APIs present unique challenges and opportunities for automated testing, driving the recent development of many techniques and tools that generate test cases ...
RESTful API Automated Test Case Generation with EvoMaster

RESTful APIs are widespread in industry, especially in enterprise applications developed with a microservice architecture. A RESTful web service will provide data via an API over the network using HTTP, possibly interacting with databases and other web ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2022: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 2022

1822 pages

ISBN:9781450394130

DOI:10.1145/3540250

General Chair:
Abhik Roychoudhury
National University of Singapore, Singapore
,
Program Chairs:
Cristian Cadar
Imperial College London, UK
,
Miryung Kim
University of California at Los Angeles, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 November 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ESEC/FSE '22

Sponsor:

ESEC/FSE '22: 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 14 - 18, 2022

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
204
Total Downloads

Downloads (Last 12 months)69
Downloads (Last 6 weeks)3

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alonso JSegura SRuiz-Cortés AJust RFraser G(2023)AGORA: Automated Generation of Test Oracles for REST APIsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598114(1018-1030)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598114
Hamberger PKlammer CLuger TMoser MPfeiffer MPiereder C(2023)Specification-based Test Case Generation for C++ Engineering Software2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00066(519-529)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00066

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents