Abstract
Data-driven Learning-enabled Systems are limited by the quality of available training data, particularly when trained offline. For systems that must operate in real-world environments, the space of possible conditions that can occur is vast and difficult to comprehensively predict at design time. Environmental uncertainty arises when run-time conditions diverge from design-time training conditions. To address this problem, automated methods can generate synthetic data to fill in gaps for training and test data coverage. We propose an evolution-based technique to assist developers with uncovering limitations in existing data when previously unseen environmental phenomena are introduced. This technique explores unique contexts for a given environmental condition, with an emphasis on diversity. Synthetic data generated by this technique may be used for two purposes: (1) to assess the robustness of a system to uncertain environmental factors and (2) to improve the system’s robustness. This technique is demonstrated to outperform random and greedy methods for multiple adverse environmental conditions applied to image-processing Deep Neural Networks.
- Alejandro Barredo Arrieta, Natalia Diaz-Rodriguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Info. Fusion 58 (June 2020), 82–115. DOI:https://doi.org/10.1016/j.inffus.2019.12.012Google Scholar
Digital Library
- David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, Piscataway, NJ, 3319–3327. DOI:https://doi.org/10.1109/CVPR.2017.354Google Scholar
Cross Ref
- Yoshua Bengio. 2020. Priors For Deep Learning of Semantic Representations. In Proceedings of the ACM/IEEE International Conference on Model-driven Engineering Languages and Systems (MODELS’20).Google Scholar
- James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13 (Feb. 2012), 281–305.Google Scholar
Digital Library
- Mohamed Boussaa, Olivier Barais, Gerson Sunyé, and Benoît Baudry. 2015. A novelty search approach for automatic test data generation. In Proceedings of the 8th International Workshop on Search-based Software Testing (SBST’15). IEEE, Piscataway, NJ, 40–43. DOI:https://doi.org/10.1109/SBST.2015.17Google Scholar
Digital Library
- Mohamed Boussaa, Olivier Barais, Gerson Sunyé, and Benoît Baudry. 2015. A novelty search-based test data generator for object-oriented programs. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO’15). ACM, New York, NY, 1359–1360. DOI:https://doi.org/10.1145/2739482.2764716Google Scholar
Digital Library
- Carla E. Brodley and Mark A. Friedl. 1999. Identifying mislabeled training data. J. Artif. Int. Res. 11, 1 (July 1999), 131–167.Google Scholar
Digital Library
- Simon Burton, Lydia Gauerhof, and Christian Heinzemann. 2017. Making the case for safety of machine learning in highly automated driving. In Proceedings of the International Conference on Computer Safety, Reliability, and Security. Lecture Notes in Computer Science (LNCS), Vol. 10489. Springer, Cham.Google Scholar
- Gul Calikli and Ayse Bener. 2010. Empirical analyses of the factors affecting confirmation bias and the effects of confirmation bias on software developer/tester performance. In Proceedings of the 6th International Conference on Predictive Models in Software Engineering (PROMISE’10). ACM, New York, NY, Article 10, 11 pages. DOI:https://doi.org/10.1145/1868328.1868344Google Scholar
Digital Library
- Anthony Clark, Byron DeVries, Jared Moore, Betty H. C. Cheng, and Philip McKinley. 2016. An evolutionary approach to discovering execution mode boundaries for adaptive controllers. In Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI’16). IEEE, Piscataway, NJ, 1–8. DOI:https://doi.org/10.1109/SSCI.2016.7850178Google Scholar
Cross Ref
- Fabiano Dalpiaz, Alexander Borgida, Jennifer Horkoff, and John Mylopoulos. 2013. Runtime goal models: Keynote. In Proceedings of 7th IEEE International Conference on Research Challenges in Information Science (RCIS’13). IEEE, Piscataway, NJ, 1–11. DOI:https://doi.org/10.1109/RCIS.2013.6577674Google Scholar
Cross Ref
- Mirko D’Angelo, Simos Gerasimou, Sona Ghahremani, Johannes Grohmann, Ingrid Nunes, Evangelos Pournaras, and Sven Tomforde. 2019. On learning in collective self-adaptive systems: State of practice and a 3D framework. In Proceedings of the 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’19). IEEE, Piscataway, NJ, 13–24. DOI:https://doi.org/10.1109/SEAMS.2019.00012Google Scholar
Digital Library
- Junhua Ding, Xiaojun Kang, and Xin-Hua Hu. 2017. Validating a deep learning framework by metamorphic testing. In Proceedings of the 2nd International Workshop on Metamorphic Testing (MET’17). IEEE, Piscataway, NJ, 28–34. DOI:https://doi.org/10.1109/MET.2017.2Google Scholar
Digital Library
- Agoston E. Eiben and James E. Smith. 2015. Introduction to Evolutionary Computing (2nd ed.). Springer-Verlag, Berlin, Heidelberg, DEU.Google Scholar
Digital Library
- Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2018. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). IEEE, Piscataway, NJ, 1625–1634. DOI:https://doi.org/10.1109/CVPR.2018.00175Google Scholar
Cross Ref
- Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE’11). ACM, New York, NY, 416–419. DOI:https://doi.org/10.1145/2025113.2025179Google Scholar
Digital Library
- Hans Gaiser, Maarten de Vries, Valeriu Lacatusu, V. Carpani, Ashley Williamson, Enrico Liscio, Andreas, Yann Henon, jjiun, Cristian Gratie, Mihai Morariu, Charles Ye, Martin Zlocha, Ben Weinstein, Rodrigo Meira de Andrade, Pedro Conceição, Alexander Pacha, hannesedvartsen, Daniyal Shahrokhian, Wudi Fang, Mike Clark, meagerYak, Iver Jordal, Max Van Sande, Jin, Etienne-Meunier, Andrew Grigorev, Guillaume Erhard, Eduardo Ramos, and Denis Dowling. 2019. fizyr/keras-retinanet 0.5.1. Retrieved from https://github.com/fizyr/keras-retinanet/releases. DOI:https://doi.org/10.5281/zenodo.3250670Google Scholar
- Kshitiz Garg and Shree K. Nayar. 2004. Photometric Model of a Rain Drop. Technical Report. Columbia University. Retrieved from https://www1.cs.columbia.edu/CAVE/publications/pdfs/Garg_TR04.pdf.Google Scholar
- Xavier Gastaldi. 2017. Shake-Shake regularization. Retrieved from https://arXiv:1705.07485.Google Scholar
- Andreas Geiger, Philip Lenz, and Raquel Urtasun. 2012. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, Piscataway, NJ, 3354–3361. DOI:https://doi.org/10.1109/CVPR.2012.6248074Google Scholar
Cross Ref
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT, Cambridge, MA.Google Scholar
Digital Library
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. Vol. 27. Curran Associates, Red Hook, NY, 2672–2680.Google Scholar
- Ian J. Goodfellow, Jonathons Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. Retrieved from https://arXiv:1412.6572.Google Scholar
- Ronan Hamon, Henrik Junklewitz, and Ignacio Sanchez. 2020. Robustness and Explainability of Artificial Intelligence. Technical Report EUR 30040 EN. European Commission/Joint Research Centre (JRC).Google Scholar
- Karen Hao. 2019. This is how AI bias really happens—And why it’s so hard to fix. MIT Technology Review Tech policy/AI Ethics (Feb. 2019), 1 page. Retrieved from https://www.technologyreview.com/2019/02/04/137602/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix/.Google Scholar
- Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based software engineering: Trends, techniques, and applications. ACM Comput. Surv. 45, 1, Article 11 (Dec. 2012), 61 pages pages. DOI:https://doi.org/10.1145/2379776.2379787Google Scholar
- William Grant Hatcher and Wei Yu. 2018. A survey of deep learning: Platforms, applications and emerging research trends. IEEE Access 6 (2018), 24411–24432. DOI:https://doi.org/10.1109/ACCESS.2018.2830661Google Scholar
Cross Ref
- Haibo He and Edwardo A. Garcia. 2009. Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21, 9 (Sept. 2009), 1263–1284. DOI:https://doi.org/10.1109/TKDE.2008.239Google Scholar
Digital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). IEEE, Piscataway, NJ, 770–778. DOI:https://doi.org/10.1109/CVPR.2016.90Google Scholar
Cross Ref
- John F. Hughes, Andries van Dam, Morgan McGuire, David F. Sklar, James D. Foley, Steven Feiner, and Kurt Akeley. 2013. Computer Graphics: Principles and Practice. Addison-Wesley, Upper Saddle River, NJ.Google Scholar
- Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning (ICML’15), Vol. 37. Microtome Publishing, Brookline, MA, 448–456.Google Scholar
- Nick Jacobi, Phil Husbands, and Inman Harvey. 1995. Noise and the reality gap: The use of simulation in evolutionary robotics. In Proceedings of the 3rd European Conference on Advances in Artificial Life. Springer-Verlag, Berlin, Heidelberg, DEU, 704–720.Google Scholar
Cross Ref
- Joel Janai, Fatma Güney, Aseem Behl, and Andreas Geiger. 2017. Computer vision for autonomous vehicles: Problems, datasets and state-of-the-art. Retrieved from https://arXiv:1704.05519.Google Scholar
- Jason Jo and Yoshua Bengio. 2017. Measuring the tendency of CNNs to learn surface statistical regularities. Retrieved from https://arXiv:1711.11561.Google Scholar
- Siddhartha Kumar Khaitan and James D. McCalley. 2015. Design techniques and applications of cyber physical systems: A survey. IEEE Syst. J. 9, 2 (2015), 350–365.Google Scholar
Cross Ref
- Saeed Reza Kheradpisheh, Masoud Ghodrati, Mohammad Ganjtabesh, and Timothée Masquelier. 2016. Deep networks can resemble human feed-forward vision in invariant object recognition. Sci. Rep. 6, 32672 (2016), 24 pages. DOI:https://doi.org/10.1038/srep32672Google Scholar
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Retrieved from https://arXiv:1412.6980.Google Scholar
- Wilhelm Kirch (Ed.). 2008. Pearson’s Correlation Coefficient. Springer Netherlands, Dordrecht, NLD, 1090–1091. DOI:https://doi.org/10.1007/978-1-4020-5614-7_2569Google Scholar
- Will Knight. 2017. The dark secret at the heart of AI. MIT Technology Review Artificial intelligence/Machine learning (Apr. 2017), 1. Retrieved from https://www.technologyreview.com/2017/04/11/5113/the-dark-secret-at-the-heart-of-ai/.Google Scholar
- Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. Technical Report. University of Toronto.Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 6 (May 2017), 84–90. DOI:https://doi.org/10.1145/3065386Google Scholar
Digital Library
- Christian Krupitzer, Felix Maximilian Roth, Martin Pfannemüller, and Christian Becker. 2016. Comparison of approaches for self-improvement in self-adaptive systems. In Proceedings of the IEEE International Conference on Autonomic Computing (ICAC’16). IEEE, Piscataway, NJ, 308–314. DOI:https://doi.org/10.1109/ICAC.2016.18Google Scholar
Cross Ref
- Alexey Kurakin, Ian J. Goodfellow, and Sami Bengio. 2016. Adversarial machine learning at scale. Retrieved from https://arXiv:1611.01236.Google Scholar
- Michael A. Langford and Betty H. C. Cheng. 2019. Enhancing learning-enabled software systems to address environmental uncertainty. In Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC’19). IEEE, Piscataway, NJ, 115–124. DOI:https://doi.org/10.1109/ICAC.2019.00023Google Scholar
- Michael A. Langford, Glen A. Simon, Philip K. McKinley, and Betty H. C. Cheng. 2019. Applying evolution and novelty search to enhance the resilience of autonomous systems. In Proceedings of the 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’19). ACM, New York, NY, 63–69. DOI:https://doi.org/10.1109/SEAMS.2019.00017Google Scholar
- Joel Lehman. 2012. Evolution Through the Search for Novelty. Ph.D. Dissertation. University of Central Florida.Google Scholar
- Joel Lehman and Kenneth O. Stanley. 2011. Abandoning objectives: Evolution through the search for novelty alone. Evolution. Comput. 19, 2 (June 2011), 189–223. DOI:https://doi.org/10.1162/EVCO_a_00025Google Scholar
Digital Library
- Zenan Li, Xiaoxing Ma, Chang Xu, and Chun Cao. 2019. Structural coverage criteria for neural networks could be misleading. In Proceedings of the 41st International Conference on Software Engineering (ICSE-NIER’19). IEEE, Piscataway, NJ, 89–92. DOI:https://doi.org/10.1109/ICSE-NIER.2019.00031Google Scholar
Digital Library
- Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. 2017. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). IEEE, Piscataway, NJ, 2999–3007. DOI:https://doi.org/10.1109/ICCV.2017.324Google Scholar
Cross Ref
- Fredrik Lundh, Alex Clark, and Contributors. 2015. Pillow (PIL Fork) 3.0x. Read the Docs. Retrieved from https://pillow.readthedocs.io/en/3.0.x/about.html.Google Scholar
- Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, and Yang Liu. 2018. DeepGauge: Multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 120–131. DOI:https://doi.org/10.1145/3238147.3238202Google Scholar
Digital Library
- Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA’16). ACM, New York, NY, 94–105. DOI:https://doi.org/10.1145/2931037.2931054Google Scholar
Digital Library
- Phil McMinn. 2011. Search-based software testing: Past, present and future. In Proceedings of the 4th International Conference on Software Testing, Verification and Validation Workshops (ICST’11). IEEE, Piscataway, NJ, 153–163. DOI:https://doi.org/10.1109/ICSTW.2011.100Google Scholar
Digital Library
- Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML’10). Omnipress, Madison, WI, 807–814.Google Scholar
Digital Library
- Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). IEEE, Piscataway, NJ, 427–436. DOI:https://doi.org/10.1109/CVPR.2015.7298640Google Scholar
Cross Ref
- Nvidia Corporation. 2017. prepare_kitti_data.py. Retrieved from https://github.com/NVIDIA/DIGITS/blob/master/examples/object-detection/prepare_kitti_data.py.Google Scholar
- Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepXplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP’17). ACM, New York, NY, 1–18. DOI:https://doi.org/10.1145/3132747.3132785Google Scholar
Digital Library
- Andres J. Ramirez, Adam C. Jensen, Betty H. C. Cheng, and David B. Knoester. 2011. Automatically exploring how uncertainty impacts behavior of dynamically adaptive systems. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering (ASE’11). IEEE, Piscataway, NJ, 568–571. DOI:https://doi.org/10.1109/ASE.2011.6100127Google Scholar
- Colin R. Reeves. 2000. Fitness landscapes and evolutionary algorithms. In Proceedings of the International Conference on Computer Safety, Reliability, and Security (Artificial Evolution). Springer, Berlin, 3–20.Google Scholar
Cross Ref
- Charles Richter and Nicholas Roy. 2017. Safe visual navigation via deep learning and novelty detection. In Proceedings of Robotics: Science and Systems. MIT, Cambridge, MA, 64–73. DOI:https://doi.org/10.15607/RSS.2017.XIII.064Google Scholar
Cross Ref
- Stuart J. Russell and Peter Norvig. 2010. Artificial Intelligence: A Modern Approach (3 ed.). Pearson Education, Upper Saddle River, NJ.Google Scholar
Digital Library
- Ahmad Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. 2017. Deep reinforcement learning framework for autonomous driving. In Proceedings of the IS&T International Symposium on Electronic Imaging, Autonomous Vehicles and Machines (AVM’17). IS&T, Springfield, VA, 70–76. DOI:https://doi.org/10.2352/ISSN.2470-1173.2017.19.AVM-023Google Scholar
Cross Ref
- Yutaka Sasaki. 2007. The Truth of the F-measure. Technical Report. University of Manchester. Retrieved from https://www.cs.odu.edu/ mukka/cs795sum09dm/Lecturenotes/Day3/F-measure-YS-26Oct07.pdf.Google Scholar
- Johann Schumann, Pramod Gupta, and Yan Liu. 2010. Application of Neural Networks in High Assurance Systems: A Survey. Studies in Computational Intelligence (SCI), Vol. 268. Springer, Berlin.Google Scholar
- Jasmine Sekhon and Cody Fleming. 2019. Towards improved testing for deep learning. In Proceedings of the 41st International Conference on Software Engineering (ICSE-NIER’19). IEEE, Piscataway, NJ, 85–88. DOI:https://doi.org/10.1109/ICSE-NIER.2019.00030Google Scholar
Digital Library
- Valerie Sessions and Marco Valtorta. 2006. The effects of data quality on machine learning algorithms. In Proceedings of the the International Conference on Information Quality (ICIQ’06). MIT, Cambridge, MA, 485–498.Google Scholar
- Burr Settles. 2009. Active Learning Literature Survey. Computer Sciences Technical Report 1648. University of Wisconsin–Madison.Google Scholar
- James V. Stone. 2015. Information Theory: A Tutorial Introduction (1st ed.). Sebtel Press.Google Scholar
- Oliver Strömgren. 2018. Deep Learning for Autonomous Collision Avoidance. MSc Thesis. Linköping University.Google Scholar
- Youcheng Sun, Xiaowei Huang, Daniel Kroening, James Sharp, Matthew Hill, and Rob Ashmore. 2019. DeepConcolic: Testing and debugging deep neural networks. In Proceedings of the 41st International Conference on Software Engineering (ICSE’19). IEEE, Piscataway, NJ, 111–114. DOI:https://doi.org/10.1109/ICSE-Companion.2019.00051Google Scholar
Digital Library
- Youcheng Sun, Min Wu, Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska, and Daniel Kroening. 2018. Concolic testing for deep neural networks. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 109–119. DOI:https://doi.org/10.1145/3238147.3238172Google Scholar
Digital Library
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. Retrieved from https://arXiv:1312.6199.Google Scholar
- Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). ACM, New York, NY, 303–314. DOI:https://doi.org/10.1145/3180155.3180220Google Scholar
Digital Library
- Cumhur Erkan Tuncali, James Kapinski, Hisahiro Ito, and Jyotirmoy V. Deshmukh. 2018. Reasoning about safety of learning-enabled components in autonomous cyber-physical systems. In Proceedings of the 55th Annual Design Automation Conference (DAC’18). ACM, New York, NY, Article 30, 6 pages. DOI:https://doi.org/10.1145/3195970.3199852Google Scholar
- U.S. Department of Defense. 2016. Report of the Defense Science Board Summer Study on Autonomy. Technical Report. DoD.Google Scholar
- U.S. National Transportation Safety Board. 2017. Accident Report, NTSB/HAR-17/02. Technical Report PB2017-102600. NTSB.Google Scholar
- U.S. National Transportation Safety Board. 2018. Preliminary Report, Highway. Technical Report HWY18FH011. NTSB.Google Scholar
- U.S. National Transportation Safety Board. 2019. Vehicle Automation Report, Highway. Technical Report HWY18MH010. NTSB.Google Scholar
- Ronald E. Walpole, Raymond H. Myers, Sharon L. Myers, and Keying Ye. 2012. Probability & Statistics For Engineers & Scientists (9th ed.). Prentice Hall, Boston, MA.Google Scholar
- Zongben Xu and Jian Sun. 2017. Model-driven deep-learning. Natl. Sci. Rev. 5, 1 (08 2017), 22–24. DOI:https://doi.org/10.1093/nsr/nwx099Google Scholar
- Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. 2015. Understanding neural networks through deep visualization. Retrieved from https://arXiv:1506.06579.Google Scholar
- Fuxun Yu, Zhuwei Qin, Chenchen Liu, Liang Zhao, Yanzhi Wang, and Xiang Chen. 2019. Interpreting and evaluating neural network robustness. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI’19). ijcai.org, 4199–4205. DOI:https://doi.org/10.24963/ijcai.2019/583Google Scholar
Digital Library
- Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. Retrieved from https://arXiv:1605.07146.Google Scholar
- Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 132–142. DOI:https://doi.org/10.1145/3238147.3238187Google Scholar
Digital Library
- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). IEEE, Piscataway, NJ, 2242–2251.Google Scholar
Index Terms
Enki: A Diversity-driven Approach to Test and Train Robust Learning-enabled Systems
Recommendations
Validating Code-Level Behavior of Dynamic Adaptive Systems in the Face of Uncertainty
SSBSE 2013: Proceedings of the 5th International Symposium on Search Based Software Engineering - Volume 8084A dynamically adaptive system DAS self-reconfigures at run time in order to handle adverse combinations of system and environmental conditions. Techniques are needed to make DASs more resilient to system and environmental uncertainty. Furthermore, ...
A mathematical model for designing a network of sustainable medical waste management under uncertainty
Highlights- Framing a sustainability-based network of medical waste management under uncertainty.
AbstractMedical waste generation is on the rise in most societies due to population growth, ease of access to healthcare, and wide-ranging healthcare services. Given this concern, it would be highly important to manage medical waste generated ...
Making Decisions in a Sustainable Development Context: A State-of-the-Art Survey and Proposal of a Multi-period Single Synthesizing Criterion Approach
Making sustainable decisions is a major concern for government departments seeking to develop best practices and innovative methods to deal with complex decision-making problems. This article is concerned with decision-making in sustainable development ...






Comments