research-article

Simulating a logistics enterprise using an asymmetrical wargame simulation with soar reinforcement learning and coevolutionary algorithms

Authors:

Nate Derbinsky,

Una-May O'ReillyAuthors Info & Claims

GECCO '21: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Pages 1907 - 1915

https://doi.org/10.1145/3449726.3463172

Published: 08 July 2021 Publication History

Abstract

We demonstrate an innovative framework (CoEvSoarRL) that leverages machine learning algorithms to optimize and simulate a resilient and agile logistics enterprise to improve the readiness and sustainment, as well as reduce the operational risk. The CoEvSoarRL is an asymmetrical wargame simulation that leverages reinforcement learning and coevolutionary algorithms to improve the functions of a total logistics enterprise value chain. We address two of the key challenges: (1) the need to apply holistic prediction, optimization, and wargame simulation to improve the total logistics enterprise readiness; (2) the uncertainty and lack of data which require large-scale systematic what-if scenarios and analysis of alternatives to simulate potential new and unknown situations. Our CoEvSoarRL learns a model of a logistic enterprise environment from historical data with Soar reinforcement learning. Then the Soar model is used to evaluate new decisions and operating conditions. We simulate the logistics enterprise vulnerability (risk) and evolve new and more difficult operating conditions (tests); meanwhile we also coevolve better logistics enterprise decision (solutions) to counter the tests. We present proof-of-concept results from a US Marine Corps maintenance and supply chain data set.

References

[1]

Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. 2017. Generalization and equilibrium in generative adversarial nets (gans). In International Conference on Machine Learning. PMLR, 224--232.

[2]

Thomas Back. 1996. Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford university press.

[3]

Noam Brown and Tuomas Sandholm. 2017. Safe and nested subgame solving for imperfect-information games. arXiv preprint arXiv:1705.02955 (2017).

[4]

David E Goldberg. 1989. Genetic algorithms in search. Optimization, and MachineLearning (1989).

Digital Library

[5]

Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014).

[6]

Krzysztof Krawiec and Malcolm Heywood. 2019. Solving complex problems with coevolutionary algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 975--1001.

Digital Library

[7]

John E Laird. 2019. The Soar cognitive architecture. MIT press.

Digital Library

[8]

John Loch and Satinder P Singh. 1998. Using Eligibility Traces to Find the Best Memoryless Policy in Partially Observable Markov Decision Processes. In ICML. 323--331.

[9]

John F Nash et al. 1950. Equilibrium points in n-person games. Proceedings of the national academy of sciences 36, 1 (1950), 48--49.

[10]

Una-May O'Reilly, Jamal Toutouh, Marcos Pertierra, Daniel Prado Sanchez, Dennis Garcia, Anthony Erb Luogo, Jonathan Kelly, and Erik Hemberg. 2020. Adversarial genetic programming for cyber security: A rising application domain where GP matters. Genetic Programming and Evolvable Machines 21, 1 (2020), 219--250.

Digital Library

[11]

Judea Pearl and Dana Mackenzie. 2018. The book of why: the new science of cause and effect. Basic books.

[12]

Elena Popovici, Anthony Bucci, R Paul Wiegand, and Edwin D De Jong. 2012. Coevolutionary Principles.

[13]

Lawrence R Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 2 (1989), 257--286.

[14]

Eric Rasmusen. 1989. Games and information: An introduction to game theory.

[15]

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017. Mastering the game of go without human knowledge. nature 550, 7676 (2017), 354--359.

[16]

Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.

Digital Library

[17]

Jamal Toutouh, Erik Hemberg, and Una-May O'Reilly. 2019. Spatial evolutionary generative adversarial networks. In Proceedings of the Genetic and Evolutionary Computation Conference. 472--480.

Digital Library

[18]

Ying Zhao, Nate Derbinsky, Lydia Y Wong, Jonah Sonnenshein, and Tony Kendall. 2018. Continual and Real-time Learning for Modeling Combat Identification in a Tactical Environment. In Proceedings of the NIPS 2018 Workshop on Continual Learning.

[19]

Eric A Zilli and Michael E Hasselmo. 2008. The influence of Markov decision process structure on the possible strategic use of working memory and episodic memory. PloS one 3, 7 (2008), e2756.

Cited By

Yu SZhu WWang Y(2023)Research on Wargame Decision-Making Method Based on Multi-Agent Deep Deterministic Policy GradientApplied Sciences10.3390/app1307456913:7(4569)Online publication date: 4-Apr-2023
https://doi.org/10.3390/app13074569
Liu YTao XLi XColombo AHu S(2023)Artificial Intelligence in Smart Logistics Cyber-Physical Systems: State-of-The-Arts and Potential ApplicationsIEEE Transactions on Industrial Cyber-Physical Systems10.1109/TICPS.2023.32832301(1-20)Online publication date: 2023
https://doi.org/10.1109/TICPS.2023.3283230
Rolf BJackson IMüller MLang SReggelin TIvanov D(2022)A review on reinforcement learning algorithms and applications in supply chain managementInternational Journal of Production Research10.1080/00207543.2022.214022161:20(7151-7179)Online publication date: 3-Nov-2022
https://doi.org/10.1080/00207543.2022.2140221
Show More Cited By

Index Terms

Simulating a logistics enterprise using an asymmetrical wargame simulation with soar reinforcement learning and coevolutionary algorithms
1. Security and privacy
  1. Security services
    1. Access control
  2. Systems security
    1. Vulnerability management
2. Theory of computation
  1. Design and analysis of algorithms
    1. Mathematical optimization
      1. Discrete optimization
        Optimization with randomized search heuristics
        Evolutionary algorithms

Recommendations

Using domain knowledge in coevolution and reinforcement learning to simulate a logistics enterprise
GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference Companion

We demonstrate a framework (CoEv-Soar-RL) for a logistics enterprise to improve readiness, sustainment, and reduce operational risk. The CoEv-Soar-RL uses reinforcement learning and coevolutionary algorithms to improve the functions of a logistics ...
Reinforcement learning algorithms: A brief survey
Highlights
- RL can be used to solve problems involving sequential decision-making.
- RL is based on trial-and-error learning through rewards and punishments.
- The ultimate goal of an RL agent is to maximize cumulative reward.
- RL agent tries ...
Abstract
Reinforcement Learning (RL) is a machine learning (ML) technique to learn sequential decision-making in complex problems. RL is inspired by trial-and-error based human/animal learning. It can learn an optimal policy autonomously with knowledge ...
Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems

Recent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '21: Proceedings of the Genetic and Evolutionary Computation Conference Companion

July 2021

2047 pages

ISBN:9781450383516

DOI:10.1145/3449726

Editor:
Francisco Chicano
University of Malaga
,
General Chair:
Krzysztof Krawiec
Poznan University of Technology

Copyright © 2021 ACM.

© 2021 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the United States Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO '21

Sponsor:

SIGEVO

GECCO '21: Genetic and Evolutionary Computation Conference

July 10 - 14, 2021

Lille, France

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
169
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)3

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yu SZhu WWang Y(2023)Research on Wargame Decision-Making Method Based on Multi-Agent Deep Deterministic Policy GradientApplied Sciences10.3390/app1307456913:7(4569)Online publication date: 4-Apr-2023
https://doi.org/10.3390/app13074569
Liu YTao XLi XColombo AHu S(2023)Artificial Intelligence in Smart Logistics Cyber-Physical Systems: State-of-The-Arts and Potential ApplicationsIEEE Transactions on Industrial Cyber-Physical Systems10.1109/TICPS.2023.32832301(1-20)Online publication date: 2023
https://doi.org/10.1109/TICPS.2023.3283230
Rolf BJackson IMüller MLang SReggelin TIvanov D(2022)A review on reinforcement learning algorithms and applications in supply chain managementInternational Journal of Production Research10.1080/00207543.2022.214022161:20(7151-7179)Online publication date: 3-Nov-2022
https://doi.org/10.1080/00207543.2022.2140221
Kegenbekov ZJackson I(2021)Adaptive Supply Chain: Demand–Supply Synchronization Using Deep Reinforcement LearningAlgorithms10.3390/a1408024014:8(240)Online publication date: 15-Aug-2021
https://doi.org/10.3390/a14080240

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents