Abstract

Mission design is a challenging problem, requiring designers to consider complex design spaces and dynamically evolving mission environments. In this paper, we adapt computational design approaches, widely used by the engineering design community, to address unique challenges associated with mission design. We present a framework to enable efficient mission design by efficiently building a surrogate model of the mission simulation environment to assist with design tasks. This framework combines design of experiments (DOEs) techniques for data collection, meta-modeling with machine learning models, and uncertainty quantification (UQ) and explainable AI (XAI) techniques to validate the model and explore the mission design space. We demonstrate this framework using an open-source real-time strategy (RTS) game called microRTS as our mission environment. The objective considered in this use case is game balance, observed through the probability of each player winning. Mission parameters are varied according to a DOE over chosen player bots and possible initial conditions of the microRTS game. A neural network model is then trained based on gameplay data obtained from the specified experiments to predict the probability of a player winning given any game state. The model confidence is evaluated using Monte Carlo Dropout Networks (MCDN), and an explanation model is built using SHapley Additive exPlanations (SHAP). Design changes to a sample game are introduced based on important features of the game identified by SHAP analysis. Results show that this analysis can successfully capture feature importance and uncertainty in predictions to guide additional data collection for mission design exploration.

References

1.
Zimmerman
,
P.
, and
Dahmann
,
J.
,
2018
, “
Digital Engineering to Support Mission Engineering
,”
21st Annual National Defense Industrial Association Systems and Mission Engineering Conference
,
Tampa, FL
,
Oct. 24
, p.
22
.
2.
West
,
C. T. D.
, and
Birkmire
,
B.
,
2020
, “
AFSIM
,”
CSIAC
,
7
(
3
), p.
50
.
3.
Hanlon
,
N.
,
Garcia
,
E.
,
Casbeer
,
D.
, and
Pachter
,
M.
,
2018
, “
AFSIM Implementation and Simulation of the Active Target Defense Differential Game
,”
2018 AIAA Guidance, Navigation, and Control Conference
,
Kissimmee, FL
,
Jan. 8–12
, p.
1582
.
4.
Soleyman
,
S.
, and
Khosla
,
D.
,
2020
, “
Multi-Agent Mission Planning with Reinforcement Learning
,”
AAAI Symposium on the 2nd Workshop on Deep Models and Artificial Intelligence for Defense Applications: Potentials, Theories, Practices, Tools, and Risks
,
Virtual
,
Nov. 11–12
.
5.
Bunyard
,
J.
,
2021
, “
Virtual Training: Preparing Future Naval Officers for 21st Century Warfare
”.
Center for International Maritime Security
.
6.
Doyle
,
S.
,
2020
, “
Gaming-Defence. Game of Duty [wargaming]
,”
Eng. Technol.
,
15
(
2
), pp.
24
27
.
7.
Goodman
,
J.
,
Risi
,
S.
, and
Lucas
,
S.
,
2020
, “
AI and Wargaming
,”
ArXiV, Preprint
.
8.
Raina
,
A.
,
McComb
,
C.
, and
Cagan
,
J.
,
2019
, “
Learning to Design From Humans: Imitating Human Designers Through Deep Learning
,”
ASME J. Mech. Des.
,
141
(
11
), p.
111102
.
9.
Bayrak
,
A. E.
, and
Sha
,
Z.
,
2021
, “
Integrating Sequence Learning and Game Theory to Predict Design Decisions Under Competition
,”
ASME J. Mech. Des.
,
143
(
5
), p.
051401
.
10.
Panchal
,
J. H.
,
Sha
,
Z.
, and
Kannan
,
K. N.
,
2017
, “
Understanding Design Decisions Under Competition Using Games With Information Acquisition and a Behavioral Experiment
,”
ASME J. Mech. Des.
,
139
(
9
), p.
091402
.
11.
Ren
,
Y.
,
Bayrak
,
A. E.
, and
Papalambros
,
P. Y.
,
2016
, “
EcoRacer: Game-Based Optimal Electric Vehicle Design and Driver Control Using Human Players
,”
ASME J. Mech. Des.
,
138
(
6
), p.
061407
.
12.
Jaffe
,
A.
,
Miller
,
A.
,
Andersen
,
E.
,
Liu
,
Y.-E.
,
Karlin
,
A.
, and
Popovic
,
Z.
,
2012
, “
Evaluating Competitive Game Balance with Restricted Play
,”
Eighth Artificial Intelligence and Interactive Digital Entertainment Conference
,
Stanford, Palo Alto, CA
,
Oct. 8–12
.
13.
Tomašev
,
N.
,
Paquet
,
U.
,
Hassabis
,
D.
, and
Kramnik
,
V.
,
2020
, “
Assessing Game Balance with Alphazero: Exploring Alternative Rule Sets in Chess
,”
ArXiV, Preprint
.
14.
Jaffe
,
A. B.
,
2013
,
“Understanding Game Balance with Quantitative Methods,” Ph.D. thesis, University of Washington, Seattle, WA
.
15.
Beyer
,
M.
,
Agureikin
,
A.
,
Anokhin
,
A.
,
Laenger
,
C.
,
Nolte
,
F.
,
Winterberg
,
J.
,
Renka
,
M.
,
Rieger
,
M.
,
Pflanzl
,
N.
,
Preuss
,
M.
, and
Volz
,
V.
,
2016
, “
An Integrated Process for Game Balancing
,”
2016 IEEE Conference on Computational Intelligence and Games (CIG)
,
Santorini, Greece
,
Sept. 20–23
, pp.
1
8
.
16.
Leigh
,
R.
,
Schonfeld
,
J.
, and
Louis
,
S. J.
,
2008
, “
Using Coevolution to Understand and Validate Game Balance in Continuous Games
,”
Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation, GECCO ’08
,
Atlanta, GA
,
July 12
, pp.
1563
1570
.
17.
Xia
,
W.
, and
Anand
,
B.
,
2016
, “
Game Balancing with Ecosystem Mechanism
,”
In 2016 International Conference on Data Mining and Advanced Computing (SAPIENCE)
,
Ernakulam, India
,
Mar. 16–18
, pp.
317
324
.
18.
Ontañón
,
S.
,
2020
,
microRTS GitHub Code
, https://github.com/santiontanon/microrts
19.
Ontañón
,
S.
, and
Buro
,
M.
,
2015
, “
Adversarial Hierarchical-Task Network Planning for Complex Real-Time Games
,”
Proceedings of the 24th International Conference on Artificial Intelligence
,
Buenos Aires, Argentina
,
July 25–31
, pp.
1652
1658
.
20.
Ontañón
,
S.
,
2016
, “
Experiments on Learning Unit-Action Models From Replay Data From RTS Games
,”
Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment
,
Burlingame, CA
,
Oct. 8–12
.
21.
Barriga
,
N. A.
,
Stanescu
,
M.
, and
Buro
,
M.
,
2017
, “
Game Tree Search Based on Nondeterministic Action Scripts in Real-Time Strategy Games
,”
IEEE Trans. Games
,
10
(
1
), pp.
69
77
.
22.
Stanescu
,
M.
,
Barriga
,
N. A.
,
Hess
,
A.
, and
Buro
,
M.
,
2016
, “
Evaluating Real-Time Strategy Game States Using Convolutional Neural Networks
,”
2016 IEEE Conference on Computational Intelligence and Games (CIG)
,
Santorini, Greece
,
Sept. 20–23
, IEEE, pp.
1
7
.
23.
Gunning
,
D.
,
2017
, “
Explainable Artificial Intelligence (XAI)
.”
Defense Advanced Research Projects Agency (DARPA), nd Web
,
2
(
2
).
24.
Turek
,
M.
,
2018
, “
Explainable Artificial Intelligence (xai)
”.
Defense Advanced Research Projects Agency
. http://web.archive.org/web/20190728055815/https://www.darpa.mil/program/explainable-artificial-intelligence
25.
Arrieta
,
A. B.
,
Díaz-Rodríguez
,
N.
,
Del Ser
,
J.
,
Bennetot
,
A.
,
Tabik
,
S.
,
Barbado
,
A.
,
García
,
S.
et al.,
2020
, “
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges Toward Responsible AI
,”
Inf. Fusion
,
58
, pp.
82
115
.
26.
Das
,
A.
, and
Rad
,
P.
,
2020
, “
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
,”
ArXiV, Preprint
.
27.
Janzing
,
D.
,
Minorics
,
L.
, and
Blöbaum
,
P.
,
2020
, “
Feature Relevance Quantification in Explainable AI: A Causal Problem
,”
International Conference on Artificial Intelligence and Statistics, PMLR
,
Virtual
,
Aug. 20–28
, PMLR, pp.
2907
2916
.
28.
Ribeiro
,
M. T.
,
Singh
,
S.
, and
Guestrin
,
C.
,
2016
, “
“Why Should I Trust You?” Explaining the Predictions of Any Classifier
,”
Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
,
San Francisco, CA
,
Aug. 13–17
, pp.
1135
1144
.
29.
Binder
,
A.
,
Montavon
,
G.
,
Lapuschkin
,
S.
,
Müller
,
K.-R.
, and
Samek
,
W.
,
2016
, “
Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers
,”
International Conference on Artificial Neural Networks
,
Barcelona, Spain
,
Sept. 6–9
, Springer, pp.
63
71
.
30.
Lundberg
,
S. M.
, and
Lee
,
S. I.
,
2017
, “
A Unified Approach to Interpreting Model Predictions
,”
Advances in Neural Information Processing Systems
,
Long Beach, CA
,
Dec. 4–9
, pp.
4765
4774
.
31.
Molnar
,
C.
,
2019
,
Interpretable Machine Learning
,
Lulu Press, Inc.
, Online 318. https://www.lulu.com/en/us/shop/christophmolnar/interpretable-machine-learning/paperback/product-1z92gw8n.html
32.
Joung
,
J.
, and
Kim
,
H. M.
,
2021
, “
Approach for Importance–Performance Analysis of Product Attributes From Online Reviews
,”
ASME J. Mech. Des.
,
143
(
8
), p.
081705
.
33.
Lara-Cabrera
,
R.
,
Cotta
,
C.
, and
Fernández-Leiva
,
A. J.
,
2013
, “
A Review of Computational Intelligence in RTS Games
,”
2013 IEEE Symposium on Foundations of Computational Intelligence (FOCI)
,
Singapore
,
Apr. 16–19
, IEEE, pp.
114
121
.
34.
Synnaeve
,
G.
, and
Bessiere
,
P.
,
2015
, “
Multiscale Bayesian Modeling for Rts Games: An Application to Starcraft AI
,”
IEEE Transactions on Computational Intelligence and AI in Games
,
8
(
4
), pp.
338
350
.
35.
Richoux
,
F.
,
2020
, “
Microphantom: Playing Microrts Under Uncertainty and Chaos
,”
2020 IEEE Conference on Games (CoG)
,
Virtual
,
Aug. 24–27
, IEEE, pp.
670
677
.
36.
Fernández-Ares
,
A.
,
García-Sánchez
,
P.
,
Mora
,
A. M.
,
Castillo
,
P. A.
, and
Merelo
,
J.
,
2016
, “
There Can Be Only One: Evolving RTS Bots Via Joust Selection
,”
European Conference on the Applications of Evolutionary Computation
,
Porto, Portugal
,
Mar. 30– Apr. 1
, Springer, pp.
541
557
.
37.
Liu
,
S.
,
Louis
,
S. J.
, and
Ballinger
,
C.
,
2014
, “
Evolving Effective Micro Behaviors in RTS Game
,”
2014 IEEE Conference on Computational Intelligence and Games
,
Dortmund, Germany
,
Aug. 26–29
, IEEE, pp.
1
8
.
38.
Fernández-Ares
,
A.
,
García-Sánchez
,
P.
,
Mora
,
A. M.
, and
Merelo
,
J.
,
2012
, “
Adaptive Bots for Real-Time Strategy Games Via Map Characterization
,”
2012 IEEE Conference on Computational Intelligence and Games (CIG)
,
Granada, Spain
,
Sept. 11–14
, IEEE, pp.
417
721
.
39.
Garćıa-Sánchez
,
P.
,
Tonda
,
A.
,
Mora
,
A. M.
,
Squillero
,
G.
, and
Merelo
,
J.
,
2015
, “
Towards Automatic Starcraft Strategy Generation Using Genetic Programming
,”
2015 IEEE Conference on Computational Intelligence and Games (CIG)
,
Tainan, Taiwan
,
Aug. 31–Sept. 1
, IEEE, pp.
284
291
.
40.
Vinyals
,
O.
,
Babuschkin
,
I.
,
Czarnecki
,
W. M.
,
Mathieu
,
M.
,
Dudzik
,
A.
,
Chung
,
J.
,
Choi
,
D. H.
,
Powell
,
R.
,
Ewalds
,
T.
,
Georgiev
,
P.
, and
Oh
,
J.
,
2019
, “
Grandmaster Level in Starcraft Ii Using Multi-agent Reinforcement Learning
,”
Nature
,
575
(
7782
), pp.
350
354
.
41.
Ontanón
,
S.
,
Synnaeve
,
G.
,
Uriarte
,
A.
,
Richoux
,
F.
,
Churchill
,
D.
, and
Preuss
,
M.
,
2013
, “
A Survey of Real-time Strategy Game Ai Research and Competition in Starcraft
,”
IEEE Trans. Comput. Intell. AI in Games
,
5
(
4
), pp.
293
311
.
42.
Tang
,
Z.
,
Shao
,
K.
,
Zhu
,
Y.
,
Li
,
D.
,
Zhao
,
D.
, and
Huang
,
T.
,
2018
, “
A Review of Computational Intelligence for Starcraft AI
,”
2018 IEEE Symposium Series on Computational Intelligence (SSCI)
,
Bengaluru, India
,
Nov. 18–21
, IEEE, pp.
1167
1173
.
43.
Bayrak
,
A. E.
,
McComb
,
C.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2021
, “
A Strategic Decision-Making Architecture Toward Hybrid Teams for Dynamic Competitive Problems
,”
Decision Support Syst.
,
144
, p.
113490
.
44.
Chollet
,
F.
,
2021
,
Deep learning with Python
, 2nd ed.,
Simon and Schuster
, pp.
1
504
.
45.
Kingma
,
D. P.
, and
Ba
,
J.
,
2014
, “
Adam: A Method for Stochastic Optimization
.”
arXiv preprint arXiv:1412.6980
.
46.
Gal
,
Y.
, and
Ghahramani
,
Z.
,
2016
, “
Dropout As a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
,”
International Conference on Machine Learning
,
New York City
,
June 19–24
, pp.
1050
1059
.
47.
Sicking
,
J.
,
Akila
,
M.
,
Wirtz
,
T.
,
Houben
,
S.
, and
Fischer
,
A.
,
2020
, “
Characteristics of Monte Carlo Dropout in Wide Neural Networks
.”
arXiv preprint
.
48.
Lundberg
,
S.
,
2021
, A Game Theoretic Approach to Explain the Output of Any Machine Learning Model, Jan. original-date: 2016-11-22T19:17:08Z.
49.
Erion
,
G.
,
Janizek
,
J. D.
,
Sturmfels
,
P.
,
Lundberg
,
S.
, and
Lee
,
S.-I.
,
2019
, “
Learning Explainable Models Using Attribution Priors
.”
arXiv preprint arXiv:1906.10670
.
50.
Powell
,
W.
,
2012
,
Optimal Learning
,
Wiley
,
Hoboken, NJ
.
You do not currently have access to this content.