Abstract

Engineering systems provide essential services to society, e.g., power generation, transportation. Their performance, however, is directly affected by their ability to cope with uncertainty, especially given the realities of climate change and pandemics. Standard design methods often fail to recognize uncertainty in early conceptual activities, leading to rigid systems that are vulnerable to change. Real options and flexibility in design are important paradigms to improve a system’s ability to adapt and respond to unforeseen conditions. Existing approaches to analyze flexibility, however, do not leverage sufficiently recent developments in machine learning enabling deeper exploration of the computational design space. There is untapped potential for new solutions that are not readily accessible using existing methods. Here, a novel approach to analyze flexibility is proposed based on deep reinforcement learning (DRL). It explores available datasets systematically and considers a wider range of adaptability strategies. The methodology is evaluated on an example waste-to-energy (WTE) system. Low and high flexibility DRL models are compared against stochastically optimal inflexible and flexible solutions using decision rules. The results show highly dynamic solutions, with action space parametrized via artificial neural network (ANN). They show improved expected economic value up to 69% compared with previous solutions. Combining information from action space probability distributions along expert insights and risk tolerance helps make better decisions in real-world design and system operations. Out of sample testing shows that the policies are generalizable, but subject to tradeoffs between flexibility and inherent limitations of the learning process.

References

1.
Pate
,
A.
,
2015
, “
Terrorism Trends With a Focus on Energy and Mining
,”
START—National Consortium for the Study of Terrorism and Responses to Terrorism
,
College Park, MD
,
June
.
2.
Jemielniak
,
D.
, and
Przegalinska
,
A.
,
2020
,
Collaborative Society
,
MIT Press
,
Cambridge, MA
.
3.
Haskins
,
C.
,
Forsberg
,
K.
,
Krueger
,
M.
,
Walden
,
D.
, and
Hamelin
,
D.
,
2013
, “
Systems Engineering Handbook
,”
Proceedings of the 23rd Annual International Symposium of the International Council on Systems Engineering (INCOSE 2013)
,
Philadelphia, PA
,
June 24–27
, pp.
13
16
.
4.
MacCormack
,
A.
, and
Herman
,
K.
,
2001
,
The Rise and Fall of Iridium
,
Harvard Business School
,
Cambridge, MA
.
5.
Lim
,
R.
, and
Ng
,
K.
,
2011
, “Recycling Firm IUT Global Being Wound Up,”
The Business Times
,
Marshall Cavendish Business Information
,
Singapore
.
6.
Shepard
,
W.
,
2016
, “
An Update On China's Largest Ghost City – What Ordos Kangbashi Is Like Today
,” https://www.forbes.com/sites/wadeshepard/2016/04/19/an-update-on-chinas-largest-ghost-city-what-ordos-kangbashi-is-like-today/?sh=1a21cd502327, Accessed April 23, 2021.
7.
de Neufville
,
R.
, and
Scholtes
,
S.
,
2011
,
Flexibility in Engineering Design
,
MIT Press
,
Cambridge, MA
.
8.
Trigeorgis
,
L.
,
1996
,
Real Options: Managerial Flexibility and Strategy in Resource Allocation
,
MIT Press
,
Cambridge, MA
.
9.
Cardin
,
M.-A.
,
2013
, “
An Integrated Screening Framework to Analyze Flexibility in Engineering Systems Design
,”
Proceedings of the International Conference on Engineering Design (ICED13), Design for Harmonies, Vol.9: Design Methods and Tools
,
Seoul, South Korea
,
Aug. 19–22
, pp.
135
144
.
10.
Guma
,
A.
,
Pearson
,
J.
,
Wittels
,
K.
,
de Neufville
,
R.
, and
Geltner
,
D.
,
2009
, “
Vertical Phasing as a Corporate Real Estate Strategy and Development Option
,”
J. Corp. Real Estate.
,
11
(
3
), pp.
144
157
.
11.
Cardin
,
M.-A.
,
Xie
,
Q.
,
Ng
,
T. S.
,
Wang
,
S.
, and
Hu
,
J.
,
2017
, “
An Approach for Analyzing and Managing Flexibility in Engineering Systems Design Based on Decision Rules and Multistage Stochastic Programming
,”
IISE Trans.
,
49
(
1
), pp.
1
12
.
12.
Cardin
,
M.-A.
,
Zhang
,
S.
, and
Nuttall
,
W. J.
,
2017
, “
Strategic Real Option and Flexibility Analysis for Nuclear Power Plants Considering Uncertainty in Electricity Demand and Public Acceptance
,”
Energy Econ.
,
64
(
1
), pp.
226
237
.
13.
Zhang
,
S.
, and
Cardin
,
M.-A.
,
2017
, “
Flexibility and Real Options Analysis in Emergency Medical Services Systems Using Decision Rules and Multi-Stage Stochastic Programming
,”
Transp. Res. E Logist. Transp. Rev.
,
107
, pp.
120
140
.
14.
Cardin
,
M.-A.
,
Kolfschoten
,
G. L.
,
Frey
,
D. D.
,
de Neufville
,
R.
,
De Weck
,
O. L.
, and
Geltner
,
D. M.
,
2013
, “
Empirical Evaluation of Procedures to Generate Flexibility in Engineering Systems and Improve Lifecycle Performance
,”
Res. Eng. Des.
,
24
(
3
), pp.
277
295
.
15.
Andriotis
,
C. P.
, and
Papakonstantinou
,
K. G.
,
2019
, “
Managing Engineering Systems With Large State and Action Spaces Through Deep Reinforcement Learning
,”
Reliab. Eng. Syst. Saf.
,
191
, p.
106483
.
16.
Bellman
,
R.
,
1952
, “
On the Theory of Dynamic Programming
,”
Proceedings of the National Academy of Sciences of United States of America
,
Santa Monica, CA
,
Aug. 1
, pp.
716
719
.
17.
Sethi
,
A. K.
, and
Sethi
,
S. P.
,
1990
, “
Flexibility in Manufacturing: A Survey
,”
Int. J. Flexible Manuf. Syst.
,
2
(
4
), pp.
289
328
.
18.
Linsey
,
J. S.
,
Green
,
M. G.
,
van Wie
,
M.
,
Wood
,
K. L.
, and
Stone
,
R.
,
2005
, “
Functional Representations in Conceptual Design: A First Study in Experimental Design and Evaluation
,”
Proceedings of the American Society for Engineering Education Annual Conference and Exposition
,
Fayetteville, AR
,
Sept. 14–26
, pp.
652
668
.
19.
Nilchiani
,
R.
, and
Hastings
,
D. E.
,
2007
, “
Measuring the Value of Flexibility in Space Systems: A Six-Element Framework
,”
Sys. Eng.
,
10
(
1
), pp.
26
44
.
20.
Mikaelian
,
T.
,
Nightingale
,
D. J.
,
Rhodes
,
D. H.
, and
Hastings
,
D. E.
,
2011
, “
Real Options in Enterprise Architecture: A Holistic Mapping of Mechanisms and Types for Uncertainty Management
,”
IEEE Trans. Eng. Manage.
,
54
(
3
), pp.
457
470
.
21.
Ferguson
,
S.
,
Siddiqi
,
A.
,
Lewis
,
K.
, and
de Weck
,
O. L.
,
2007
, “
Flexible and Reconfigurable Systems: Nomenclature and Review
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE
,
Las Vegas, NV
,
Sept. 4–7
, pp.
249
263
.
22.
Saleh
,
J. H.
,
Mark
,
G.
, and
Jordan
,
N. C.
,
2009
, “
Flexibility: a Multi-Disciplinary Literature Review and a Research Agenda for Designing Flexible Engineering Systems
,”
J. Eng. Des.
,
20
(
3
), pp.
307
323
.
23.
Cardin
,
M.-A.
,
2014
, “
Enabling Flexibility in Engineering Systems: A Taxonomy of Procedures and a Design Framework
,”
ASME J. Mech. Des.
,
136
(
1
), p.
011005
.
24.
Copeland
,
T.
, and
Antikarov
,
V.
,
2003
,
Real Options: A Practitioner’s Guide
,
Thomson Texere
,
New York, NY
.
25.
Cox
,
J. C.
,
Ross
,
S. A.
, and
Rubinstein
,
M.
,
1979
, “
Options Pricing: A Simplified Approach
,”
J. Financ. Econ.
,
7
(
3
), pp.
229
263
.
26.
Caunhye
,
A. M.
, and
Cardin
,
M.-A.
,
2017
, “
An Approach Based on Robust Optimization and Decision Rules for Analyzing Real Options in Engineering Systems Design
,”
IISE Trans.
,
49
(
8
), pp.
753
767
.
27.
Garstka
,
S. J.
, and
Wets
,
R. J.-B.
,
1974
, “
On Decision Rules in Stochastic Programming
,”
Math. Program.
,
7
(
1
), pp.
117
143
.
28.
Sutton
,
R. S.
, and
Barto
,
A. G.
,
2018
,
Reinforcement Learning: An Introduction
,
MIT Press
,
Cambridge, MA
.
29.
Markov
,
A. A.
,
1954
, “
The Theory of Algorithms
,”
Proc. Steklov Inst. Math.
,
42
(
1
), pp.
3
375
.
30.
Li
,
Y.
,
2017
, “
Deep Reinforcement Learning: An Overview
,”
arXiv Preprint arXiv:1701.07274
.
31.
Brockman
,
G.
,
Cheung
,
V.
,
Pettersson
,
L.
,
Schneider
,
J.
,
Schulman
,
J.
,
Tang
,
J.
, and
Zaremba
,
W.
,
2016
, “
Openai Gym
,”
arXiv Preprint arXiv:1606.01540
.
32.
Arulkumaran
,
K.
,
Deisenroth
,
M. P.
,
Brundage
,
M.
, and
Bharath
,
A. A.
,
2017
, “
Deep Reinforcement Learning: A Brief Survey
,”
IEEE Signal Process. Mag.
,
34
(
6
), pp.
26
38
.
33.
Yonekura
,
K.
, and
Hattori
,
H.
,
2019
, “
Framework for Design Optimization Using Deep Reinforcement Learning
,”
Struct. Multidiscipl. Optim.
,
60
(
4
), pp.
1709
1713
.
34.
Cui
,
H.
,
Turan
,
O.
, and
Sayer
,
P.
,
2012
, “
Learning-Based Ship Design Optimization Approach
,”
Comput.-Aided Des.
,
44
(
3
), pp.
186
195
.
35.
Zhang
,
J.
,
Wang
,
Z.
, and
Zhang
,
H.
,
2018
, “
Data-based Optimal Control of Multiagent Systems: A Reinforcement Learning Design Approach
,”
IEEE Trans. Cybern.
,
49
(
12
), pp.
4441
4449
.
36.
Baker
,
B.
,
Gupta
,
O.
,
Naik
,
N.
, and
Raskar
,
R.
,
2016
, “
Designing Neural Network Architectures Using Reinforcement Learning
,”
arXiv Preprint arXiv:1611.02167
.
37.
Van Moffaert
,
K.
,
Drugan
,
M. M.
, and
Nowé
,
A.
,
2013
, “
Scalarized Multi-Objective Reinforcement Learning: Novel Design Techniques
,”
2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)
,
Singapore
,
Apr. 16–19
, IEEE, pp.
191
199
.
38.
Lee
,
X. Y.
,
Balu
,
A.
,
Stoecklein
,
D.
,
Ganapathysubramanian
,
B.
, and
Sarkar
,
S.
,
2019
, “
A Case Study of Deep Reinforcement Learning for Engineering Design: Application to Microfluidic Devices for Flow Sculpting
,”
ASME J. Mech. Des.
,
141
(
11
), p.
111401
.
39.
Perera
,
A. T. D.
,
Wickramasinghe
,
P. U.
,
Nik
,
V. M.
, and
Scartezzini
,
J.-L.
,
2020
, “
Introducing Reinforcement Learning to the Energy System Design Process
,”
Appl. Energy
,
262
, p.
114580
.
40.
Cardin
,
M.-A.
, and
Hu
,
J.
,
2016
, “
Analyzing the Tradeoffs Between Economies of Scale, Time-Value of Money, and Flexibility in Design Under Uncertainty: Study of Centralized Versus Decentralized Waste-to-Energy Systems
,”
ASME J. Mech. Des.
,
138
(
1
), p.
011401
.
41.
Pardo
,
F.
,
Tavakoli
,
A.
,
Levdik
,
V.
, and
Kormushev
,
P.
,
2018
, “
Time Limits in Reinforcement Learning
,”
Proceedings of the International Conference on Machine Learning
,
Stockholm, Sweden
,
July 10–15
, pp.
4045
4054
.
42.
Schulman
,
J.
,
Levine
,
S.
,
Abbeel
,
P.
,
Jordan
,
M.
, and
Moritz
,
P.
,
2015
, “
Trust Region Policy Optimization
,”
Proceedings of the International Conference on Machine Learning
,
Lille, France
,
July 7–9
, PMLR, pp.
1889
1897
.
43.
Schulman
,
J.
,
Moritz
,
P.
,
Levine
,
S.
,
Jordan
,
M.
, and
Abbeel
,
P.
,
2015
, “
High-Dimensional Continuous Control Using Generalized Advantage Estimation
,”
arXiv Preprint arXiv:1506.02438
.
44.
Petsagkourakis
,
P.
,
Sandoval
,
I. O.
,
Bradford
,
E.
,
Zhang
,
D.
, and
Chanona
,
E. A. D. R.
,
2020
, “
Constrained Reinforcement Learning for Dynamic Optimization Under Uncertainty
,”
arXiv Preprint arXiv:2006.02750
.
45.
Zhang
,
C.
,
Vinyals
,
O.
,
Munos
,
R.
, and
Bengio
,
S.
,
2018
, “
A Study on Overfitting in Deep Reinforcement Learning
,”
arXiv Preprint arXiv:1804.06893
.
46.
Caunhye
,
A. M.
, and
Cardin
,
M.-A.
,
2018
, “
Towards More Resilient Integrated Power Grid Capacity Expansion: A Robust Optimization Approach with Operational Flexibility
,”
Energy Economics
,
72
, pp.
20
34
.
47.
Blume
,
S. O. P.
,
Sansavini
,
G.
, and
Cardin
,
M.-A.
,
2021
, “
Fuzzy Control-Enabled Flexible Short-Turning for Real-Time Disruption Management in Urban Transit Systems
,” Manuscript in Preparation.
48.
National Environmental Agency
,
2013
, “
Solid Waste Management
,” https://www.nea.gov.sg/our-services/waste-management/overview/, Accessed April 23, 2021.
49.
Shell
,
2014
, “
Shell Station Price Board
,” http://www.shell.com.sg/products-services/on-the-road/fuels/price-board.html, Accessed April 23, 2021.
50.
National Environment Agency
,
2011
, “
Environmental Protection Division Annual Report
,” https://www.nea.gov.sg/docs/default-source/resource/publications/environmental-protection-division-annual-report/epd-2011.pdf.
51.
IUT Global Pte Ltd
,
2006
, “
9.5 MW Food Waste Based Grid Connected Power Project Implemented by IUT Singapore Pte Ltd
,”
Clean Development Mechanism
,
IUT Global Pte. Ltd.
,
Singapore
.
52.
RIS international Ltd.
,
2005
,
Feasibility of Generating Green Power through Anaerobic Digestion of Garden Refuse from the Sacramento Area
,
RIS International Ltd
.
54.
Hu
,
J.
, and
Cardin
,
M.-A.
,
2015
, “
Generating Flexibility in the Design of Engineering Systems to Enable Better Sustainability and Lifecycle Performance
,”
Res. Eng. Des.
,
26
(
2
), pp.
121
143
.
55.
Bai
,
R.
, and
Sutanto
,
M.
,
2002
, “
The Practice and Challenges of Solid Waste Management in Singapore
,”
Waste Manage.
,
22
(
5
), pp.
557
567
.
56.
Evangelisti
,
S.
,
Lettieri
,
P.
,
Borello
,
D.
, and
Clift
,
R.
,
2014
, “
Life Cycle Assessment of Energy From Waste via Anaerobic Digestion: A UK Case Study
,”
Waste Manage.
,
34
(
1
), pp.
226
237
.
You do not currently have access to this content.