Modeling dynamic systems incurring stochastic disturbances for deriving a control policy is a ubiquitous task in engineering. However, in some instances obtaining a model of a system may be impractical or impossible. Alternative approaches have been developed using a simulation-based stochastic framework, in which the system interacts with its environment in real time and obtains information that can be processed to produce an optimal control policy. In this context, the problem of developing a policy for controlling the system’s behavior is formulated as a sequential decision-making problem under uncertainty. This paper considers the problem of deriving a control policy for a dynamic system with unknown dynamics in real time, formulated as a sequential decision-making under uncertainty. The evolution of the system is modeled as a controlled Markov chain. A new state-space representation model and a learning mechanism are proposed that can be used to improve system performance over time. The major difference between the existing methods and the proposed learning model is that the latter utilizes an evaluation function, which considers the expected cost that can be achieved by state transitions forward in time. The model allows decision-making based on gradually enhanced knowledge of system response as it transitions from one state to another, in conjunction with actions taken at each state. The proposed model is demonstrated on the single cart-pole balancing problem and a vehicle cruise-control problem.
Skip Nav Destination
Article navigation
July 2009
Research Papers
A Real-Time Computational Learning Model for Sequential Decision-Making Problems Under Uncertainty
Andreas A. Malikopoulos,
Andreas A. Malikopoulos
Department of Mechanical Engineering,
amalik@umich.edu
University of Michigan
, Ann Arbor, MI 48109
Search for other works by this author on:
Panos Y. Papalambros,
Panos Y. Papalambros
Department of Mechanical Engineering,
pyp@umich.edu
University of Michigan
, Ann Arbor, MI 48109
Search for other works by this author on:
Dennis N. Assanis
Dennis N. Assanis
Department of Mechanical Engineering,
assanis@umich.edu
University of Michigan
, Ann Arbor, MI 48109
Search for other works by this author on:
Andreas A. Malikopoulos
Panos Y. Papalambros
Dennis N. Assanis
J. Dyn. Sys., Meas., Control. Jul 2009, 131(4): 041010 (8 pages)
Published Online: May 20, 2009
Article history
Received:
March 18, 2008
Revised:
February 4, 2009
Published:
May 20, 2009
Citation
Malikopoulos, A. A., Papalambros, P. Y., and Assanis, D. N. (May 20, 2009). "A Real-Time Computational Learning Model for Sequential Decision-Making Problems Under Uncertainty." ASME. J. Dyn. Sys., Meas., Control. July 2009; 131(4): 041010. https://doi.org/10.1115/1.3117200
Download citation file:
Get Email Alerts
Cited By
Robust Periodical Tracking for Fast Tool Servo Systems With Selective Disturbance Compensation
J. Dyn. Sys., Meas., Control (August 2022)
Feasibility of a Wearable Cold-Gas Thruster for Fall Prevention
J. Dyn. Sys., Meas., Control (August 2022)
Comparing Three Different Decoupling Control Approaches for Roll-To-Roll Printing Systems
J. Dyn. Sys., Meas., Control
Saturated Output Feedback Control for Robot Manipulators with Joints of Arbitrary Flexibility
J. Dyn. Sys., Meas., Control
Related Articles
Probabilistic Control for Uncertain Systems
J. Dyn. Sys., Meas., Control (March,2012)
Analytical Target Setting: An Enterprise Context in Optimal Product Design
J. Mech. Des (January,2006)
Convergence Properties of a Computational Learning Model for Unknown Markov Chains
J. Dyn. Sys., Meas., Control (July,2009)
Output–Feedback Regulation of the Contact-Force in High-Speed Train Pantographs
J. Dyn. Sys., Meas., Control (March,2004)
Related Proceedings Papers
Related Chapters
Decision Making in Two-Dimensional Warranty Planning (PSAM-0186)
Proceedings of the Eighth International Conference on Probabilistic Safety Assessment & Management (PSAM)
Decision Making Methods Evaluating Uncertainty in Risk Assessment Analysis of Complex Techical Systems (PSAM-0011)
Proceedings of the Eighth International Conference on Probabilistic Safety Assessment & Management (PSAM)
An Bayesian Assessment Model for Equipment Techonlogy State
International Conference on Software Technology and Engineering (ICSTE 2012)