As we continue to model more complex systems, the validation of dynamical responses has come to the forefront of modeling and simulation. One form of dynamic response is when the output is a function of time. The proper evaluation of functional data over an array of desired input parameters is critical to achieving a robust validation assessment of a simulation model. We extend the correlation analysis (CORA) objective rating system to validate functional data across experimental regions. Functional regression analysis is used to generate surrogate estimations of the system response functions at points within the region where experimental observations are absent. These CORA scores provide a measure of disagreement at each desired parameter configuration. An overall score for model validity is achieved using a weighted linear combination of the individual CORA scores. Finally, an improved CORA size scoring metric is introduced.
Issue Section:
Research Papers
References
1.
ISO,
2014
, “Road Vehicles—Objective Rating Metric for Non-Ambiguous Signals,” International Organization for Standardization
, Geneva, Switzerland
, Standard No. ISO/TS 18571:2014
.https://www.iso.org/standard/62937.html2.
Storm
, S.
, Hill
, R. R.
, and Pignatiello
, J. J.
, 2013
, “A Response Surface Methodology for Modeling Time Series Response Data
,” Qual. Reliab. Eng. Int.
, 29
(5
), pp. 771
–778
.3.
Balci
, O.
, 1994
, “Validation, Verification, and Testing Techniques Throughout the Life Cycle of a Simulation Study
,” Ann. Oper. Res.
, 53
(1
), pp. 121
–173
.4.
Davis
, P. K.
, 1992
, “Generalizing Concepts and Methods of Verification, Validation, and Accreditation (VV&A) for Military Simulations,” RAND Corporation, Santa Monica, CA, Technical Report No. RAND/R-4249-ACQ
.https://www.rand.org/pubs/reports/R4249.html5.
Sargent
, R. G.
, 2010
, “Verification and Validation of Simulation Models
,” IEEE
Winter Simulation Conference
, Washington, DC, Dec. 9–12, pp. 166
–183
.6.
Law
, A. M.
, 2007
, Simulation Modeling and Analysis
, 4th ed., McGraw-Hill
, New York
.7.
Shannon
, R. E.
, 1975
, Systems Simulation: The Art and Science
, Prentice Hall
, Englewood Cliffs, NJ
.8.
Balci
, O.
, and Sargent
, R. G.
, 1981
, “A Methodology for Cost-Risk Analysis in the Statistical Validation of Simulation Models
,” Commun. ACM
, 24
(4
), pp. 190
–197
.9.
Naylor
, T. H.
, and Finger
, J. M.
, 1967
, “Verification of Computer Simulation Models
,” Manage. Sci.
, 14
(2
), pp. B92
–B101
.10.
Garratt
, M.
, 1974
, “Statistical Validation of Simulation Models
,” Summer Computer Simulation Conference
, Houston, TX, pp. 915
–926
.11.
Fishman
, G. S.
, and Kiviat
, P. J.
, 1967
, “The Analysis of Simulation-Generated Time Series
,” Manage. Sci.
, 13
(7
), pp. 525
–557
.12.
Gallant
, A.
, Gerig
, T. M.
, and Evans
, J.
, 1974
, “Time Series Realizations Obtained According to an Experimental Design
,” J. Am. Stat. Assoc.
, 69
(347
), pp. 639
–645
.13.
Sarin
, H.
, Kokkolaras
, M.
, Hulbert
, G.
, Papalambros
, P.
, Barbat
, S.
, and Yang
, R.-J.
, 2010
, “Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics
,” ASME J. Dyn. Syst. Meas. Control
, 132
(6
), p. 061401
.14.
Van Horn
, R. L.
, 1971
, “Validation of Simulation Results
,” Manage. Sci.
, 17
(5
), pp. 247
–258
.15.
Balci
, O.
, 1998
, “Verification, Validation, and Testing
,” Handbook of Simulation
, Vol. 10
, J.
Banks
, ed., Wiley
, New York
, pp. 335
–393
.16.
Hills
, R. G.
, and Trucano
, T. G.
, 1999
, “Statistical Validation of Engineering and Scientific Models: Background,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND99-1256
.https://inis.iaea.org/search/search.aspx?orig_q=RN:3004712217.
Buranathiti
, T.
, Cao
, J.
, Chen
, W.
, Baghdasaryan
, L.
, and Xia
, Z. C.
, 2006
, “Approaches for Model Validation: Methodology and Illustration on a Sheet Metal Flanging Process
,” ASME J. Manuf. Sci. Eng.
, 128
(2
), pp. 588
–597
.18.
Chen
, W.
, Baghdasaryan
, L.
, Buranathiti
, T.
, and Cao
, J.
, 2004
, “Model Validation Via Uncertainty Propagation and Data Transformations
,” AIAA J.
, 42
(7
), pp. 1406
–1415
.19.
Ghanem
, R. G.
, Doostan
, A.
, and Red-Horse
, J.
, 2008
, “A Probabilistic Construction of Model Validation
,” Comput. Methods Appl. Mech. Eng.
, 197
(29–32
), pp. 2585
–2595
.20.
Hills
, R. G.
, 2006
, “Model Validation: Model Parameter and Measurement Uncertainty
,” ASME J. Heat Transfer
, 128
(4
), pp. 339
–351
.21.
Rebba
, R.
, and Mahadevan
, S.
, 2008
, “Computational Methods for Model Reliability Assessment
,” Reliab. Eng. Syst. Saf.
, 93
(8
), pp. 1197
–1207
.22.
Oberkampf
, W. L.
, and Barone
, M. F.
, 2006
, “Measures of Agreement Between Computation and Experiment: Validation Metrics
,” J. Comput. Phys.
, 217
(1
), pp. 5
–36
.23.
Ferson
, S.
, Oberkampf
, W. L.
, and Ginzburg
, L.
, 2008
, “Model Validation and Predictive Capability for the Thermal Challenge Problem
,” Comput. Methods Appl. Mech. Eng.
, 197
(29–32
), pp. 2408
–2430
.24.
Chen
, W.
, Xiong
, Y.
, Tsui
, K.-L.
, and Wang
, S.
, 2008
, “A Design-Driven Validation Approach Using Bayesian Prediction Models
,” ASME J. Mech. Des.
, 130
(2
), p. 021101
.25.
Jiang
, X.
, and Mahadevan
, S.
, 2008
, “Bayesian Wavelet Method for Multivariate Model Assessment of Dynamic Systems
,” J. Sound Vib.
, 312
(4–5
), pp. 694
–712
.26.
Jiang
, X.
, and Mahadevan
, S.
, 2008
, “Bayesian Validation Assessment of Multivariate Computational Models
,” J. Appl. Stat.
, 35
(1
), pp. 49
–65
.27.
Rebba
, R.
, Huang
, S.
, Liu
, Y.
, and Mahadevan
, S.
, 2005
, “Statistical Validation of Simulation Models
,” Int. J. Mater. Prod. Technol.
, 25
(1–3
), pp. 164
–181
.28.
Rebba
, R.
, and Mahadevan
, S.
, 2006
, “Model Predictive Capability Assessment Under Uncertainty
,” AIAA J.
, 44
(10
), pp. 2376
–2384
.29.
Rebba
, R.
, Mahadevan
, S.
, and Huang
, S.
, 2006
, “Validation and Error Estimation of Computational Models
,” Reliab. Eng. Syst. Saf.
, 91
(10–11
), pp. 1390
–1397
.30.
Rebba
, R.
, and Mahadevan
, S.
, 2006
, “Validation of Models With Multivariate Output
,” Reliab. Eng. Syst. Saf.
, 91
(8
), pp. 861
–871
.31.
Wang
, S.
, Chen
, W.
, and Tsui
, K.-L.
, 2009
, “Bayesian Validation of Computer Models
,” Technometrics
, 51
(4
), pp. 439
–451
.32.
Kleijnen
, J. P.
, 1992
, “Regression Metamodels for Simulation With Common Random Numbers: Comparison of Validation Tests and Confidence Intervals
,” Manage. Sci.
, 38
(8
), pp. 1164
–1185
.33.
Kleijnen
, J. P.
, 1995
, “Verification and Validation of Simulation Models
,” Eur. J. Oper. Res.
, 82
(1
), pp. 145
–162
.34.
Kleijnen
, J. P.
, 1995
, “Sensitivity Analysis and Optimization of System Dynamics Models: Regression Analysis and Statistical Design of Experiments
,” Syst. Dyn. Rev.
, 11
(4
), pp. 275
–288
.35.
Kleijnen
, J. P.
, Feelders
, A. J.
, and Cheng
, R. C.
, 1998
, “Bootstrapping and Validation of Metamodels in Simulation
,” IEEE
Winter Simulation Conference
, Washington, DC, Dec. 13–16, pp. 701
–706
.36.
Kleijnen
, J. P.
, and Sargent
, R. G.
, 2000
, “A Methodology for Fitting and Validating Metamodels in Simulation
,” Eur. J. Oper. Res.
, 120
(1
), pp. 14
–29
.37.
Kleijnen
, J. P.
, and Deflandre
, D.
, 2006
, “Validation of Regression Metamodels in Simulation: Bootstrap Approach
,” Eur. J. Oper. Res.
, 170
(1
), pp. 120
–131
.38.
Bendat
, J. S.
, and Piersol
, A. G.
, 1980
, Engineering Applications of Correlation and Spectral Analysis
, Vol. 315
, Wiley-Interscience
, New York
, p. 1
.39.
Jenkins
, G. M.
, 1961
, “General Considerations in the Analysis of Spectra
,” Technometrics
, 3
(2
), pp. 133
–166
.40.
Donelly
, B.
, Morgan
, R. M.
, and Eppinger
, R. H.
, 1983
, “Durability, Repeatability and Reproducibility of the NHTSA Side Impact Dummy
,” SAE
Paper No. 831624.41.
Russell
, D. M.
, 1997
, “Error Measures for Comparing Transient Data—Part I: Development of a Comprehensive Error Measure
,” 68th Shock and Vibration Symposium
, Hunt Valley, MD, Nov. 3–6, pp. 175
–184
.http://roadsafellc.com/NCHRP22-24/Literature/Papers/Metrics/Russell-ErrorMeasures.pdf42.
Sprague
, M.
, and Geers
, T.
, 2004
, “A Spectral-Element Method for Modeling Cavitation in Transient Fluid-Structure Interaction
,” Int. J. Numer. Methods Eng.
, 60
(15
), pp. 2467
–2499
.43.
Gehre
, C.
, Gades
, H.
, and Wernicke
, P.
, 2009
, “Objective Rating of Signals Using Test and Simulation Responses
,” 21st International Technical Conference on the Enhanced Safety of Vehicles Conference
(ESV
), Stuttgart, Germany, June 15–18, pp. 15
–18
.44.
Zhan
, Z.
, Fu
, Y.
, and Yang
, R.-J.
, 2011
, “Enhanced Error Assessment of Response Time Histories (EEARTH) Metric and Calibration Process,” SAE
Paper No. 2011-01-0245.45.
Jiang
, X.
, Yang
, R.-J.
, Barbat
, S.
, and Weerappuli
, P.
, 2009
, “Bayesian Probabilistic PCA Approach for Model Validation of Dynamic Systems
,” SAE Int. J. Mater. Manuf.
, 2
(1
), pp. 555
–563
.46.
Lamb
, D.
, Castanier
, M.
, Pan
, H.
, Kokkolaras
, M.
, and Hulbert
, G.
, 2012
, “Model Validation for Simulations of Vehicle Systems,” Ford Motor Company, Dearborn, MI, DTIC Document No. A566158
.http://www.dtic.mil/docs/citations/ADA56603747.
Zhan
, Z.
, Fu
, Y.
, Yang
, R.-J.
, and Peng
, Y.
, 2011
, “An Enhanced Bayesian Based Model Validation Method for Dynamic Systems
,” ASME J. Mech. Des.
, 133
(4
), p. 041005
.48.
Jiang
, X.
, and Mahadevan
, S.
, 2011
, “Wavelet Spectrum Analysis Approach to Model Validation of Dynamic Systems
,” Mech. Syst. Signal Process.
, 25
(2
), pp. 575
–590
.49.
Cheng
, Z.
, Pellettiere
, J. A.
, and Wright
, N. L.
, 2006
, “Wavelet-Based Test-Simulation Correlation Analysis for the Validation of Biodynamical Modeling
,” Conference and Exposition on Structural Dynamics
, St Louis, MO, Jan. 30–Feb. 2, pp. 2124–2132.https://pdfs.semanticscholar.org/ef0c/bc3e2f9f09e0a87a6414ffe36a4a93216f60.pdf50.
Morris
, J. S.
, 2015
, “Functional Regression
,” Annu. Rev. Stat. Appl.
, 2
(1
), pp. 321
–359
.51.
Faraway
, J. J.
, 1997
, “Regression Analysis for a Functional Response
,” Technometrics
, 39
(3
), pp. 254
–261
.52.
Faraway
, J. J.
, 2000
, “Modeling Reach Motions Using Functional Regression Analysis,” SAE
Paper No. 2000-01-2175.53.
Untaroiu
, C.
, Shin
, J.
, and Lu
, Y.-C.
, 2013
, “Assessment of a Dummy Model in Crash Simulations Using Rating Methods
,” Int. J. Autom. Technol.
, 14
(3
), pp. 395
–405
.54.
Thunert
, C.
, 2012
, “Cora Release 3.6 User's Manual,” Partnership for Dummy Technology and Biomechanics
, Ingolstadt, Germany.55.
Montgomery
, D. C.
, 2009
, Design and Analysis of Experiments
, 7th ed., Wiley
, Hoboken, NJ
.56.
Myers
, R. H.
, Montgomery
, D. C.
, and Anderson-Cook
, C. M.
, 2009
, Response Surface Methodology: Process and Product Optimization Using Designed Experiments
, 3rd ed., Wiley
, Hoboken, NJ
.57.
Gehre
, C.
, and Stahlschmidt
, S.
, 2011
, “Assessment of Dummy Models by Using Objective Rating Methods
,” 22nd International Technical Conference on the Enhanced Safety of Vehicles
, Washington, DC, June 13–16.Copyright © 2017 by American Society of Mechanical Engineers
You do not currently have access to this content.
