Abstract

This paper describes the process for assessing the predictive capability of the Consortium for the advanced simulation of light-water reactors (CASL) virtual environment for reactor applications code suite (VERA—CS) for different challenge problems. The assessment process is guided by the two qualitative frameworks, i.e., phenomena identification and ranking table (PIRT) and predictive capability maturity model (PCMM). The capability and credibility of VERA codes (individual and coupled simulation codes) are evaluated. Capability refers to evidence of required functionality for capturing phenomena of interest while credibility refers to the evidence that provides confidence in the calculated results. For this assessment, each challenge problem defines a set of phenomenological requirements (based on PIRT) against which the VERA software is evaluated. This approach, in turn, enables the focused assessment of only those capabilities that are relevant to the challenge problem. The credibility assessment using PCMM is based on different decision attributes that encompass verification, validation, and uncertainty quantification (VVUQ) of the CASL codes. For each attribute, a maturity score from zero to three is assigned to ascertain the acquired maturity level of the VERA codes with respect to the challenge problem. Credibility in the assessment is established by mapping relevant evidence obtained from VVUQ of codes to the corresponding PCMM attribute. The illustration of the proposed approach is presented using one of the CASL challenge problems called chalk river unidentified deposit (CRUD) induced power shift (CIPS). The assessment framework described in this paper can be considered applicable to other M & S code development efforts.

References

1.
CSAU
,
1989
, “
Quantifying Reactor Safety Margins: Application of Code Scaling Applicability and Uncertainty Evaluation Methodology to a Large-Break Loss-of-Coolant Accident
,” U.S. Nuclear Regulatory Commission, Office of Nuclear Reactor Regulation, Rockville, MD, Standard No.
NUREG/CR-5249
.https://www.nrc.gov/docs/ML0303/ML030380473.pdf
2.
EMDAP
,
2005
, “
Regulatory Guide 1.203: Transient and Accident Analysis Methods
,” U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research, Rockville, MD.
3.
Oberkampf
,
W. L.
,
Pilch
,
M.
, and
Trucano
,
T. G.
,
2007
, Predictive Capability Maturity Model for Computational Modeling and Simulation,
Sandia National Laboratories
,
Albuquerque, NM
, Stadnard No.
SAND2007-5948
.https://cfwebprod.sandia.gov/cfdocs/CompResearch/docs/Oberkampf-Pilch-Trucano-SAND2007-5948.pdf
4.
Turinsky
,
P. J.
, and
Kothe
,
D. B.
,
2016
, “
Modeling and Simulation Challenges Pursued by the Consortium for Advanced Simulation of Light Water Reactors (CASL)
,”
J. Comput. Phys.
,
313
, pp.
367
376
.10.1016/j.jcp.2016.02.043
5.
Rider
,
J.
,
Kamm
,
J. R.
,
Weirs
,
V. G.
, and
Cacui
,
D. G.
,
2010
,
SAND2010-234P: Verification, Validation and Uncertainty Quantification Workflow in CASL
,
Sandia National Laboratories
,
Albuquerque, NM
.
6.
Turner
,
J. A.
,
Clarno
,
K.
,
Sieger
,
M.
,
Bartlett
,
R.
,
Collins
,
B.
,
Pawlowski
,
R.
,
Schmidt
,
R.
, and
Summers
,
R.
,
2016
, “
The Virtual Environment for Reactor Applications (VERA): Design and Architecture
,”
J. Comput. Phys.
,
326
, pp.
544
568
.10.1016/j.jcp.2016.09.003
7.
Downar
,
T.
,
2018
, “
Update MPACT Documentation – Theory Manual, V&V Documentation
,” Consortium for Advanced Simulation of Light Water Reactors, CASL Report, Oak Ridge, TN, Report No. CASL-U-2018-1641-000.
8.
Downar
,
T.
,
Kochunas
,
B.
, and
Collins
,
B.
,
2015
, “
MPACT Verification and Validation: Status and Plans
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2015-0134-000.
9.
Downar
,
T.
,
Kochunas
,
B.
, and
Collins
,
B.
,
2017
, “
MPACT Verification and Validation Manual (Rev 2)
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2016-1199-000.
10.
Downar
,
T.
,
Kochunas
,
B.
,
Liu
,
Y.
,
Collins
,
B.
, and
Stimpson
,
S.
,
2018
, “
MPACT Verification and Validation Manual (Rev 4)
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2018-1641-000.
11.
Kochunas
,
B.
,
2019
, “
MPACT Software Test Plan, Requirements, and Test Report
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2019-1858-000.
12.
Pandya
,
T. M.
,
Davidson
,
G. G.
,
Evans
,
T. M.
, and
Johnson
,
S. R.
,
2016
, “
Shift Validation Plan for CASL
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2016-1186-000.
13.
Pilch
,
M.
,
Wang
,
J.
,
Martin
,
W.
,
Downar
,
T.
,
Kochunas
,
B.
,
Andrews
,
N.
, and
Gilkey
,
L.
,
2019
, “
MPACT Code Verification and Solution Verification
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2019-1935-000.
14.
Avramova
,
M. N.
,
2019
,
CTF 4.0 User's Manual
,
Oak Ridge National Laboratory
,
Oak Ridge, TN
.
15.
Avramova
,
M. N.
,
2016
, “
CTF User's Manual
,” North Carolina State University, Raleigh, NC, Report No. CASL-U-2016-1111-000.
16.
Pilch
,
M.
, and
Salko
,
R.
,
2019
, “
CTF Code Verification and Solution Verification
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-X-201X-0XXX-000.
17.
Porter
,
N.
,
Mousseau
,
V.
, and
Salko
,
R. K.
, January
2017
, “
V&V for Residual Formulation CTF
,” CASL Verification Workshop, Oak National Laboratory, Oak Ridge, TN.
18.
Salko
,
R. K.
,
2019
,
CTF 4.0 Validation and Verification
,
Oak Ridge National Laboratory
,
Oak Ridge, TN
.
19.
Salko
,
R. K.
,
2016
,
CTF Validation and Verification
,
Penn State University
,
Pennsylvania, PA
.
20.
Salko
,
R. K.
, and
Avramova
,
M. N.
,
2019
,
CTF 4.0 Theory Manual
,
Oak Ridge National Laboratory
,
Oak Ridge, TN
.
21.
Salko
,
R. K.
, and
Avramova
,
M. N.
,
2016
,
CTF Theory Manual
,
North Carolina State University
,
Raleigh, NC
.
22.
Salko
,
R. K.
,
Delchini
,
M. O.
,
Zhao
,
X.
,
Pointer
,
D.
, and
Gurecky
,
W.
,
2017
, “
Summary of CTF Accuracy and Fidelity Improvements in FY17
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2017-1428-000.
23.
Toptan
,
A.
,
Porter
,
N. W.
,
Salko
,
R. K.
, and
Avramova
,
M. N.
,
2018
, “
Implementation and Assessment of Wall Friction Models for LWR Core Analysis
,”
Ann Nucl Energy
,
115
, pp.
565
572
.10.1016/j.anucene.2018.02.022
24.
Wysocki
,
A.
,
Hu
,
J.
,
Salko
,
R.
, and
Kochunas
,
B.
, April 30,
2018
, “
Improvement of CTF for RIA Analyses
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2018-1608-000.
25.
Anderson
,
D.
, and
Kendrick
,
B.
,
2016
, “MAMBA Validation and Verification Plan,” CASL, Oak Ridge, TN, Report No. CASL-I-2016-1132-000.
26.
Kendrick
,
B.
, and
Barber
,
J.
, October 08,
2012
, “
Initial Validation and Benchmark Study of 3D MAMBA v2.0 Against the Walt Loop Experiment and BOA 3.0
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-I-2012-0238-000 (L2:MPO.CRUD.P5.02).
27.
Okhyusen
,
2018
, “MAMBA Theory Manual,” CASL, Oak Ridge, TN, Report No. CASL-I-2018-1975-000.
28.
Hales
,
J.
,
2017
, “
Bison Verification and Validation Update
,” CASL Verification Workshop, Oak National Laboratory, Oak Ridge, TN.
29.
PItts
,
2018
, “
Bison Documentation Expansion for Tensor Mechanics and Layered 1D Capabilities
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2018-1644-000.
30.
Williamson
,
R.
,
2016
, “
BISON Verification and Validation Plan for LWR Fuel: Status and Plans
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-X-2016-1060-000.
31.
Pointer
,
W. D.
,
2016
, “
Star CCM+ Verification and Validation Plan
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2016-1198-000.
32.
CASL,
2014
, “
Phase 2 Proposal: The Consortium for Advanced Simulation of Light Water Reactors (CASL)
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-I-2014-0046-000.
33.
Karoutas
,
Z. E.
,
2010
, “
Challenge Problem Technical Specification
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-I-2010-0009-000.
34.
Jina
,
M.
, and
Short
,
M.
,
2014
, “
L3: MPO.CRUD.P8.02 Two-Phase Fluid Flow Modeling in CRUD Using MAMBA-BDM
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2014-0143-000.
35.
Simon
,
H.
,
1962
, “
The Architecture of Complexity
,”
Proc. Am. Philos. Soc.
,
106
(
6
), pp.
467
482
.
36.
Aksan
,
S. R.
,
Bessette
,
D.
,
Brittain
,
I.
,
D'Auria
,
F. S.
,
Gruber
,
P.
,
Holmstrom
,
H. L. O.
,
Landry
,
R.
,
Naff
,
S.
,
Pochard
,
R.
, and
Preusche
,
G.
,
1987
, “
CSNI Code Validation Matrix of Thermo-Hydraulic Codes for LWR LOCA and Transients
,” Committee on the Safety of Nuclear Installations, OECD Nuclear Energy Agency, Paris, France.
37.
Lewis
,
M. J.
,
Pochard
,
R.
,
D'Auria
,
F. S.
,
Karwat
,
H.
,
Wolfert
,
K.
,
Yadigaroglu
,
G.
, and
Holmstrom
,
H. L. O.
,
1989
, “
Thermohydraulics of Emergency Core Cooling in Light Water Reactors-A State-of-the-Art Report. CSNI Report
,” Committee on the Safety of Nuclear Installations, OECD Nuclear Energy Agency, Paris, France.
38.
Boyack
,
B.
,
Duffey
,
R.
,
Wilson
,
G.
,
Griffith
,
P.
,
Lellouche
,
G.
,
Levy
,
S.
,
Rohatgi
,
U.
,
Wulff
,
W.
, and
Zuber
,
N.
,
1989
, “
NUREG/CR-5249: Quantifying Reactor Safety Margins: Application of Code Scaling, Applicability, and Uncertainty Evaluation Methodology to a Large-Break, Loss-of-Coolant Accident
,” Nuclear Regulatory Commission, Rockville, MD.
39.
Griffiths
,
M.
,
Schlegel
,
J. P.
,
Hibiki
,
T.
,
Ishii
,
M.
,
Kinoshita
,
I.
, and
Yoshida
,
Y.
,
2014
, “
Phenomena Identification and Ranking Table for Thermal-Hydraulic Phenomena During a Small-Break LOCA With Loss of High Pressure Injection
,”
Prog. Nucl. Energy
,
73
, pp.
51
63
.10.1016/j.pnucene.2014.01.008
40.
Olivier
,
T. J.
, and
Nowlen
,
S.
,
2008
, “
A Phenomena Identification and Ranking Table (PIRT) Exercise for Nuclear Power Plant Fire Modeling Applications
,” U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research, Rockville, MD.
41.
Diamond
,
D.
,
Edgar
,
C.
,
Fratoni
,
M.
,
Gougar
,
H.
,
Hawari
,
A.
,
Hu
,
J.
,
Hudson
,
N.
,
Llas
,
D.
,
Maldonado
,
I.
,
Petrovic
,
B.
,
Rahnema
,
F.
,
Serghiuta
,
D.
, and
Zhang
,
D.
,
2016
, “
Phenomena Identification and Ranking Tables (PIRT) Report for Fluoride High-Temperature Reactor (FHR) Neutronics
,” U. S. Department of Energy, N.E.U.P, Idaho Falls, ID, Report No. CRMP-2016-08-001.
42.
Wilson
,
G. E.
, and
Boyack
,
B. E.
,
1998
, “
The Role of the PIRT Process in Experiments, Code Development and Code Applications Associated With Reactor Safety Analysis
,”
Nucl. Eng. Des.
,
186
(
1–2
), pp.
23
37
.10.1016/S0029-5493(98)00216-7
43.
Oberkampf
,
W. L.
, and
Roy
,
C. J.
,
2010
,
Verification and Validation in Scientific Computing
,
Cambridge University Press
,
Cambridge, UK
.
44.
Wang
,
R. Y.
, and
Strong
,
D. M.
,
1996
, “
Beyond Accuracy: What Data Quality Means to Data Consumers
,”
J. Manage. Inf. Syst.
,
12
(
4
), pp.
5
33
.10.1080/07421222.1996.11518099
45.
Oberkampf
,
W. L.
, and
Trucano
,
T. G.
,
2002
, “
Verification and Validation in Computational Fluid Dynamics
,”
Prog. Aerosp. Sci.
,
38
(
3
), pp.
209
272
.10.1016/S0376-0421(02)00005-2
46.
Jones
,
C.
,
Mattie
,
P.
,
Dinh
,
N.
,
Athe
,
P.
, and
Moore
,
L.
,
2019
, “
Updated Verification and Validation Assessment for VERA
,” Consortium for Advanced Simulation of Light Water Reactors, CASL, Oak Ridge, TN, Report No. CASL-U-2019-1864-000.
47.
Toulmin
,
S. E.
,
2003
,
The Uses of Arguments
,
Cambridge University Press
,
Cambridge, UK
.
48.
Kelly
,
T. P.
,
1999
,
Arguing Safety: A Systematic Approach to Managing Safety Cases
,
University of York
,
York, UK
.
49.
Ganapol
,
B.
,
2008
, “
Analytical Benchmarks for Nuclear Engineering Applications
,” Case Studies in Neutron Transport Theory, Nuclear Energy Agency (NEA), Organisation for Economic Co-operation and Development (OECD), Paris, France.
50.
Athe
,
P.
, and
Dinh
,
N.
,
2019
, “
A Framework for Assessment of Predictive Capability Maturity and Its Application in Nuclear Thermal Hydraulics
,”
Nucl. Eng. Des.
,
354
, p.
110201
.10.1016/j.nucengdes.2019.110201
51.
Salko
,
R. K.
,
2016
,
CTF Validation and Verification
,
Penn State University
,
University Park, PA
.
52.
Palmtag
,
S.
,
2016
, “Investigation of Thermal Expansion Effects in MPACT,” CASL, Oak Ridge, TN, Report No. CASL-U-2016-1015-000.
53.
Salko
,
R.
,
Slattery
,
S.
,
Lange
,
T.
,
Delchini
,
M.-O.
,
Gurecky
,
W.
,
Tatli
,
E.
, and
Collins
,
B.
,
2018
, “
Development of Preliminary VERA—CS Crud-Induced Localized Corrosion Modeling Capability (Milestone: L2:PHI.P17.03)
,” CASL, Oak Ridge, TN, Report No. CASL-U-2018-1617-000.
54.
Godfrey
,
A. T.
,
Collins
,
B. S.
,
Gentry
,
C. A.
,
Stimpson
,
S. G.
, and
Ritchie
,
J. A.
,
2017
, “
Watts Bar Unit 2 Startup Results With VERA
,” CASL, Oak Ridge, TN, Report No. CASL-U-2017-1306-000.
55.
Collins
,
B.
,
Gurecky
,
W.
,
Elliott
,
A.
,
Lindsay
,
G.
,
Coleman
,
K.
,
Smith
,
R.
, and
Andersson
,
D.
,
2019
, “
Inference of Crud Model Parameters From Plant Data
,” CASL, Oak Ridge, TN, Report No. Milestone FY19.CASL.005.
56.
Anderson
,
D.
,
2019
,
MAMBA Code and Solution Verification Status Report
,
Los Alamos National Laboratory
,
Los Alamos, NM
.
57.
Rizk
,
J.
,
Wirth
,
B. D.
, and
McMurray
,
J.
,
2019
, “
CALPHAD Modeling of PWR CRUD Internal Chemistry
,” CASL, Oak Ridge, TN, Report No. CASL-U-2019-4026-000.
58.
Williamson
,
R. L.
,
Gamble
,
K. A.
,
Perez
,
D. M.
,
Novascone
,
S. R.
,
Pastore
,
G.
,
Gardner
,
R. J.
,
Hales
,
J. D.
,
Liu
,
W.
, and
Mai
,
A.
,
2016
, “
Validating the BISON Fuel Performance Code to Integral LWR Experiments
,”
Nucl. Eng. Des.
,
301
, pp.
232
244
.10.1016/j.nucengdes.2016.02.020
59.
Williamson
,
R. L.
,
Pastore
,
G.
,
Gardner
,
R. J.
,
Gamble
,
K. A.
,
Novascone
,
S.
,
Tompkins
,
J.
, and
Liu
,
W.
,
2019
, “
LOCA Challenge Problem Final Report
,” CASL, Oak Ridge, TN, Report No. CASL-U-2019-1856-000.
60.
Hamilton
,
S.
,
Berrilla
,
M. A.
,
Clarno
,
K. T.
, and
Pawlowski
,
R. P.
,
2014
, “
An Assessment of Coupling Algorithms for Nuclear Reactor Core Physics Simulations
,” CASL, Oak Ridge, TN, Report No. CASL-U-2014-0149-000-b.
61.
Berrilla
,
M. A.
,
Clarno
,
K. T.
, and
Hamilton
,
S. P.
,
2014
, “
Evaluation of Coupling Approaches
,” CASL, Oak Ridge, TN, Report No. CASL-U-2014-0081-000.
62.
Clarno
,
K. T.
, and
Pawlowski
,
R. P.
,
2014
, “
Incorporate MPACT Into TIAMAT and Demonstrate Pellet-Clad Interaction (PCI) Calculations
,” CASL, Oak Ridge, TN, Report No. CASL-U-2015-0022-000.
63.
Godfrey
,
A. T.
,
Collins
,
B. S.
,
Kim
,
K. S.
,
Lee
,
R.
,
Powers
,
J.
,
Salko
,
R.
,
Stimpson
,
S. G.
,
Wieselquist
,
W. A.
,
Montgomery
,
R.
,
Montgomery
,
R.
,
Kochunas
,
B.
,
Jabaay
,
D. R.
,
Capps
,
N.
, and
Secker
,
J.
, June 26,
2015
, “
VERA Benchmarking Results for Watts Bar Nuclear Plant
,” CASL, Oak Ridge, TN, Report No. CASL-U-2015-0206-000.
64.
Khuwaileh
,
B. A.
, and
Turinsky
,
P. J.
,
2016
, “
Data Assimilation and Uncertainty Quantification Using VERA—CS for a Core Wide LWR Problem With Depletion (L2:VMA.P12.01)
,” CASL, Oak Ridge, TN, Report No. CASL-U-2016-1054-000.
65.
Hooper
,
R.
,
2016
, “
Initial UQ of CIPS
,” CASL, Oak Ridge, TN, Report No. CASL-U-2016-XXXX-XXX.
66.
Brown
,
C. S.
, and
Zhang
,
H.
,
2016
, “
Uncertainty Quantification and Sensitivity Analysis With CASL Core Simulator VERA—CS
,” Idaho National Laboratory, Idaho Falls, ID, Report No.
INL/JOU-16-38314
.https://www.osti.gov/pages/servlets/purl/1357750
67.
Khalik
,
H. S. A.
,
2013
, “
Uncertainty Quantification and Data Assimilation (UQ/DA) Study on a VERA Core Simulator Component for CRUD Analysis
,” CASL, Oak Ridge, TN, Report No. CASL-U-2013-0184-000.
You do not currently have access to this content.