Abstract

This paper presents a practical methodology for propagating and processing uncertainties associated with random measurement and estimation errors (that vary from test-to-test) and systematic measurement and estimation errors (uncertain but similar from test-to-test) in inputs and outputs of replicate tests to characterize response variability of stochastically varying test units. Also treated are test condition control variability from test-to-test and sampling uncertainty due to limited numbers of replicate tests. These aleatory variabilities and epistemic uncertainties result in uncertainty on computed statistics of output response quantities. The methodology was developed in the context of processing experimental data for “real-space” (RS) model validation comparisons against model-predicted statistics and uncertainty thereof. The methodology is flexible and sufficient for many types of experimental and data uncertainty, offering the most extensive data uncertainty quantification (UQ) treatment of any model validation method the authors are aware of. It handles both interval and probabilistic uncertainty descriptions and can be performed with relatively little computational cost through use of simple and effective dimension- and order-adaptive polynomial response surfaces in a Monte Carlo (MC) uncertainty propagation approach. A key feature of the progressively upgraded response surfaces is that they enable estimation of propagation error contributed by the surrogate model. Sensitivity analysis of the relative contributions of the various uncertainty sources to the total uncertainty of statistical estimates is also presented. The methodologies are demonstrated on real experimental validation data involving all the mentioned sources and types of error and uncertainty in five replicate tests of pressure vessels heated and pressurized to failure. Simple spreadsheet procedures are used for all processing operations.

Issue Section:
Research Papers
Topics:
Uncertainty, Failure, Model validation

References

1.
Romero
,
V. J.
,
Luketa
,
A.
, and
Sherman
,
M.
,
2010
, “
Application of a Versatile ‘Real Space’ Validation Methodology to a Fire Model
,”
AIAA J. Thermophys. Heat Transfer
,
24
(
4
), pp.
730
–
744
.10.2514/1.46358
2.
Romero
,
V. J.
,
2011
, “
Comparison of Several Model Validation Conceptions Against a ‘Real Space’ End-to-End Approach
,”
Soc. Automot. Eng. Int. J. Mater. Manuf.
,
4
(
1
), pp.
396
–
420
.10.4271/2011-01-0238
3.
Romero
,
V.
,
Dempsey
,
F.
, and
Antoun
,
B.
,
2016
, “
Application of UQ and V&V to Experiments and Simulations of Heated Pipes Pressurized to Failure
,”
Simulation Credibility—Advances in Verification, Validation, and Uncertainty Quantification
,
U.
Mehta
,
D.
Eklund
,
V.
Romero
,
J.
Pearce
, and
N.
Keim
, eds.,
Joint Army/Navy/NASA/Air Force (JANNAF)
, Document No. NASA/TP-2016-219422 and JANNAF/GL-2016-0001, Nov. 2016, Chap. 11.https://www.osti.gov/servlets/purl/1427042
4.
Jamison
,
R.
,
Romero
,
V.
,
Stavig
,
M.
,
Buchheit
,
T.
, and
Newton
,
C.
,
2016
, “
Experimental Data Uncertainty, Calibration, and Validation of a Viscoelastic Potential Energy Clock Model for Inorganic Sealing Glasses
,” Sandia National Laboratories, Albuquerque, NM, Document No. SAND2016-4635 C.
5.
Romero
,
V.
,
Heaphy
,
R.
,
Rutherford
,
B.
, and
Lewis
,
J. R.
,
2016
, “
Uncertainty Quantification and Model Validation for III-V SSICs in Annular Core Research Reactor Shots
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2016-11772.
6.
Romero
,
V.
,
2019
, “
Real-Space Model Validation and Predictor-Corrector Extrapolation Applied to the Sandia Cantilever Beam End-to-End UQ Problem
,”
AIAA
Paper No. 2019-1488.10.2514/6.2019-1488
7.
Schaefer
,
J.
,
Romero
,
V.
,
Schafer
,
S.
,
Leyde
,
B.
, and
Denham
,
C.
,
2020
, “
Approaches for Quantifying Uncertainties in Computational Modeling for Aerospace Applications
,”
AIAA
Paper No. 2020-1520.10.2514/6.2020-1520
8.
ISO
,
1995
,
Guide to the Expression of Uncertainty in Measurement
,
International Organization for Standardization
,
Geneva, Switzerland
.
9.
Coleman
,
H. W.
, and
Steele
, and
W. G.
, Jr. ,
1999
,
Experimentation and Uncertainty Analysis for Engineers
, 2nd ed.,
Wiley
,
New York
.
10.
Ferson
,
S.
,
Oberkampf
,
W. L.
, and
Ginzburg
,
L.
,
2008
, “
Model Validation and Predictive Capability for the Thermal Challenge Problem
,”
Comput. Methods Appl. Mech. Eng.
,
197
(
29–32
), pp.
2408
–
2430
.10.1016/j.cma.2007.07.030
11.
Oberkampf
,
W. L.
, and
Roy
,
C. J.
,
2010
,
Verification and Validation in Scientific Computing
,
Cambridge University Press
, Cambridge, GB.
12.
Roy
,
C. J.
, and
Oberkampf
,
W. L.
,
2011
, “
A Comprehensive Framework for Verification, Validation, and Uncertainty Quantification in Scientific Computing
,”
Comput. Methods Appl. Mech. Eng.
,
200
(
25–28
), pp.
2131
–
2144
.10.1016/j.cma.2011.03.016
13.
ASME
,
2012
, “
An Illustration of the Concepts of Verification and Validation in Computational Solid Mechanics
,” ASME Standard No. V&V 10.1-2012.
14.
Jung
,
B. C.
,
Park
,
J.
,
Oh
,
H.
,
Kim
,
J.
, and
Youn
,
B. D.
,
2015
, “
A Framework of Model Validation and Virtual Product Qualification With Limited Experimental Data Based on Statistical Inference
,”
Struct. Multidiscip. Optim.
,
51
(
3
), pp.
573
–
583
.10.1007/s00158-014-1155-2
15.
Voyles
,
I. T.
, and
Roy
,
C. J.
,
2015
, “
Evaluation of Model Validation Techniques in the Presence of Aleatory and Epistemic Input Uncertainties
,”
AIAA
Paper No. 2015-1374.10.2514/6.2015-1374
16.
Lee
,
H. B.
,
Ghia
,
U.
,
Bayyuk
,
S.
,
Oberkampf
,
W. L.
,
Roy
,
C. J.
,
Benek
,
J. A.
,
Rumsey
,
C. R.
,
Powers
,
J. M.
,
Bush
,
R. H.
, and
Mani
,
M.
,
2016
, “
Development and Use of Engineering Standards for Computational Fluid Dynamics for Complex Aerospace Systems
,”
AIAA
Paper No. 2016-3811.10.2514/6.2016-3811
17.
Moon
,
M.
,
Choi
,
K. K.
, and
Lamb
,
D.
,
2019
, “
Target Output Distribution and Distribution of Bias for Statistical Model Validation Given a Limited Number of Test Data
,”
Struct. Multidiscip. Optim.
,
60
(
4
), pp.
1327
–
1353
.10.1007/s00158-019-02338-z
18.
Hu
,
J.
,
Zhou
,
Q.
,
McKeand
,
A.
,
Xie
,
T.
, and
Choi
,
S.-K.
,
2020
, “
A Model Validation Framework Based on Parameter Calibration Under Aleatory and Epistemic Uncertainty
,”
Struct. Multidiscip. Optim.
,
63
, pp.
645
–
660
.10.1007/s00158-020-02715-z
19.
Oberkampf
,
W. L.
, and
Barone
,
M. F.
,
2006
, “
Measures of Agreement Between Computation and Experiment: Validation Metrics
,”
J. Comput. Phys.
,
217
(
1
), pp.
5
–
36
.10.1016/j.jcp.2006.03.037
20.
Mullins
,
J.
, and
Mahadevan
,
S.
,
2016
, “
Bayesian Uncertainty Integration for Model Calibration, Validation, and Prediction
,”
ASME J. Verif., Validation Uncertainty Quantif.
,
1
(
1
), p.
011006
.10.1115/1.4032371
21.
Coleman
,
H. W.
, and
Stern
,
F.
,
1997
, “
Uncertainties in CFD Code Validation
,”
ASME J. Fluids Eng.
,
119
(
4
), pp.
795
–
803
.10.1115/1.2819500
22.
ASME
,
2009
, “
Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer
,” ASME Standard No. V&V 20-2009.
23.
Romero
,
V.
,
2020
, “
Relationships Between ASME VV 10 & 20, AIAA CFD, and Real Space Model Validation Frameworks
,” Sandia National Laboratories, Albueuqrque, NM, Document No. SAND2020-5230C.
24.
Romero
,
V.
,
Mullins
,
J.
,
Swiler
,
L.
, and
Urbina
,
A.
,
2013
, “
A Comparison of Methods for Representing and Aggregating Experimental Uncertainties Involving Sparse Data—More Results
,”
Soc. Automot. Eng. Int. J. Mater. Manuf.
,
6
(
3
), pp.
447
–
473
.10.4271/2013-01-0946
25.
Romero
,
V.
,
Swiler
,
L.
,
Urbina
,
A.
, and
Mullins
,
J.
,
2013
, “
A Comparison of Methods for Representing Sparsely Sampled Random Quantities
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2013-4561.
26.
Romero
,
V.
,
Schroeder
,
B.
,
Dempsey
,
J. F.
,
Breivik
,
N.
,
Orient
,
G.
,
Antoun
,
B.
,
Lewis
,
J. R.
, and
Winokur
,
J.
,
2018
, “
Simple Effective Conservative Treatment of Uncertainty From Sparse Samples of Random Variables and Functions
,”
ASCE-ASME J. Uncertainty Risk Eng. Syst., Part B: Mech. Eng.
,
4
(
4
), p.
041006
.10.1115/1.4039558
27.
Romero
,
V.
,
2021
, “
Arguments for the Generality and Effectiveness of ‘Discrete Direct’ Model Calibration and Uncertainty Propagation Vs. Other Calibration-UQ Approaches
,” Sandia National Laboratories, Albuquerque, NM, Jan. 3–8, Document No. SAND2021-6491 C submitted for AIAA SciTech Conference.
28.
Romero
,
V.
,
2015
, “
The Real Space Model Validation Approach as a (Unifying?) Extended Hybrid of the ASME VV10 and VV20 Approaches
,” Sandia National Laboratories, Albuquerque, NM, Document No. SAND2015-3752 C.
29.
Romero
,
V.
,
2010
, “
Data & Model Conditioning for Multivariate Systematic Uncertainty in Model Calibration, Validation, and Extrapolation
,”
AIAA
Paper No. 2010-2511.10.2514/6.2010-2511
30.
Black
,
A.
,
Romero
,
V.
,
Breivik
,
N.
,
Orient
,
G.
,
Antoun
,
B.
,
Dodd
,
A.
, and
Suo-Anttila
,
J.
,
2019
, “
Predictive Capability Assessment Project: Abnormal Thermal-Mechanical Breach V&V/UQ
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2019-13790.
31.
Black
,
A.
,
Romero
,
V.
,
Breivik
,
N.
,
Orient
,
G.
,
Suo-Anttila
,
J.
,
Antoun
,
B.
, and
Dodd
,
A.
,
2015
, “
Verification, Validation, and Uncertainty Quantification of a Thermal-Mechanical Pressurization and Breach Application
,”
Archives of the ASME Verification & Validation Symposium
, Las Vegas, NV, May 13–15, Presentation No. VVS2015-8047.
32.
Romero
,
V.
,
Black
,
A.
,
Orient
,
G.
, and
Antoun
,
B.
,
2020
, “
Propagating Stress-Strain Curve Variability in Multi-Material Problems: Temperature-Dependent Material Tests to Plasticity Models to Structural Failure Predictions
,”
Engineering Failure Analysis
,
IntechOpen Publishers
, London, GB.
33.
Romero
,
V.
,
2020
, “
Propagating and Combining Aleatory Uncertainties Characterized by Continuous Random Variables and Sparse Discrete Realizations From Random Functions
,”
AIAA
Paper No. 2020-1415.10.2514/6.2020-1415
34.
Romero
,
V.
,
Black
,
A.
,
Dodd
,
A.
,
Orient
,
G.
,
Breivik
,
N.
,
Antoun
,
B.
, and
Suo-Anttila
,
J.
, “
Real-Space Model Validation Methodology Applied to Thermal-Chemical-Mechanical Response and Weld Failure in Heated Pressurizing Canisters
,” Sandia National Laboratories, Albuquerque, NM (Document preparation).
35.
Romero
,
V.
, and
Black
,
A.
,
2021
, “
Adaptive Polynomial Response Surfaces and Level-1 Probability Boxes for Propagating and Representing Aleatory and Epistemic Components of Uncertainty
,”
AIAA
Paper No. 2021-1366.10.2514/6.2021-1366
36.
Antoun
,
B. R.
,
2012
, “
Material Characterization and Coupled Thermal-Mechanical Experiments for Pressurized, High Temperature Systems
,” Sandia National Laboratories, Livermore, CA, Technical Report.
37.
Suo-Anttila
,
J. M.
,
Dodd
,
A. B.
, and
Jernigan
,
D. A.
,
2012
, “
Thermal Mechanical Exclusion Region Barrier Breach Foam Experiments (800C Upright and Inverted 20 lb/ft3 PMDI Cans)
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2012-7600.
38.
Erickson
,
K. L.
,
Dodd
,
A. B.
, and
Hogan
,
R. E.
,
2010
, “
Modeling Pressurization Caused by Thermal Decomposition of Highly Charring Foam in Sealed Containers
,”
Proceedings of the BCC 2010
, Stamford, CT, May 23–26.
39.
Erickson
,
K. L.
,
Dodd
,
A. B.
,
Hogan
,
R. E.
, and
Dowding
,
K. J.
,
2010
, “
Heat Transfer, Foam Decomposition, and Container Pressurization: Comparison of Experimental and Modeling Results
,”
Proceedings of the Interflam 2010
, Nottingham, UK, July 5–7.
40.
Erickson
,
K. L.
,
Dodd
,
A. B.
, and
Quintana
,
E. C.
,
2011
, “
Physical Behavior and Container Pressurization During Thermal Decomposition of Polyurethane Foams
,”
Proceedings of the BCC 2011
, Stamford, CT, May 23–25.
41.
Nakos
,
J. T.
,
2004
, “
Uncertainty Analysis of Thermocouple Measurements Used in Normal and Abnormal Thermal Environments Experiments at the Radiant Heat Facility and the Lurance Canyon Burn Site
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2004-1023.
42.
Romero
,
V. J.
,
Shelton
,
J. W.
, and
Sherman
,
M. P.
,
2006
, “
Modeling Boundary Conditions and Thermocouple Response in a Thermal Experiment
,”
ASME
Paper No. IMECE2006-15046.10.1115/IMECE2006-15046
43.
Larsen
,
M. E.
, and
Dodd
,
A. B.
,
2014
, “
Modeling and Validation of the Thermal Response of TDI Encapsulating Foam as a Function of Initial Density
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2014-17850.
44.
Romero
,
V.
, and
Black
,
A.
,
2021
, “
Processing Aleatory and Epistemic Uncertainties in Experimental Data From Sparse Replicate Tests of Stochastic Systems for Real-Space Model Validation
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2021-0796J.
45.
Romero
,
V. J.
,
2007
, “
Model-Discretization Sizing and Calculation Verification for Multipoint Simulations Over Large Parameter Spaces
,”
AIAA
Paper No. 2007-1953.10.2514/6.2007-1953
46.
Schroeder
,
B.
,
Silva
,
H.
, III , and
Smith
,
K. D.
,
2018
, “
Separability of Mesh Bias and Parametric Uncertainty for a Full System Thermal Analysis
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2018-1007.
47.
Romero
,
V.
,
Bonney
,
M.
,
Schroeder
,
B.
, and
Weirs
,
V. G.
,
2017
, “
Evaluation of a Class of Simple and Effective Uncertainty Methods for Sparse Samples of Random Variables and Functions
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2017-12349.
48.
Winokur
,
J.
, and
Romero
,
V.
,
2016
, “
Optimal Design of Computer Experiments for Uncertainty Quantification With Sparse Discrete Sampling
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2016-12608.
Copyright © 2021 by Sandia National Laboratories (SNL)
You do not currently have access to this content.