In computational physics and engineering, numerical models are developed to predict the behavior of a system whose response cannot be measured experimentally. A key aspect of science-based predictive modeling is the assessment of prediction credibility. Credibility, which is usually demonstrated through the activities of model Verification and Validation (V&V), quantifies the extent to which simulation results can be analyzed with confidence to represent the phenomenon of interest with an accuracy consistent with the intended use of the model.
The paper develops the idea that assessing the credibility of a mathematical or numerical model must combine three components: 1) Improving the fidelity to test data; 2) Studying the robustness of prediction-based decisions to variability, uncertainty, and lack of knowledge; and 3) Establishing the expected prediction accuracy of the models in situations where test measurements are not available. A Theorem demonstrates the irrevocable trade-off between "the Good, the Bad, and the Ugly," or robustness to uncertainty, fidelity to data, and confidence in prediction. Even though the conventional activities of model V&V are generally restricted to improving the fidelity to data through the correlation of test and simulation results and the calibration of model parameters, the other two components are equally important. The main reason is that optimal models - in the sense of models that minimize the prediction errors with respect to the available test data - possess exactly zero robustness to uncertainty and lack of knowledge. This means that small variations in the setting of model parameters, or small errors in the knowledge of the functional form of the models, can lead to an actual fidelity that is significantly poorer than the one demonstrated through calibration.
Clearly, fidelity to data matters because no analyst will trust a numerical simulation that does not reproduce the measurements of past experiments or historical databases. Robustness to uncertainty is equally critical to minimize the vulnerability of decisions to uncertainty and lack of knowledge. It may be argued, however, that the most important aspect of credibility is the assessment of confidence in prediction, which is generally not addressed in the literature. Assessing the confidence in prediction here refers to an assessment of prediction error away from settings where physical experiments have been performed, which must include a rigorous quantification of the sources of variability, uncertainty, and lack of knowledge, and their sensitivity effects on model prediction.
The theoretical results are illustrated with an engineering example. Trading off the objectives of fidelity to data, robustness to uncertainty, and confidence in prediction suggests a decision-making strategy that differs from performance optimization. Current efforts to develop a framework for science-based certification will be briefly overviewed.